Video processing method and apparatus, computing device, medium, and program product

Abstract

The application provides a video processing method and device, a computing device, a medium and a program product, and belongs to the field of video processing. The video processing method comprises: for each to-be-processed frame in a to-be-processed video: determining a first pixel set and a second pixel set corresponding to the to-be-processed frame from a plurality of pixels based on motion state information of each pixel of the to-be-processed frame; determining pixel positions of a plurality of first pixels in a prediction frame respectively according to motion vectors of the plurality of first pixels, to obtain a first prediction pixel set corresponding to the prediction frame; determining a second prediction pixel set corresponding to the prediction frame based on the second pixel sets corresponding to at least one to-be-processed frame before the to-be-processed frame in the to-be-processed video; and generating the prediction frame based on the first prediction pixel set and the second prediction pixel set. The method can improve the video rendering effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video processing, and in particular to a video processing method and device, a computing device, a medium and a program product. BACKGROUND

[0002] Real-time rendering video stream interpolation inserts one or more new intermediate frames between two frames rendered by a rendering engine natively, to improve the smoothness of the picture. Improving the game frame rate to obtain a smoother visual experience is the most important and most widely used application scenario of this technology.

[0003] Currently, in the field of real-time rendering, interpolation techniques based on motion compensation are commonly used. However, the existing real-time rendering video stream interpolation methods, whether academic research or commercial frame generation systems, may have various visual defects, such as blurring and ghosting, tearing and deformation, etc., in the edge zone where the moving object and the background intersect. SUMMARY

[0004] The present application aims to at least solve the technical problem of visual defects in the process of real-time rendering video stream interpolation in the background art. To this end, one object of the present application is to provide a video processing method to improve the video rendering effect.

[0005] Embodiments of the first aspect of the present application provide a video processing method, comprising: for any one to-be-processed frame in a to-be-processed video: determining a first pixel set and a second pixel set corresponding to the to-be-processed frame from a plurality of pixels based on motion state information of each pixel in the plurality of pixels, wherein the first pixel set includes a plurality of first pixels, any one of the plurality of first pixels has a first motion attribute, the second pixel set includes a plurality of second pixels, any one of the plurality of second pixels has a second motion attribute, and the second motion attribute is different from the first motion attribute; determining pixel positions of the plurality of first pixels in a prediction frame respectively according to motion vectors of the plurality of first pixels, to obtain a first prediction pixel set corresponding to the prediction frame, the first prediction pixel set including a plurality of first prediction pixels corresponding to the plurality of first pixels respectively, the plurality of first prediction pixels respectively having pixel positions of the corresponding first pixels in the prediction frame; determining a second prediction pixel set corresponding to the prediction frame based on second pixel sets corresponding to at least one to-be-processed frame before the to-be-processed frame in the to-be-processed video, the second prediction pixel set including a plurality of second prediction pixels; and generating the prediction frame based on the first prediction pixel set and the second prediction pixel set.

[0006] According to the technical scheme of the embodiment of the present application, the pixels in the current frame are split into a moving object layer and a background layer with different motion attributes according to the motion state information before the frame insertion, and different processing strategies are respectively performed on different layers, so that the visual defects of the inserted intermediate frame at the edge of the junction of the moving object and the background are effectively reduced, and the visual effect is effectively improved.

[0007] In some embodiments, determining the first pixel set and the second pixel set corresponding to the frame to be processed from the plurality of pixels comprises: determining a plurality of first pixels and a plurality of second pixels from the plurality of pixels according to the velocity vectors corresponding to the plurality of pixels. In the rendering engine level, a specific motion attribute is usually specified for an object in a scene, so that the pixels in the current frame are quickly identified as belonging to a moving object or a background area according to the corresponding velocity vector, and the division of different layers is realized.

[0008] In some embodiments, determining the second prediction pixel set corresponding to the prediction frame according to the second pixel set corresponding to at least one frame to be processed located before the frame to be processed in the video to be processed comprises: generating a history pixel set based on the second pixel set corresponding to at least one frame to be processed located before the frame to be processed in the video to be processed, the history pixel set including the second pixels in the second pixel set corresponding to at least one frame to be processed; determining a plurality of filling pixels from the history pixel set according to the second pixel set corresponding to the frame to be processed; and determining the second prediction pixel set based on the second pixel set corresponding to the frame to be processed and the plurality of filling pixels. The second pixel set in the current frame, i.e., the background layer, can be filled according to the history pixels in each frame in the processing history, effectively reducing the visual defects such as background area holes and artifacts caused by the movement of the moving object.

[0009] In some embodiments, determining the plurality of filling pixels from the history pixel set according to the second pixel set corresponding to the frame to be processed comprises: determining at least one spatial coordinate displacement between at least one frame to be processed and the frame to be processed; performing coordinate transformation on the second pixels in the second pixel set corresponding to at least one frame to be processed in the history pixel set based on the at least one spatial coordinate displacement to obtain the coordinate-transformed second pixels corresponding to at least one frame to be processed; and determining the plurality of filling pixels from the history pixel set based on the second pixel set corresponding to the frame to be processed and the coordinate-transformed second pixels corresponding to at least one frame to be processed. By performing coordinate transformation on the pixels in the history pixel set according to the spatial coordinate displacement between different frames in the processing history and the current frame, the pixels corresponding to each frame in the history pixel set can be aligned with the perspective of the current frame, so that accurate filling pixels are found in the history pixel set to fill the background layer.

[0010] In some embodiments, the video processing method further comprises: for any one of the to-be-processed frames in the to-be-processed video: determining a third pixel set corresponding to the to-be-processed frame from the plurality of pixels based on the material information of each of the plurality of pixels, wherein the third pixel set comprises a plurality of third pixels, and any one of the plurality of third pixels has a predetermined material attribute; and determining a third prediction pixel set corresponding to the prediction frame based on the plurality of third pixels, wherein the third prediction pixel set comprises a plurality of third prediction pixels corresponding to the plurality of third pixels respectively. The third pixel set is determined according to the material information, so that the object with the predetermined material can be accurately identified and processed, thereby generating a more realistic and accurate prediction frame.

[0011] In some embodiments, generating the prediction frame based on the first prediction pixel set and the second prediction pixel set comprises: generating the prediction frame based on the first prediction pixel set, the second prediction pixel set, and the third prediction pixel set. The prediction frame is generated based on the pixel sets with different motion attributes and the pixel set with the predetermined material, so that the complex visual special effects can be correctly processed, thereby generating a more realistic and accurate frame insertion result.

[0012] In some embodiments, the video processing method further comprises: determining a plurality of second pixel subsets from the second pixel set according to the depth information of each of the plurality of second pixels and a predetermined depth threshold condition. The layers are further subdivided into a plurality of levels according to the depth of the pixels, so that the scene with complex and multi-level static geometry can be processed more accurately, the accuracy of video rendering is improved, and the rendering effect is improved.

[0013] In some embodiments, determining the second prediction pixel set corresponding to the prediction frame based on the second pixel sets corresponding to at least one to-be-processed frame before the to-be-processed frame in the to-be-processed video comprises: determining a plurality of second pixel subsets corresponding to the at least one to-be-processed frame respectively according to the depth information of each of the plurality of second pixels and a depth threshold condition; determining a plurality of second prediction pixel subsets corresponding to the prediction frame based on the plurality of second pixel subsets corresponding to the at least one to-be-processed frame respectively; and obtaining the second prediction pixel set based on the plurality of second prediction pixel subsets corresponding to the prediction frame. The layers with different depths in the background layer are filled respectively, and finally fused to obtain the background layer in the prediction frame, so that the error filling caused by the parallax effect can be effectively reduced, and the background layer in the prediction frame has high spatial correctness.

[0014] An embodiment of the second aspect of the present application provides a video processing apparatus, comprising: a determination module configured to, for any to-be-processed frame in a to-be-processed video: determine, based on motion state information of each of a plurality of pixels in the to-be-processed frame, a first pixel set and a second pixel set corresponding to the to-be-processed frame from the plurality of pixels, wherein the first pixel set comprises a plurality of first pixels, any first pixel in the plurality of first pixels has a first motion attribute, the second pixel set comprises a plurality of second pixels, any second pixel in the plurality of second pixels has a second motion attribute, and the second motion attribute is different from the first motion attribute; a first prediction module configured to determine, according to a motion vector of each of the plurality of first pixels, a pixel position of each of the plurality of first pixels in a prediction frame, to obtain a first prediction pixel set corresponding to the prediction frame, the first prediction pixel set comprising a plurality of first prediction pixels corresponding to the plurality of first pixels respectively, and each of the plurality of first prediction pixels having a pixel position in the prediction frame corresponding to the first pixel; a second prediction module configured to determine, based on a second pixel set corresponding to each of at least one to-be-processed frame located before the to-be-processed frame in the to-be-processed video, a second prediction pixel set corresponding to the prediction frame, the second prediction pixel set comprising a plurality of second prediction pixels; and a generation module configured to generate the prediction frame based on the first prediction pixel set and the second prediction pixel set.

[0015] An embodiment of the third aspect of the present application provides a computing device, comprising: at least one processor; and at least one memory communicatively connected with the at least one processor, the at least one memory storing instructions, which, when executed by the at least one processor alone or jointly, cause the computing device to perform the video processing method in the above-described embodiments.

[0016] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, storing instructions, which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the video processing method in the above-described embodiments.

[0017] An embodiment of the fifth aspect of the present application provides a computer program product, comprising instructions, which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the video processing method in the above-described embodiments.

[0018] The above description is only a summary of the technical solutions of the present application. In order to enable one of ordinary skill in the art to better understand the technical means of the present application and to implement the same according to the contents of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the drawings, like reference numerals refer to same or similar components throughout the several views. The drawings are not necessarily to scale. It should be understood that the drawings are merely schematic and certain

[0020] Figure 1 Flowchart of a method for video processing according to some embodiments of the present application;

[0021] Figure 2 Flowchart of determining a second set of predicted pixels according to some embodiments of the present application;

[0022] Figure 3 Flowchart of determining a padding pixel according to some embodiments of the present application;

[0023] Figure 4 Flowchart of determining a third set of predicted pixels according to some embodiments of the present application;

[0024] Figure 5 Flowchart of determining a second set of predicted pixels according to some embodiments of the present application;

[0025] Figure 6 Schematic diagram of a video processing apparatus according to some embodiments of the present application;

[0026] Figure 7 Schematic diagram of a computing device according to some embodiments of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the terms "comprising" and "having," and any variations thereof, as used herein are intended to cover a non-exclusive inclusion.

[0029] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0030] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. As those skilled in the art will appreciate, the embodiments described herein can be adapted to a wide variety of applications.

[0031] In the description of the embodiments of the application, the term“and / or” only means an association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character“ / ” herein generally means that the front and rear associated objects are in an“or” relationship.

[0032] In the description of the embodiments of the application, the term“a plurality of” refers to two or more (including two), and similarly, “a plurality of groups” refers to two or more groups (including two groups), and “a plurality of pieces” refers to two or more pieces (including two pieces).

[0033] In the description of the embodiments of the application, the technical terms“center”,“longitudinal”,“transverse”,“length”,“width”,“thickness”,“upper”,“lower”,“front”,“rear”,“left”,“right”,“vertical”,“horizontal”,“top”,“bottom”,“inner”,“outer”,“clockwise”,“counterclockwise”,“axial”,“radial”,“circumferential” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the embodiments of the application.

[0034] In the description of the embodiments of the application, unless otherwise explicitly specified and limited, the technical terms“mounting”,“connection”,“connection”,“fixing” and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the application can be understood according to the specific circumstances.

[0035] The real-time rendering video stream interpolation process synthesizes one or more new intermediate frames between the two frames rendered by the rendering engine to improve the smoothness of the picture. Improving the game frame rate to obtain a smoother visual experience is the most important and most widely used application scenario of this technology.

[0036] Currently, in the field of real-time rendering, interpolation techniques based on motion compensation are commonly used. However, this approach does not work well when dealing with occlusion problems. When objects in the picture move to block or expose the background behind them, the motion information of these newly appearing areas is missing. With the rise of deep learning, neural networks have also been used in the field of real-time rendering. However, this way of using neural networks for interpolation usually "black-boxes" the fusion of the precise geometric motion vectors provided by the rendering engine and the optical flow information used to capture shadow, particle, and other effects, treating all data sources as a whole and inputting them into a complex network for processing. This approach also cannot well solve the problem of data conflict and ambiguity at the junction of moving objects and static backgrounds, and still cannot well solve the problem of boundary flaws.

[0037] Existing real-time rendering video stream interpolation methods, whether academic research or commercial frame generation systems, treat the rendered picture as a single, whole entity when processing the interpolation process, without fundamentally distinguishing and isolating the pixels belonging to moving foreground objects from the pixels belonging to static backgrounds. This undifferentiated holistic approach is the root cause of the persistence of various visual flaws at the edge zone where moving objects and backgrounds intersect.

[0038] In order to obtain better rendering effects in real-time rendering and improve the visual effects at the edge of moving objects and backgrounds, the moving foreground and static background in the current frame can be separated before interpolation, and the pixels in the current frame can be divided into a foreground layer belonging to moving objects and a background layer belonging to static backgrounds. Different processing strategies can be used for different layers after separation. For the foreground layer, prediction can be based on its motion vector; and for the background layer, the background layer appearing in the history can be filled. Then the predicted foreground layer and the filled background layer can be fused to obtain a predicted frame (i.e. an intermediate frame), which is inserted between two frames to improve the smoothness of the picture.

[0039] Using such a video processing method, by accurately dividing the foreground layer of moving objects and the background layer of static backgrounds, and using different ways to predict and fuse the two layers, the display effect of the edge where moving objects and backgrounds intersect can be effectively improved, and the boundary flaw problem can be better solved.

[0040] Embodiments of the present application provide a video processing method. Referring to Figure 1 , the video processing method 100 includes steps 110 to 140.

[0041] In step 110, for any to-be-processed frame in the to-be-processed video, based on motion state information of each pixel in the to-be-processed frame, a first pixel set and a second pixel set corresponding to the to-be-processed frame are determined from the pixels. The first pixel set includes a plurality of first pixels. Any first pixel in the plurality of first pixels has a first motion attribute. The second pixel set includes a plurality of second pixels. Any second pixel in the plurality of second pixels has a second motion attribute. The second motion attribute is different from the first motion attribute.

[0042] In step 120, based on motion vectors of each first pixel, pixel positions of the plurality of first pixels in the predicted frame are determined to obtain a first predicted pixel set corresponding to the predicted frame. The first predicted pixel set includes a plurality of first predicted pixels corresponding to the plurality of first pixels respectively. The plurality of first predicted pixels respectively have pixel positions of the corresponding first pixels in the predicted frame.

[0043] In step 130, based on the second pixel sets respectively corresponding to at least one to-be-processed frame before the to-be-processed frame in the to-be-processed video, a second predicted pixel set corresponding to the predicted frame is determined. The second predicted pixel set includes a plurality of second predicted pixels.

[0044] In step 140, based on the first predicted pixel set and the second predicted pixel set, the predicted frame is generated.

[0045] In embodiments of the present application, the to-be-processed video can be a video scene that needs to be rendered in real time, for example, a video in a game scene, etc. The to-be-processed frame can be understood as a current frame that needs to be processed for rendering. The predicted frame can also be referred to as an intermediate frame. Assuming that the current frame is the Nth frame, the predicted frame is an intermediate frame generated and inserted between the Nth frame and the N+1th frame.

[0046] In the to-be-processed video, there can be moving objects and background objects behind the moving objects. Taking an action game as an example, a character controlled by a player is running from the left side to the right side of the screen. The character is a moving object. Behind the character is a brick wall with a graffiti poster, which is a background object. In the video, only the character moves in the picture, and the brick wall of the background remains stationary.

[0047] In step 110, the foreground layer belonging to moving objects and the background layer belonging to static background are divided according to different motion state information, and the pixels in the current frame are divided into pixels belonging to moving objects and pixels belonging to static background. The first pixel set, i.e. the foreground layer, includes the first pixels belonging to moving objects. These first pixels have a movable motion attribute, i.e. the first motion attribute. The second pixel set, i.e. the background layer, includes the second pixels belonging to static background. These second pixels have a static motion attribute, i.e. the second motion attribute.

[0048] In some embodiments of the present application, step 110 includes determining a plurality of first pixels and a plurality of second pixels from a plurality of pixels according to velocity vectors corresponding to the plurality of pixels respectively.

[0049] In modern real-time rendering engines, classification information of different objects in a scene is usually provided. At the engine level, developers usually specify the motion attribute of objects in the scene, such as static or movable. In some embodiments, each pixel can be divided by accessing these classification labels provided by the engine. For example, in Unreal Engine, an encoded velocity can be used to determine whether an object to which a pixel belongs is a movable object or a static object. According to the velocity vector corresponding to each pixel, each pixel in the current frame can be classified and divided into the first pixel set (foreground layer) or the second pixel set (background layer). If a pixel belongs to a movable object, it is considered a first pixel and is divided into the first pixel set and information related to it (such as color, depth, motion vector, etc.) is stored. If a pixel belongs to a static object, it is considered a second pixel and is divided into the second pixel set and information related to it (such as color, depth, etc.) is stored. Through the corresponding velocity vector, it can be quickly identified whether a pixel in the current frame belongs to a moving object or a background region, the division of different layers is realized, and the computational overhead is reduced. After step 110, the picture of the current frame is split into two independent layers, the foreground layer only includes pixels of moving objects, and the background layer only includes pixels of static background. Due to the splitting of the picture, a "hole" corresponding to the moving object in the foreground layer will appear in the background layer.

[0050] Different processing is performed for the first pixel set and the second pixel set respectively. In step 120, for a first pixel in the first pixel set, in order to make the object move smoothly, a prediction can be made according to the motion vector of the first pixel. In one example, the motion vector of the first pixel can be read, and it is assumed that the object moves at a constant speed in a straight line between two frames. Therefore, in order to obtain a predicted frame which is the intermediate frame between the two frames, the first pixel can be moved halfway along the direction of its motion vector, and then a new position of the first pixel in the predicted frame can be predicted, i.e., a first predicted pixel corresponding to the first pixel in the predicted frame. By predicting each first pixel in the foreground layer, a first predicted pixel set can be obtained, i.e., the corresponding position of the foreground layer in the predicted frame.

[0051] As for the second pixels in the background layer, as described above, a "hole" corresponding to the moving object in the foreground layer can appear in the background layer. In the conventional method, the surrounding pixels are usually relied on to "guess" and fill in the hole. Even if an advanced AI repair technology is used, a better filling effect cannot be achieved, and visual defects such as flickering or unstable artifacts can easily occur. In the embodiments of the present application, the hole can be filled according to the background layer appearing in the past historical frames.

[0052] In some embodiments of the present application, with reference to Figure 2 , step 130 includes steps 210 to 230.

[0053] In step 210, a historical pixel set is generated based on the second pixel sets respectively corresponding to at least one to-be-processed frame located before the to-be-processed frame in the to-be-processed video. The historical pixel set includes the second pixels in the second pixel sets respectively corresponding to the at least one to-be-processed frame.

[0054] In step 220, a plurality of filling pixels are determined from the historical pixel set according to the second pixel set corresponding to the to-be-processed frame.

[0055] In step 230, a second predicted pixel set is determined based on the second pixel set corresponding to the to-be-processed frame and the plurality of filling pixels.

[0056] In the processing process, after each to-be-processed frame in the to-be-processed video is processed, the second pixels in the background layer corresponding to each to-be-processed frame can be accumulated to obtain a historical pixel set. The historical pixel set stores the second pixels included in each frame before the current frame. With the passage of time, the second pixels in the historical pixel set will gradually increase. For example, when a camera recording a video sweeps through a scene, as the second pixels in the historical pixel set gradually increase, the background layer will be gradually completed.

[0057] According to the historical pixel set, the holes in the background layer of the current frame can be filled. In step 220, the filling pixels can be understood as the pixels corresponding to the hole positions in the background layer of the current frame. Since the holes in the background layer are caused by the splitting of the foreground layer, the filling pixels can also be understood as the pixels corresponding to the positions of the foreground layer of the current frame. According to the historical pixels in the frames in the processing history, the second pixel set, i.e., the background layer of the current frame, can also be filled, effectively reducing the visual defects such as holes and artifacts in the background area caused by the movement of the moving object.

[0058] In some embodiments of the present application, with reference to Figure 3 , step 220 includes steps 310 to 330.

[0059] In step 310, at least one spatial coordinate displacement between at least one to-be-processed frame and the to-be-processed frame is determined.

[0060] In step 320, based on the at least one spatial coordinate displacement, the second pixels in the second pixel set corresponding to at least one to-be-processed frame in the historical pixel set are subjected to coordinate transformation to obtain the coordinate-transformed second pixels corresponding to the at least one to-be-processed frame.

[0061] In step 330, based on the second pixel set corresponding to the to-be-processed frame and the coordinate-transformed second pixels corresponding to the at least one to-be-processed frame, a plurality of filling pixels are determined from the historical pixel set.

[0062] Since the spatial coordinate system may change between the to-be-processed frames in the to-be-processed video due to camera movement and the like, when determining the filling pixels, the pixels in the historical pixel set can be subjected to coordinate transformation first, so that these pixels are converted into the spatial coordinate system of the current frame and can be aligned with the current frame. In step 310, the spatial coordinate displacement between the previous to-be-processed frame and the current frame can be determined according to the motion vector caused by the camera movement and the like. In step 320, the pixels in the historical pixel set are subjected to coordinate transformation so as to be aligned with the perspective of the current frame. In step 330, according to the hole positions on the background layer of the current frame, the filling pixels can be determined from the aligned second pixels in the historical pixel set, which are used to fill the holes in the background layer of the current frame.

[0063] In step 230, the filling pixels are used to fill the background layer, i.e., the second pixel set, of the current frame. For example, the related information of the filling pixels can be copied to the current frame for filling, i.e., a second predicted pixel set, i.e., the background layer in the predicted frame, is obtained. The background layer in the predicted frame not only includes the background layer with holes in the current frame, but also fills the holes according to the historical pixel set.

[0064] By transforming the pixels in the historical pixel set according to the spatial coordinate displacement between each frame in the processing history and the current frame, the pixels corresponding to each frame in the historical pixel set can be aligned with the viewpoint of the current frame, thereby finding the accurate fill pixels in the historical pixel set to fill the background layer.

[0065] In step 140, the first predicted pixel set (i.e., the foreground layer in the predicted frame) and the second predicted pixel set (i.e., the background layer in the predicted frame) can be merged to form a complete image. In one example, the second predicted pixel set can be drawn onto the final predicted frame output first, and then the first predicted pixel set can be superimposed on the second predicted pixel set. During superposition, the depth information of each pixel can be utilized. By comparing the depth values ​​of the first and second predicted pixels, the foreground and background occlusion relationships between objects can be accurately determined.

[0066] This video processing method splits the pixels in the current frame based on motion state information before frame interpolation, dividing them into a foreground layer and a background layer with different motion attributes. Targeted processing strategies are then applied to each layer, effectively reducing visual imperfections at the edges where the interpolated frames meet the moving object and background, thus improving the visual effect.

[0067] In some embodiments of this application, the video processing method 100 further includes a first process 400. (See reference...) Figure 4 The first process 400 includes steps 410 to 420.

[0068] Step 410: For any frame in the video to be processed: Based on the material information of each pixel in the frame to be processed, determine the third pixel set corresponding to the frame from the multiple pixels. The third pixel set includes multiple third pixels. Any one of the multiple third pixels has a predetermined material attribute.

[0069] Step 420: Based on multiple third pixels, determine the set of third predicted pixels corresponding to the predicted frame. The set of third predicted pixels includes multiple third predicted pixels that correspond to the multiple third pixels respectively.

[0070] In embodiments of this application, material information can be obtained, for example, through parameters such as material number and shading model information in the rendering engine's buffer.

[0071] In some real-time rendering scenarios, there can be semi-transparent objects or volume effects such as glass, smoke, fire, water, etc. For these semi-transparent objects, they can be divided into a third pixel set for separate processing. In an example, the predetermined material attribute can be a semi-transparent material attribute. It should be understood that the predetermined material attribute can be set according to the needs of video processing, and in other embodiments, if other specific material objects need to be processed separately, the predetermined material attribute can also be set as other material attributes. In this application, the semi-transparent material attribute is described.

[0072] For these third pixels, they can be predicted separately to obtain third predicted pixels of these third pixels in the predicted frame. In an example, these semi-transparent objects are moving objects, and then the processing manner for the first pixels in the first pixel set in step 120 can be referred to, for example, the motion vector of each third pixel can be read to predict the new position of it in the predicted frame, that is, the third predicted pixel. By predicting each third pixel included in the semi-transparent object, a third predicted pixel set, that is, the corresponding position of the semi-transparent object in the predicted frame, can be obtained. In another example, there can be moving objects and stationary objects in the semi-transparent object, and then the third pixel set can be further split according to the processing procedure of step 110 to obtain the foreground layer and the background layer corresponding to the semi-transparent object, and the processing manner for the first pixel set in step 120 and the processing manner for the second pixel set in step 130 are referred to to obtain the foreground layer and the background layer of the semi-transparent object in the predicted frame, respectively.

[0073] According to the material information to determine the third pixel set, the object with the predetermined material can be accurately identified and processed accordingly, so as to generate a more real and accurate predicted frame.

[0074] In some embodiments of the present application, step 140 comprises generating a predicted frame based on the first predicted pixel set, the second predicted pixel set and the third predicted pixel set.

[0075] After obtaining the first predicted pixel set, the second predicted pixel set corresponding to the opaque object and the third predicted pixel set corresponding to the semi-transparent object, they can be fused. In an example, an Alpha blending technique can be used to fuse the semi-transparent object with the background layer and / or the foreground layer of the opaque moving object according to the transparency value to obtain the final predicted frame.

[0076] Based on the pixel sets with different motion attributes and the pixel set with the predetermined material attribute to generate a predicted frame, complex visual effects can be correctly processed to generate a more real and accurate frame insertion result.

[0077] In some embodiments of the present application, the video processing method 100 further comprises: determining a plurality of second pixel subsets from the second pixel set according to the depth information of each second pixel and a predetermined depth threshold condition.

[0078] When processing the background layer of the current frame, the background layer can also be further divided into different sub-layers, i.e., different second pixel subsets, according to the depth information of each second pixel. The depths of the second pixels in each sub-layer respectively belong to a predetermined depth range. In some embodiments, the depth threshold condition can be set according to processing requirements, for example, it can be set according to the depth range of the second pixel. When the depth of a certain second pixel is between a first depth threshold and a second depth threshold, it is divided into a first sub-layer; when the depth of a certain second pixel is between the second depth threshold and a third depth threshold, it is divided into a second sub-layer, and so on. Further subdividing the background layer into multiple levels according to the depth of the pixel can more accurately process scenes with complex, multi-level static geometry, improve the accuracy of video rendering, and improve the rendering effect.

[0079] In some embodiments of the present application, referring to Figure 5 , step 130 comprises steps 510 to 530.

[0080] Step 510, determining a plurality of second pixel subsets corresponding to at least one to-be-processed frame respectively according to the depth information of each second pixel and a depth threshold condition.

[0081] Step 520, determining a plurality of second prediction pixel subsets corresponding to the prediction frame based on the plurality of second pixel subsets corresponding to at least one to-be-processed frame respectively.

[0082] Step 530, obtaining a second prediction pixel set based on the plurality of second prediction pixel subsets corresponding to the prediction frame.

[0083] In step 510, according to the depth threshold condition, each second pixel can be divided into different sub-layers. In step 520, for each sub-layer, the corresponding history pixel set and the corresponding filling pixel can be determined respectively with reference to steps 210 to 230 described above. When filling the hole position of the background layer, the sub-layer corresponding to the hole position can be determined according to the depth information around the hole, and the corresponding filling pixel in the history pixel set corresponding to the sub-layer can be determined to fill the hole, thereby obtaining the second prediction pixel subset corresponding to each sub-layer in the prediction frame. In step 530, after filling each sub-layer, the complete background layer in the prediction frame can be synthesized by layer-by-layer superimposition of these filled sub-layers from back to front (from the distant view to the close view) according to the depth information.

[0084] The background layer in the predicted frame is filled according to the layers with different depths in the background layer, and finally fused to obtain the background layer in the predicted frame, which can effectively reduce the error filling caused by the parallax effect, and make the background layer in the predicted frame have higher spatial correctness.

[0085] Based on the same technical concept, an embodiment of the present application provides a video processing device. The embodiment of the video processing device can refer to the embodiment of the video processing method, and the repeated parts will not be described herein. Referring to Figure 6 The video processing device 600 includes a determination module 610, a first prediction module 620, a second prediction module 630, and a generation module 640.

[0086] The determination module 610 is configured to, for any to-be-processed frame in the to-be-processed video: determine, based on motion state information of each of a plurality of pixels in the to-be-processed frame, a first pixel set and a second pixel set corresponding to the to-be-processed frame from the plurality of pixels. The first pixel set includes a plurality of first pixels. Any first pixel in the plurality of first pixels has a first motion attribute. The second pixel set includes a plurality of second pixels. Any second pixel in the plurality of second pixels has a second motion attribute. The second motion attribute is different from the first motion attribute.

[0087] The first prediction module 620 is configured to determine, according to a motion vector of each of the plurality of first pixels, a pixel position of each of the plurality of first pixels in a predicted frame, to obtain a first predicted pixel set corresponding to the predicted frame. The first predicted pixel set includes a plurality of first predicted pixels corresponding to the plurality of first pixels respectively. Each of the plurality of first predicted pixels has a pixel position of a corresponding first pixel in the predicted frame.

[0088] The second prediction module 630 is configured to determine, based on the second pixel set corresponding to at least one to-be-processed frame before the to-be-processed frame in the to-be-processed video, a second predicted pixel set corresponding to the predicted frame. The second predicted pixel set includes a plurality of second predicted pixels.

[0089] The generation module 640 is configured to generate the predicted frame based on the first predicted pixel set and the second predicted pixel set.

[0090] The determination module 610, the first prediction module 620, the second prediction module 630, and the generation module 640 in the video processing device 600 can correspond to steps 110 to 140 in the video processing method 100, and for the sake of brevity, will not be described herein. It should be understood that, corresponding to the embodiment of the video processing method 100, the embodiment of the video processing device 600 can also include more modules.

[0091] It should be noted that the functions of the various modules discussed herein can be split into multiple modules, and / or at least some of the functions of multiple modules can be combined into a single module. A particular module discussed herein performing an action includes that particular module itself performing the action, or alternatively, that particular module invoking or otherwise accessing another component or module that performs the action (or performs the portion of the action that is relevant for the particular module). Thus, a particular module performing an action includes that particular module itself performing the action and / or another module invoked or otherwise accessed by the particular module performing the action.

[0092] It should also be understood that various techniques described herein can be described in the general context of software hardware elements or program modules. The various software -based aspects of the technology described herein are stored in and / or transmitted over as one or more sets of instructions in a computer-readable medium. The software is implemented at least partially in software programs and is executed by one or more processing units. However, other hardware and / or software elements according to other configurations can also be used. Figure 6 The various modules described above can be implemented in hardware or in hardware combined with software and / or firmware. For example, the modules can be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, the modules can be implemented as hardware logic / circuitry. The hardware logic / circuitry can include an integrated circuit chip (which includes one or more components such as a processor (e.g., a Central Processing Unit (CPU), a microcontroller, a microprocessor, a Digital Signal Processor (DSP), etc.), a memory, one or more communication interfaces, and / or other circuitry), and can optionally execute received program code and / or include embedded firmware to execute functionality.

[0093] Embodiments of the present application provide a computing device 700, as shown in Figure 7 Figure 7 An example configuration of a computing device 700 that can be used to implement the video processing method 100 described herein is shown. For example, the video processing apparatus 600 described above can be implemented in whole or at least in part by the computing device 700 or a similar device or system.

[0094] The computing device 700 can include at least one processor 705, memory 707, communication interface(s) 702, display device 701, other input / output (I / O) devices 703, and one or more mass storage devices 706, which are able to communicate with each other by way of a bus 704 or other appropriate connection. The memory 707 has instructions stored thereon which, when executed by the processor 705, cause the processor 705 to perform the video processing method as in the above-described embodiments.

[0095] ​The computing device 700 can be various different types of devices. Examples of the computing device 700 include, but are not limited to: a desktop computer, a server computer, a laptop or notebook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smart phone), a notepad computer, a mobile station), a wearable device (e.g., glasses, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box, a game console), a television or other display device, an automobile computer, etc.

[0096] The processor 705 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing units or multiple cores. The processor 705 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 705 can be configured to fetch and execute computer-readable instructions stored in the memory 707, the mass storage device 706, or any other computer-readable medium.

[0097] The memory 707 and the mass storage device 706 are examples of computer-readable storage media for storing instructions that are executed by the processor 705 to cause the various functionalities described above. For example, the memory 707 can generally include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 706 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD or DVD), storage arrays, network attached storage, storage area networks, etc. The memory 707 and the mass storage device 706 can both be collectively referred to herein as the memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processor 705 as a specific machine configured to implement the operations and functions described in examples herein.

[0098] A number of programs can be stored on the mass storage device 706. These programs include an operating system 708, one or more application programs 709, other programs 710, and program data 711, and they can be loaded into the memory 707 for execution. Examples of such application programs or program modules can include, for example, computer program logic (e.g., computer program code or instructions) for implementing the video processing apparatus 600 (including the determining module 610, the first prediction module 620, the second prediction module 630, and the generating module 640), the video processing method 100 (including any suitable steps of the video processing method 100), and / or the additional embodiments described herein, for example.

[0099] Although illustrated in Figure 7 the memory 707 of the computing device 700, the operating system 708, the application programs 709, the other programs 710, and the program data 711, or portions thereof, can be implemented using any form of computer-readable media that is accessible by the computing device 700.

[0100] One or more communication interfaces 702 are used to exchange data with other devices, such as over a network, a direct connection, or the like. Such communication interfaces can be one or more of any type of network interface (for example, network interface cards (NICs)), wired or wireless (such as, for example, IEEE 802.11 wireless LAN (WLAN) wireless interfaces), Worldwide Interoperability for Microwave Access (Wi-MAX) interfaces, Ethernet interfaces, Universal Serial Bus (USB) interfaces, cellular network interfaces, Bluetooth™ interfaces, near field communication (NFC) interfaces, or the like. The communication interfaces 702 can facilitate communications within a variety of networks and protocol types including wired networks (for example, LAN, cable, or the like), and wireless networks (for example, WLAN, cellular, satellite, or the like), the Internet, and the like. The communication interfaces 702 can also provide communications with external storage devices (not shown), such as storage arrays, network attached storage, storage area networks, or the like.

[0101] In some examples, a display device 701, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 703 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and the like.

[0102] The techniques described herein can be supported by these various configurations of the computing device 700 and are not limited to the specific examples carried through the preceding description of the techniques. For example, the functionality can also be implemented through the use of dedicated hardware as well as, or instead of, being implemented with instructions that cause one or more processors to perform the functionality. The implementation can be realized using a combination of hardware and software; one or more processors operating in conjunction with instructions that can be embedded in firmware, one or more ASICs, FPGAs, or other hardware-only state machines, without necessarily having software components to execute on the processors or state machines. In these and other implementations, the functionality described herein can be updated, augmented, modified or otherwise improved upon by the platform abstraction. For example, the functionality can be implemented by using a distributed system in which the functionality of the cloud is abstracted. The cloud includes and / or is representative of a platform for resources. The platform abstracts the underlying functionality of hardware (e.g., servers) and software resources of the cloud. The resources can include applications and / or data that can be utilized while performing a computation process on a server that is remote from the computing device 700. The resources can also include services provided over the Internet and / or over a subscriber network such as a cellular or Wi-Fi network. The platform can abstract the resources and functionality to connect the computing device 700 with other computer devices. Consequently, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partially on the computing device 700 and partially by the platform that abstracts the functionality of the cloud.

[0103] The embodiments of the present application also provide a computer readable storage medium having instructions stored thereon, which when executed by one or more processors of a computing device, alone or in combination, cause the computing device to perform the method in any of the embodiments described above.

[0104] A computer readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The computer readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computer device.

[0105] The embodiments of the present application also provide a computer program product including instructions, which when executed by one or more processors of a computing device, alone or in combination, cause the computing device to perform the method in any of the embodiments described above.

[0106] One particular embodiment of the present application is described below. It should be understood that this particular embodiment is described for illustrative purposes only and should not be construed as limiting the present application.

[0107] In one game, a player-controlled character is running from the left to the right of the screen. The character is backed by a brick wall with a poster of graffiti on it. The camera is stationary, with only the character moving in the frame. This embodiment aims to generate a predicted frame between the Nth frame and the N+1th frame. Currently, the rendering engine has generated a full image of the Nth frame and its corresponding geometry buffer (G-Buffer) data (e.g., containing encoded velocity vectors and other information). In the frame, the character's body occludes part of the poster on the wall.

[0108] By analyzing the G-Buffer of the Nth frame, for the pixels belonging to the character model, they are identified as "moving pixels", i.e. first pixels, according to their motion attributes. These first pixels (including information such as color, depth and motion vector) are extracted to form a foreground layer containing only character pixels, with the rest being transparent, i.e. the first pixel set. Correspondingly, for the pixels belonging to the background brick wall and poster, they are identified as "static pixels", i.e. second pixels. These second pixels (including information such as color and depth) form a background layer, i.e. the second pixel set. Since the character pixels are removed, a hole with the same shape as the character outline is left on the background layer, and a part of the hole is located in the area where the poster should be displayed.

[0109] For the foreground layer, each first pixel is moved half the distance along its motion vector direction according to its motion vector, so as to calculate the accurate position and pose of the character at the middle time between the two frames (i.e. the prediction frame), to obtain an "interpolated foreground layer", i.e. the foreground layer in the prediction frame, i.e. the first prediction pixel set.

[0110] For the background layer, it can be filled by the long-term maintained history pixel set. The history pixel set contains static backgrounds that have appeared in historical frames. Suppose that the camera has once moved to the left at a certain time in the past, then the history pixel set may have recorded the complete appearance of the poster that is currently blocked by the character. The second pixels in the history pixel set are coordinate-transformed to align with the coordinate space of the current frame. For the background layer hole position of the current frame, since the history pixel set saves the complete historical information of the poster, the corresponding and real filling pixels can be found and copied from the history pixel set and filled into the hole to obtain the background layer in the prediction frame, i.e. the second prediction pixel set. In the background layer of the prediction frame, the background that was previously blocked by the character and should now be exposed is filled.

[0111] The background layer of the prediction frame is drawn onto the final output canvas, and the foreground layer of the prediction frame is superimposed on the background layer, to finally output a frame of prediction frame. In the prediction frame, the character smoothly moves to the corresponding position, and the background wall and poster behind the character are clear and complete, effectively improving the visual effect of the character edge.

[0112] Using such a video processing method, the output picture maintains very high definition and sharpness at the edge of the moving object. The filled background area better solves the common problems of blur, repetition or distortion in traditional repair methods. Since the filled content is derived from the real rendering sequence, the exposed background area is relatively stable and natural when playing continuously, and visual defects such as flicker, jitter or artifacts are rarely seen. Compared with models relying on large neural networks, the video processing method in the present application requires less resources and lower computing cost, can effectively reduce the computing cost, and has good predictability and debuggability and high robustness.

[0113] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application. Especially, as long as there is no structural conflict, the technical features mentioned in each embodiment can be combined in any way. The present application is not limited to the specific embodiments disclosed in the text, but includes all technical solutions falling within the scope of the claims.

Claims

1. A video processing method, characterized in that, include: For any frame in the video to be processed: Based on the motion state information of each of the multiple pixels of the frame to be processed, a first pixel set and a second pixel set corresponding to the frame to be processed are determined from the multiple pixels. The first pixel set includes multiple first pixels, and any one of the multiple first pixels has a first motion attribute. The second pixel set includes multiple second pixels, and any one of the multiple second pixels has a second motion attribute. The second motion attribute is different from the first motion attribute. Based on the motion vectors of the plurality of first pixels, the pixel positions of the plurality of first pixels in the prediction frame are determined to obtain the first prediction pixel set corresponding to the prediction frame. The first prediction pixel set includes a plurality of first prediction pixels corresponding to the plurality of first pixels, and the plurality of first prediction pixels each have a corresponding pixel position of the first pixel in the prediction frame. Based on the second pixel sets corresponding to at least one frame in the video preceding the frame to be processed, a second predicted pixel set corresponding to the predicted frame is determined, wherein the second predicted pixel set includes a plurality of second predicted pixels; and The prediction frame is generated based on the first predicted pixel set and the second predicted pixel set.

2. The video processing method according to claim 1, characterized in that, Determining the first pixel set and the second pixel set corresponding to the frame to be processed from the plurality of pixels includes: Based on the velocity vectors corresponding to the plurality of pixels, a plurality of first pixels and a plurality of second pixels are determined from the plurality of pixels.

3. The video processing method according to claim 1, characterized in that, The step of determining the second predicted pixel set corresponding to the predicted frame based on the second pixel sets corresponding to at least one frame in the video preceding the frame to be processed includes: Based on the second pixel set corresponding to at least one frame in the video to be processed that is preceding the frame to be processed, a historical pixel set is generated, wherein the historical pixel set includes the second pixels in the second pixel set corresponding to the at least one frame to be processed. Based on the second pixel set corresponding to the frame to be processed, a plurality of fill pixels are determined from the historical pixel set; and The second predicted pixel set is determined based on the second pixel set corresponding to the frame to be processed and the plurality of padding pixels.

4. The video processing method according to claim 3, characterized in that, The step of determining multiple fill pixels from the historical pixel set based on the second pixel set corresponding to the frame to be processed includes: Determine at least one spatial coordinate displacement between the at least one frame to be processed and the frame to be processed; Based on the at least one spatial coordinate displacement, coordinate transformation is performed on the second pixels in the second pixel set corresponding to the at least one frame to be processed in the historical pixel set, to obtain the coordinate-transformed second pixels corresponding to the at least one frame to be processed; and Based on the second pixel set corresponding to the frame to be processed and the second pixel after coordinate transformation corresponding to the at least one frame to be processed, the plurality of filling pixels are determined from the historical pixel set.

5. The video processing method according to any one of claims 1-4, characterized in that, The video processing method further includes: For any frame to be processed in the video to be processed: Based on the material information of each pixel in the frame to be processed, a third pixel set corresponding to the frame to be processed is determined from the plurality of pixels, wherein the third pixel set includes a plurality of third pixels, and any one of the plurality of third pixels has a predetermined material attribute; and Based on the plurality of third pixels, a set of third predicted pixels corresponding to the predicted frame is determined, wherein the set of third predicted pixels includes a plurality of third predicted pixels that correspond to the plurality of third pixels respectively.

6. The video processing method according to claim 5, characterized in that, Generating the prediction frame based on the first predicted pixel set and the second predicted pixel set includes: The prediction frame is generated based on the first predicted pixel set, the second predicted pixel set, and the third predicted pixel set.

7. The video processing method according to any one of claims 1-4, characterized in that, The video processing method further includes: Based on the depth information of each of the plurality of second pixels and a predetermined depth threshold condition, a plurality of subsets of second pixels are determined from the set of second pixels.

8. The video processing method according to claim 7, characterized in that, The step of determining the second predicted pixel set corresponding to the predicted frame based on the second pixel sets corresponding to at least one frame in the video preceding the frame to be processed includes: Based on the depth information of each of the plurality of second pixels and the depth threshold condition, determine the plurality of second pixel subsets corresponding to the at least one frame to be processed; Based on the plurality of second pixel subsets corresponding to the at least one frame to be processed, determine the plurality of second predicted pixel subsets corresponding to the predicted frame; and The second predicted pixel set is obtained based on multiple subsets of second predicted pixels corresponding to the predicted frame.

9. A video processing apparatus, characterized in that, include: The determining module is configured to, for any frame to be processed in the video to be processed: based on the motion state information of each of the multiple pixels of the frame to be processed, determine a first pixel set and a second pixel set corresponding to the frame to be processed from the multiple pixels, wherein the first pixel set includes multiple first pixels, any one of the multiple first pixels has a first motion attribute, and the second pixel set includes multiple second pixels, any one of the multiple second pixels has a second motion attribute, wherein the second motion attribute is different from the first motion attribute; A first prediction module is configured to determine the pixel positions of the plurality of first pixels in a prediction frame based on the motion vectors of the plurality of first pixels, so as to obtain a first prediction pixel set corresponding to the prediction frame. The first prediction pixel set includes a plurality of first prediction pixels corresponding to the plurality of first pixels, and the plurality of first prediction pixels each have a corresponding pixel position of the first pixel in the prediction frame. The second prediction module is configured to determine a second prediction pixel set corresponding to the prediction frame based on the second pixel sets corresponding to at least one frame in the video preceding the frame to be processed, wherein the second prediction pixel set includes a plurality of second prediction pixels; and A generation module is used to generate the prediction frame based on the first prediction pixel set and the second prediction pixel set.

10. A computing device, comprising: At least one processor; as well as At least one memory communicatively connected to the at least one processor, the at least one memory storing instructions that, when executed individually or jointly by the at least one processor, cause the computing device to perform the video processing method of any one of claims 1 to 8.

11. A computer-readable storage medium storing instructions that, when executed individually or jointly by one or more processors of a computing device, cause the computing device to perform the video processing method of any one of claims 1 to 8.

12. A computer program product comprising instructions that, when executed individually or jointly by one or more processors of a computing device, cause the computing device to perform the video processing method of any one of claims 1 to 8.

Images