Image processing method and device and storage medium

Abstract

The invention discloses an image processing method and device and a storage medium, which can improve the picture quality of a prediction frame. The method comprises the following steps: according to motion vector information between a first real frame and a second real frame, carrying out at least one round of distortion processing on a color image of the first real frame to obtain a first target color image of a prediction frame; wherein each round of distortion processing comprises the following steps: generating a full-screen grid layer covering the first real frame; according to the motion vector information, carrying out distortion processing on at least part of grids in the full-screen grid layer; according to the distorted full-screen grid layer and the color image of the first real frame, outputting a color image corresponding to each round of distortion processing; and filling a blank area in the first target color image of the prediction frame to obtain a second target color image of the prediction frame, the second target color image being a display image of the prediction frame. The color image of the prediction frame is obtained by adopting a mode of twisting the full-screen grid layer, so that the accuracy degree of the prediction frame can be improved.

Description

Method, device and storage medium for image processing

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 26, 2022, with application number 202211319597.7 and application name “Method, device and storage medium for image processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of information technology, and in particular to a method, device, and storage medium for image processing. Background Art

[0003] With the rapid development of image processing technology, the screen refresh rate and rendering resolution of video playback are gradually increasing in some scenarios with high video quality requirements, such as game interfaces, virtual reality (VR) interfaces, and augmented reality (AR) interfaces. For example, the screen refresh rate has increased from 60Hz to 120Hz or even higher, and the rendering resolution has increased from 720p to 2k or even higher. This poses greater challenges to the performance and power consumption of electronic devices. To reduce power consumption, frame prediction technology has been applied to the video data processing process.

[0004] Frame prediction technology predicts and inserts predicted frames based on data such as the motion vector of the real frame. This insertion can improve the frame rate of the displayed image, reduce lag, and lower video rendering power consumption. During the predicted frame generation process, since there are many similarities between the two real frames, the motion vector of the pixel between the two real frames can be calculated by calculating the similarity between the pixels. The pixel's position in the predicted frame is predicted based on the motion vector, and the pixel is placed in the corresponding position. For pixel areas where a match cannot be determined between the two real frames, a pixel filling algorithm is used to predict the image data to fill the pixel area. Video frames have different image information distribution characteristics due to different scenes, environments, and visual styles. Therefore, using a fixed predicted frame generation scheme may result in poor image quality of the predicted frames.

[0005] In order to solve the above problems, the industry is focusing on researching a method that can improve the picture quality and accuracy of the predicted frame while saving power consumption and computing power.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a method, apparatus, and storage medium for image processing, which can improve the image quality of generated prediction frames.

[0008] In a first aspect, an embodiment of the present application proposes a method for image processing, the method comprising: performing at least one round of warping processing on the color image of the first real frame according to motion vector information between the first real frame and the second real frame to obtain a first target color image of the predicted frame, the first target color image of the predicted frame including a blank area caused by the at least one round of warping processing; wherein each round of warping processing in the at least one round of warping processing comprises: generating a full-screen grid layer covering the first real frame, the full-screen grid layer including a plurality of grids, each grid including a plurality of pixels; performing warping processing on at least part of the grids in the full-screen grid layer according to the motion vector information; outputting a color image corresponding to each round of warping processing according to the warped full-screen grid layer and the color image of the first real frame; filling the blank area in the first target color image of the predicted frame to obtain the second target color image of the predicted frame, the second target color image being the display image of the predicted frame.

[0009] In this way, during the predicted frame generation process, a full-screen grid layer is generated that covers the first real frame. Based on the motion vector between the two real frames, one or more rounds of warping of the full-screen grid layer are performed to obtain a color image that includes blank areas. By warping the full-screen grid layer to obtain the color image of the predicted frame, rather than performing pixel-by-pixel shifting, computing resources can be streamlined and the accuracy of the predicted frame can be improved.

[0010] In combination with the first aspect, in one possible implementation, the at least one round of distortion processing includes multiple rounds of distortion processing, and the multiple rounds of distortion processing correspond to multiple distortion types respectively, and the multiple distortion types include at least one of the following: a distortion operation performed on a static object; a distortion operation performed on a dynamic object; a distortion operation performed on a static object in a distant view; a distortion operation performed on a dynamic object in a distant view; and a distortion operation performed on a static object in a close view.

[0011] In this way, in each round of distortion processing, by distinguishing whether the object processed in each round belongs to a static object, a dynamic object, a distant image or a close-up image, different types of objects in the image are distorted, which can improve the accuracy of the final color image and further improve the image quality.

[0012] With reference to the first aspect, in a possible implementation, the at least one round of warping includes multiple rounds of warping, and resolutions of full-screen grid layers corresponding to at least two rounds of warping in the multiple rounds of warping are different.

[0013] In this way, the resolution of the full-screen grid layer can be flexibly selected according to the object of each round of distortion processing, so as to save computing power and power consumption and improve the computational efficiency of the prediction frame.

[0014] In combination with the first aspect, in a possible implementation, the method further includes: when the object processed in the i-th round in the at least one round of distortion processing is a static object in the distant view, setting the resolution of the full-screen grid layer used in the i-th round of processing to a first resolution, where i is an integer greater than or equal to 1; when the object processed in the i-th round in the at least one round of distortion processing is a static object or a dynamic object in the near view, setting the resolution of the full-screen grid layer used in the i-th round of processing to a second resolution, and the first resolution is lower than the second resolution.

[0015] This way, for static objects in the distance, since the amount of change between the two video frames is small, a lower-resolution full-screen grid layer can be used for the warping operation. However, since dynamic objects are in motion, the close-up image is closer to the camera and is more affected by the prediction error of the motion vector. Therefore, a higher-resolution grid warping is required to obtain a more accurate predicted image. By refining the objects for each round of warping operation and flexibly selecting the resolution of the full-screen grid layer based on the objects being warped, computing power and power consumption can be saved, and the computational efficiency of the predicted frames can be improved.

[0016] In combination with the first aspect, in a possible implementation, the at least one round of distortion processing includes multiple rounds of distortion processing, and the first round of distortion processing in the multiple rounds of distortion processing includes: generating a first full-screen grid layer covering the first real frame; offsetting the grid vertices in the first full-screen grid layer according to the motion vector information of the static object to obtain the distorted first full-screen grid layer; and outputting the pixels of the static object in the color image of the first real frame to the blank image according to the distorted first full-screen grid layer to obtain the first color image.

[0017] In combination with the first aspect, in a possible implementation, the outputting the pixels of the static object in the color image of the first real frame to the blank image to obtain the first color image includes: judging whether the first pixel belongs to a static object or a dynamic object based on the full-screen semantic image of the first real frame, the first pixel being any pixel in the color image of the first real frame; in the case that the first pixel belongs to a static object, outputting the first pixel to a first position in the blank image to obtain the first color image, the first position being the corresponding position of the first pixel in the first full-screen grid layer after the distortion; in the case that the first pixel belongs to a dynamic object, not outputting the first pixel to the blank image.

[0018] In combination with the first aspect, in a possible implementation method, the second round of distortion processing in the multiple rounds of distortion processing includes: generating a second full-screen grid layer covering the first real frame; offsetting the grid vertices in the second full-screen grid according to the motion vector information of the dynamic object to obtain a distorted second full-screen grid layer; based on the distorted second full-screen grid layer, outputting the pixels of the dynamic object in the color image of the first real frame to the first color image to obtain the first target color image.

[0019] In combination with the first aspect, in a possible implementation method, the pixels of the dynamic object in the color image of the first real frame are output to the first color image according to the distorted second full-screen grid layer to obtain the first target color image, including: judging whether the second pixel belongs to a static object or a dynamic object according to the full-screen semantic image of the first real frame, the second pixel being any pixel in the color image of the first real frame; in the case that the second pixel belongs to a dynamic object, the second pixel is output to a second position in the first color image to obtain the first target color image, the second position being the corresponding position of the second pixel in the distorted second full-screen grid layer; in the case that the second pixel belongs to a static object, the second pixel is not output to the first color image.

[0020] In combination with the first aspect, in a possible implementation, the at least one round of distortion processing includes multiple rounds of distortion processing, and the first round of distortion processing in the multiple rounds of distortion processing includes: generating a first full-screen grid layer covering the first real frame; offsetting the grid vertices in the first full-screen grid layer according to the motion vector information of the static object in the distance to obtain the distorted first full-screen grid layer; based on the distorted first full-screen grid layer, outputting the pixels of the static object in the distance in the color image of the first real frame to the blank image to obtain the first color image.

[0021] In combination with the first aspect, in a possible implementation, the pixels of static objects in the distance in the color image of the first real frame are output to a blank image based on the distorted first full-screen grid layer to obtain a first color image, including: judging whether a first pixel belongs to a static object or a dynamic object based on the full-screen semantic image of the first real frame, where the first pixel is any pixel in the color image of the first real frame; when the first pixel belongs to a static object, determining whether the depth of the first pixel is greater than a preset depth threshold based on the scene depth map of the first real frame; when the depth of the first pixel is greater than the preset depth threshold D, outputting the first pixel to a first position in the blank image to obtain the first color image, where the first position is the corresponding position of the first pixel in the distorted first full-screen grid layer; when the depth of the first pixel is less than the preset depth threshold D, not outputting the first pixel to the blank image; when the first pixel belongs to a dynamic object, not outputting the first pixel to the blank image.

[0022] In combination with the first aspect, in a possible implementation method, the second round of distortion processing in the multiple rounds of distortion processing includes: generating a second full-screen grid layer covering the first real frame; offsetting the grid vertices in the second full-screen grid layer according to the motion vector information of the static object in the foreground to obtain a distorted second full-screen grid layer; based on the distorted second full-screen grid layer, outputting the pixels of the static object in the foreground in the color image of the first real frame to the first color image to obtain a second color image.

[0023] In combination with the first aspect, in a possible implementation, the third round of distortion processing in the multiple rounds of distortion processing includes: generating a third full-screen grid layer covering the first real frame; offsetting the grid vertices in the third full-screen grid layer according to the motion vector information of the dynamic object in the distance to obtain a distorted third full-screen grid layer; and outputting the pixels of the dynamic object in the color image of the first real frame to the second color image according to the distorted third full-screen grid layer to obtain a third color image.

[0024] In combination with the first aspect, in a possible implementation method, the fourth round of distortion processing in the multiple rounds of distortion processing includes: generating a fourth full-screen grid layer covering the first real frame; offsetting the grid vertices in the fourth full-screen grid layer according to the motion vector information of the dynamic object in the foreground to obtain a distorted fourth full-screen grid layer; and outputting the pixels of the dynamic object in the color image of the first real frame to the third color image according to the distorted fourth full-screen grid layer to obtain the first target color image.

[0025] In combination with the first aspect, in a possible implementation, filling the blank area in the first target color image of the predicted frame to obtain the second target color image of the predicted frame includes: determining the threshold range to which the camera angle between the first real frame and the second real frame belongs; selecting a target pixel filling algorithm from at least two candidate pixel filling algorithms based on the threshold range to which the camera angle belongs; and filling the blank area in the first target color image according to the target pixel filling algorithm to obtain the second target color image.

[0026] Thus, when filling blank areas in the first target color image, a target pixel filling algorithm can be selected from multiple candidate pixel filling algorithms based on the threshold range of the camera angle to fill the blank areas. Because different pixel filling algorithms are suitable for different scenarios, flexibly selecting a pixel filling algorithm for blank area filling based on the camera angle threshold can improve the prediction accuracy and image quality of the displayed image of the predicted frame.

[0027] In combination with the first aspect, in a possible implementation, the target pixel filling algorithm is selected from at least two candidate pixel filling algorithms based on the threshold range to which the camera angle belongs, including: when the camera angle is less than a first angle threshold, selecting a first pixel filling algorithm as the target pixel filling algorithm, wherein the first pixel filling algorithm achieves pixel filling by sampling pixels of static objects around the pixels to be filled; when the camera angle is greater than the first angle threshold, selecting a second pixel filling algorithm as the target pixel filling algorithm, wherein the second pixel filling algorithm achieves pixel filling by sampling pixels of static objects and dynamic objects around the pixels to be filled.

[0028] In a second aspect, an embodiment of the present application provides a method for image processing, comprising: obtaining motion vector information between a first real frame and a second real frame; warping the color image of the first real frame according to the motion vector information to obtain a first target color image of a predicted frame, wherein the warping comprises: generating a full-screen grid layer covering the first real frame, the full-screen grid layer comprising a plurality of grids, each grid comprising a plurality of pixels; warping the grids in the full-screen grid layer according to the motion vector information; and obtaining a warped color image based on the warped full-screen grid layer and the color image of the first real frame.

[0029] In combination with the second aspect, in one possible implementation, the distortion processing includes multiple rounds of distortion processing, and the multiple rounds of distortion processing correspond to multiple types of distortion, respectively. The multiple types of distortion include at least one of the following: a distortion operation performed on a static object; a distortion operation performed on a dynamic object; a distortion operation performed on a static object in a distant view; a distortion operation performed on a dynamic object in a distant view; a distortion operation performed on a static object in a close view.

[0030] With reference to the second aspect, in a possible implementation, the warping process includes multiple rounds of warping processes, and resolutions of full-screen grid layers corresponding to at least two rounds of warping processes are different.

[0031] In combination with the second aspect, in a possible implementation, the method further includes: when the object processed in the i-th round in the multiple rounds of distortion processing is a static object in the distant view, setting the resolution of the full-screen grid layer used in the i-th round of processing to a first resolution, where i is an integer greater than or equal to 1; when the object processed in the i-th round in the at least one round of distortion processing is a static object or a dynamic object in the near view, setting the resolution of the full-screen grid layer used in the i-th round of processing to a second resolution, and the first resolution is lower than the second resolution.

[0032] In combination with the second aspect, in a possible implementation, the method further includes: filling the blank area in the first target color image of the predicted frame to obtain a second target color image of the predicted frame, wherein the second target color image is the display image of the predicted frame.

[0033] In a third aspect, an embodiment of the present application provides an apparatus for image processing, which comprises: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect or the second aspect.

[0034] In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first or second aspect.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, it enables the computer to execute the method of the first aspect or the second aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method described in the first aspect or the second aspect.

[0037] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram of the principle of generating a prediction frame according to an embodiment of the present application;

[0039] FIG2 is a software architecture block diagram of an apparatus 100 for image processing according to an embodiment of the present application;

[0040] FIG3 is a flow chart of a method for image processing in an embodiment of the present application;

[0041] FIG4 is a schematic diagram of a distortion process performed on a color image of a first real frame according to an embodiment of the present application;

[0042] FIG5 is a schematic diagram of a distortion process for a color image of a first real frame according to another embodiment of the present application;

[0043] FIG6 is a schematic flow chart of a first pixel filling algorithm according to an embodiment of the present application;

[0044] FIG7 is a schematic flow chart of a second pixel filling algorithm according to an embodiment of the present application;

[0045] FIG8 is a schematic diagram of a flow chart of a rendering process according to an embodiment of the present application;

[0046] FIG9 is a schematic diagram of the hardware structure of an apparatus 900 for image processing according to an embodiment of the present application;

[0047] FIG10 is a schematic structural diagram of an apparatus 1000 for image processing according to an embodiment of the present application;

[0048] FIG11 is a schematic diagram of the hardware structure of an apparatus 1100 for image processing according to another embodiment of the present application. DETAILED DESCRIPTION

[0049] To facilitate a clear description of the technical solutions of the embodiments of this application, the words "exemplary" or "for example" are used in the embodiments of this application to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0050] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0051] It should be noted that the "at..." in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after the situation occurs, and the embodiments of the present application do not specifically limit this. In addition, the display interface provided in the embodiments of the present application is only an example, and the display interface can also include more or less content.

[0052] FIG1 is a schematic diagram illustrating the principle of generating a predicted frame according to an embodiment of the present application. As shown in FIG1 , a predicted frame may refer to a predicted image generated based on information such as motion vectors of two real frames. The predicted frame may be inserted between, before, or after two real frames, and this embodiment of the present application does not limit this. The two real frames mentioned above may generally refer to two consecutive real frames. FIG1 illustrates the two real frames as a first real frame and a second real frame.

[0053] Before generating a predicted frame, it is necessary to first obtain relevant information about the two real frames, such as the color image of the real frame, the motion vector diagram between the two real frames, the scene depth information of the real frame, the camera angle between the two real frames, etc. This information can be obtained during the rendering process of the two real frames, for example, through the rendering instruction stream.

[0054] Based on the motion vector map between the two real frames, the positions of the pixels of the color image of the real frame in the predicted frame can be predicted, and then the pixels are placed at the corresponding positions in the predicted frame to obtain the first target color image of the predicted frame. In the process of moving pixels according to the motion vector map, some blank areas may appear where no matching relationship can be found between the two real frames. For these blank areas, the pixels to fill the blank areas can be calculated according to the pixel filling algorithm to obtain the color image of the predicted frame that is finally displayed. Among them, the pixel filling algorithm refers to a process of restoring and reconstructing the pixels inside the damaged area of ​​the image. By way of example and not limitation, the pixel filling algorithm may include an image inpainting algorithm, etc.

[0055] FIG2 is a block diagram of the software architecture of an apparatus 100 for image processing according to an embodiment of the present application. The layered architecture divides the software system of the apparatus 100 into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, namely, the application layer (applications), the application framework layer (application framework), the system library, and the hardware layer. It should be understood that FIG2 only shows the functional modules related to the method for image processing according to an embodiment of the present application. In practice, the apparatus 100 may include more or fewer functional modules, or some functional modules may be replaced by other functional modules.

[0056] The application layer can include a series of application packages. As shown in Figure 2, the application layer includes game applications, AR applications, VR applications, etc.

[0057] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. As shown in Figure 2, the application framework layer can include an application framework. The application framework (Framework) encapsulates numerous callable functional modules and API interfaces for the application layer.

[0058] As shown in Figure 2, the system library may include a rendering module, which can be used to implement 3D graphics drawing, image rendering, compositing, and layer processing. By way of example and not limitation, the rendering module includes, but is not limited to, at least one of the following: open graphics library (OpenGL), open source computer vision library (OpenCV), and open computing language library (OpenCL).

[0059] The hardware layer includes memory and graphics processing units (GPUs). Memory can be used to temporarily store computational data in the central processing unit (CPU) and exchange data with external storage devices such as hard drives.

[0060] The GPU is a processor that performs image and graphics related operations. In the embodiment of the present application, the process of the device 100 using the rendering module to perform graphics processing can be completed by the GPU.

[0061] When an application (such as a game) needs to play a video, it can send a rendering instruction to the application framework to send a video frame. The application framework calls the rendering module to render the image. During the rendering process, the rendering module can use the GPU to render the image and obtain the rendered screen data, which can be stored in the memory. When an image needs to be displayed, the application framework reads the image data from the memory and sends it to the cache area of ​​the display screen to display the image.

[0062] Optionally, in the embodiments of the present application, the device 100 may refer to an electronic device with image processing capabilities. As examples, the device 100 may include, but is not limited to, a mobile phone, a tablet computer, a PDA, a laptop computer, a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, etc., and the embodiments of the present application are not limited thereto.

[0063] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0064] As mentioned above, in order to improve the display image quality problem of generating a predicted frame, an embodiment of the present application provides a method and apparatus for image processing, in which a full-screen grid layer covering the real frame is generated during the generation of the predicted frame, and based on the motion vector between the two real frames, a color image including a blank area is obtained by one or more rounds of warping of the full-screen grid layer. By obtaining the color image of the predicted frame by warping the full-screen grid layer, the accuracy of the predicted frame can be improved. Furthermore, in each round of warping, different types of objects in the image can be warped, and the image of the predicted frame can be obtained based on the results of multiple warping, thereby further improving the accuracy and image quality of the predicted frame.

[0065] FIG3 is a flow chart of an image processing method according to an embodiment of the present application. The method can be performed by the apparatus 100 in FIG2 . As shown in FIG3 , the method includes the following contents.

[0066] S301. Based on motion vector information between the first real frame and the second real frame, perform at least one round of warping processing on the color image of the first real frame to obtain a first target color image of the predicted frame, where the first target color image of the predicted frame includes a blank area caused by at least one round of warping processing.

[0067] Among them, each round of distortion processing in at least one round of distortion processing includes: generating a full-screen grid layer covering the first real frame, the full-screen grid layer includes multiple grids, and each grid includes multiple pixels; according to motion vector information, at least part of the grids in the full-screen grid layer are distorted; according to the distorted full-screen grid layer and the color image of the first real frame, the color image corresponding to each round of distortion processing is output.

[0068] Optionally, the first real frame and the second real frame may include continuous real frames or discontinuous real frames. The predicted frame may be inserted between, before or after the first real frame and the second real frame.

[0069] Optionally, the first real frame may refer to a preceding frame image or a succeeding frame image of the two real frames.

[0070] In some examples, the motion vector information between the first real frame and the second real frame can be obtained during the rendering process of the first real frame and the second real frame. As an example, the motion vector information can include different types. For example, the motion vector information can include motion vector information for static objects and motion vector information for dynamic objects. The motion vector information for dynamic objects can include motion vector information for standard dynamic objects and motion vector information for dynamic objects with skeletal animation.

[0071] Optionally, the multiple rounds of warping may correspond to multiple warping types, with different warping types performing warping operations on different objects in the first real frame. For example, the warping types may include at least one of the following: a warping operation performed on a static object; a warping operation performed on a dynamic object; a warping operation performed on a static object in a distant view; a warping operation performed on a dynamic object in a near view; a warping operation performed on a dynamic object in a distant view; or a warping operation performed on a static object in a near view.

[0072] It should be understood that in each round of distortion processing, objects with the same attributes in the first real frame may be selected for distortion operation, for example, distortion operation may be performed on static objects or dynamic objects.

[0073] In each round of warping, different objects in the image are warped, and the predicted frame is generated based on the results of multiple warping operations. Each warping operation targets only one type of object, while the rest are discarded. The results of these multiple warping operations are then combined into a single color image with blank areas. By warping different types of objects, the accuracy of the resulting color image can be improved, further enhancing image quality.

[0074] The above-mentioned full-screen grid layer can be used to cover the entire screen. Each grid can be a square or a rectangle. The density of the grids in the full-screen grid layer can be expressed by resolution. The more and denser the grids in the full-screen grid layer, the higher the resolution, and vice versa. Each grid includes a number of pixels. The higher the resolution of the grid, the fewer pixels each grid includes. The pixels included in each grid can be expressed as m×m, where m is an integer greater than 1. For example, the pixels included in each grid can be 6×6, or 8×8. That is, each grid includes 6 rows×6 columns of pixels, or 8 rows×8 columns of pixels.

[0075] Optionally, multiple resolutions can be set for the full-screen grid layer, and the resolution of the full-screen grid layer generated in each round of distortion processing can be the same or different. This can be determined based on the object being distorted in each round of distortion processing. For example, for distant images or static objects, a lower resolution of the full-screen grid layer can be set. For close-up images or dynamic objects, a higher resolution of the full-screen grid layer can be set. Since dynamic objects are in motion, close-up images are closer to the camera and are more affected by the prediction error of the motion vector. Therefore, a higher resolution is required for grid distortion to obtain a more accurate predicted image.

[0076] A close-up image refers to an object in the scene that is relatively close to the camera, and a distant image refers to an object in the scene that is relatively far from the camera. In the embodiments of the present application, a camera refers to a virtual camera in the rendered scene, through which the three-dimensional scene can be observed from a certain angle and direction to obtain a display image of the three-dimensional scene. By changing the angle and direction of the virtual camera, different displays of the three-dimensional scene can be seen.

[0077] Optionally, a scene depth map of the first real frame can be used to determine whether the object being distorted is a near-view image or a far-view image. The scene depth map indicates depth information of pixels in the first real frame. For example, if the depth of most pixels of the object being distorted is greater than a preset threshold, the object is determined to be a far-view image. If the depth of most pixels of the object being distorted is less than the preset threshold, the object is determined to be a near-view image.

[0078] Optionally, it is possible to determine whether the object of the distortion operation is a dynamic object or a static object based on the full-screen semantic image of the first real frame. The full-screen semantic image may also be referred to as a mask image, which refers to an image used to mark different types of areas of the first real frame, and may be obtained by masking the first real frame. By sampling the full-screen semantic image, the type of any pixel in the first real frame may be determined. For example, in the full-screen semantic image, the pixel value of the area where the static object is located may be marked as 0, the pixel value of the area where the ordinary dynamic object is located may be marked as 1, and the pixel value of the dynamic object with skeletal animation may be marked as 2.

[0079] In some examples, in the scheme for determining the resolution of the full-screen mesh layer to be used, the full-screen mesh layer corresponding to multiple rounds of warping operations can use a fixed resolution or a variable resolution. For example, a higher-resolution mesh is used in multiple rounds of warping operations. Or, in order to save computing power and power consumption, the resolution to be used can be determined based on the different objects of each round of warping operation. For example, if the object of a round of warping is a distant and static object, a lower resolution can be used, and a higher resolution can be used in other cases.

[0080] For example, when the object processed in the i-th round in at least one round of distortion processing is a static object in the distance, the resolution of the full-screen grid layer used in the i-th round of processing is set to a first resolution, where i is an integer greater than or equal to 1; when the object processed in the i-th round in at least one round of distortion processing is a static object or a dynamic object in the foreground, the resolution of the full-screen grid layer used in the i-th round of processing is set to a second resolution, and the first resolution is lower than the second resolution.

[0081] For the specific implementation of each round of distortion processing, please refer to the relevant descriptions in Figures 4 and 5 below.

[0082] S302. Fill the blank area in the first target color image of the prediction frame to obtain a second target color image of the prediction frame. The second target color image is the display image of the prediction frame.

[0083] Optionally, when filling blank areas in the first target color image, a fixed pixel filling algorithm can be used, or a target pixel filling algorithm can be selected from multiple candidate pixel filling algorithms based on a threshold range within which the camera angle falls. Because different pixel filling algorithms are suitable for different scenarios, flexibly selecting a pixel filling algorithm for blank area filling based on the camera angle threshold can improve the prediction accuracy and image quality of the displayed image of the predicted frame.

[0084] For example, a threshold range of a camera angle between a first real frame and a second real frame is determined; a target pixel filling algorithm is selected from at least two candidate pixel filling algorithms based on the threshold range of the camera angle; and a blank area in the first target color image is filled according to the target pixel filling algorithm to obtain a second target color image.

[0085] In some examples, a first candidate pixel filling algorithm and a second candidate pixel filling algorithm can be set, which are respectively applicable to situations where the camera angle is large and the camera angle is small. For example, when the camera angle is less than the first angle threshold, the first pixel filling algorithm is selected as the target pixel filling algorithm, and the first pixel filling algorithm achieves pixel filling by sampling the pixels of static objects around the pixels to be filled; when the camera angle is greater than the first angle threshold, the second pixel filling algorithm is selected as the target pixel filling algorithm, and the second pixel filling algorithm achieves pixel filling by sampling the pixels of static objects and dynamic objects around the pixels to be filled. Among them, the specific value of the first angle threshold can be determined according to practice and will not be repeated here. For the specific implementation of the pixel filling algorithm, please refer to the relevant description in Figures 6 and 7 below.

[0086] As previously mentioned, each of the at least one round of warping processes targets a different type of object, and the number of warping rounds can be determined based on the type of object being warped. Next, using Figures 4 and 5 as examples, the specific process of warping an image in accordance with an embodiment of the present application will be described.

[0087] Figure 4 is a schematic diagram of warping a color image of a first real frame according to an embodiment of the present application. As shown in Figure 4 , the at least one round of warping includes two rounds of warping. The first round of warping is for static objects, and the second round of warping is for dynamic objects. Optionally, the present embodiment does not limit the order in which the objects are warped. Warping for dynamic objects may be performed first, followed by warping for static objects.

[0088] Optionally, the resolutions of the first full-screen grid layer and the second full-screen grid layer corresponding to the two rounds of warping can be the same or different. For example, when the resolutions are the same, the resolution of the first full-screen grid layer and the resolution of the second full-screen grid layer are both 8×8. Alternatively, when the resolutions are different, the resolution of the first full-screen grid layer is lower than that of the second full-screen grid layer, for example, the resolution of the first full-screen grid layer is 16×16 and the resolution of the second full-screen grid layer is 8×8.

[0089] Specifically, in the first round of distortion processing, the following steps may be included:

[0090] S401. Generate a first full-screen grid layer covering a first real frame.

[0091] S402. Offset the mesh vertices in the first full-screen mesh layer according to the motion vector information of the static object to obtain a distorted first full-screen mesh layer.

[0092] In the mesh warping process, only mesh vertices are offset and stretched according to the motion vector information of the static object, rather than offset and stretching each pixel, which can improve the computational efficiency of the warping operation.

[0093] S403. Output the pixels of the static object in the color image of the first real frame to the blank image according to the distorted first full-screen grid layer to obtain a first color image.

[0094] The blank image may be a newly created blank image. During the acquisition of the first color image, each pixel in the color image of the first real frame may be determined based on the full-screen semantic image of the first real frame to determine whether it is a pixel of a static object. If so, the pixel is output to the blank image; if not, the pixel is discarded. Specifically, only pixels of static objects are output to the blank image, while pixels of other types of objects are discarded.

[0095] For example, the specific process of obtaining the first color image includes: judging whether the first pixel belongs to a static object or a dynamic object based on the full-screen semantic image of the first real frame, where the first pixel is any pixel in the color image of the first real frame; when the first pixel belongs to a static object, outputting the first pixel to a first position in the blank image to obtain the first color image, where the first position is the corresponding position of the first pixel in the distorted first full-screen grid layer; when the first pixel belongs to a dynamic object, not outputting the first pixel to the blank image.

[0096] In the second round of distortion processing, the following steps may be included:

[0097] S404. Generate a second full-screen grid layer covering the first real frame.

[0098] S405. Offset the mesh vertices in the second full-screen mesh according to the motion vector information of the dynamic object to obtain a distorted second full-screen mesh layer.

[0099] Among them, during the mesh warping process, only the mesh vertices are offset and stretched according to the motion vector information of the dynamic object, rather than offset and stretching each pixel, which can improve the computational efficiency of the warping operation.

[0100] Optionally, the motion vector information of the dynamic object may include motion vector information of a common dynamic object and motion vector information of a dynamic object with skeletal animation.

[0101] S406 . Output the pixels of the dynamic object in the color image of the first real frame to the first color image according to the distorted second full-screen grid layer, so as to obtain a first target color image.

[0102] During the acquisition of the first target color image, the full-screen semantic image of the first real frame can be used to determine whether each pixel in the color image of the first real frame is a pixel of a dynamic object. If so, the pixel is output to the first color image; if not, the pixel is discarded. Specifically, only pixels of dynamic objects are output to the first color image, while pixels of other types of objects are discarded.

[0103] For example, the specific process of obtaining the first target color image includes: judging whether the second pixel belongs to a static object or a dynamic object based on the full-screen semantic image of the first real frame, where the second pixel is any pixel in the color image of the first real frame; when the second pixel belongs to a dynamic object, outputting the second pixel to the second position in the first color image to obtain the first target color image, where the second position is the corresponding position of the second pixel in the distorted second full-screen grid layer; when the second pixel belongs to a static object, not outputting the second pixel to the first color image.

[0104] FIG5 is a schematic diagram of another embodiment of the present application in which a color image of a first real frame is distorted. As shown in FIG5 , the at least one round of distortion processing includes four rounds of distortion processing. The first round of distortion processing is for static objects in the distant view, and the second round of distortion processing is for static objects in the near view. The third round of distortion processing is for dynamic objects in the distant view, and the fourth round of distortion processing is for dynamic objects in the near view. Optionally, the embodiment of the present application does not limit the order of the objects to be distorted. The order of the objects to be distorted in the multiple rounds of distortion processing may also be changed.

[0105] Optionally, the resolutions of the first to fourth full-screen grid layers corresponding to the four rounds of warping can be the same or different. In the case of different resolutions, the resolution of the first full-screen grid layer can be lower than that of the other three full-screen grid layers. For example, the resolution of the first full-screen grid layer is 16×16, and the resolutions of the second to fourth full-screen grid layers are 8×8.

[0106] Specifically, the first round of distortion processing is for static objects in the distant view. The first round of distortion processing may include the following steps:

[0107] S501. Generate a first full-screen grid layer covering a first real frame.

[0108] S502. According to the motion vector information of the static object in the distant view, the mesh vertices in the first full-screen mesh layer are offset to obtain a distorted first full-screen mesh layer.

[0109] S503. Output the pixels of the static objects in the distant scene in the color image of the first real frame to the blank image according to the distorted first full-screen grid layer to obtain a first color image.

[0110] Optionally, during the acquisition of the first color image, each pixel in the color image of the first real frame can be determined based on the full-screen semantic image of the first real frame to determine whether it is a pixel of a static object. If so, the scene depth map is further used to determine whether the pixel belongs to a distant image. If so, the pixel is output to the blank image. If it does not belong to a static object or a distant image, the pixel is discarded. That is, only pixels of static objects in the distant image are output to the blank image, while pixels of other types of objects are discarded.

[0111] For example, based on the full-screen semantic image of the first real frame, determine whether the first pixel belongs to a static object or a dynamic object, and the first pixel is any pixel in the color image of the first real frame; when the first pixel belongs to a static object, determine whether the depth of the first pixel is greater than a preset depth threshold D based on the scene depth map of the first real frame; when the depth of the first pixel is greater than the preset depth threshold D, output the first pixel to a first position in the blank image to obtain a first color image, and the first position is the corresponding position of the first pixel in the distorted first full-screen grid layer; when the depth of the first pixel is less than the preset depth threshold D, do not output the first pixel to the blank image; when the first pixel belongs to a dynamic object, do not output the first pixel to the blank image.

[0112] Optionally, if the depth of the first pixel is greater than a preset depth threshold D, it indicates that the first pixel belongs to a distant image; if it is less than the preset depth threshold D, it indicates that the first pixel belongs to a near image. The specific value of the preset depth threshold D can be determined in practice.

[0113] The second round of distortion processing is for the static objects in the near scene. The second round of distortion processing may include the following steps:

[0114] S504. Generate a second full-screen grid layer covering the first real frame.

[0115] S505 . Offset the mesh vertices in the second full-screen mesh layer according to the motion vector information of the static object in the near scene to obtain a distorted second full-screen mesh layer.

[0116] S506 . Output the pixels of the static objects in the near scene in the color image of the first real frame to the first color image according to the distorted second full-screen grid layer, so as to obtain a second color image.

[0117] Optionally, during the acquisition of the second color image, each pixel in the color image of the first real frame can be determined based on the full-screen semantic image of the first real frame to determine whether it is a pixel of a static object. If so, the scene depth map is further used to determine whether the pixel belongs to the near-field image. If so, the pixel is output to the first color image. If the pixel does not belong to a static object or the near-field image, the pixel is discarded. That is, only pixels of static objects in the near-field are output to the first color image, while pixels of other types of objects are discarded.

[0118] The third round of distortion processing is for the dynamic objects in the distant scene. The third round of distortion processing may include the following steps:

[0119] S507. Generate a third full-screen grid layer covering the first real frame.

[0120] S508. According to the motion vector information of the dynamic object in the distant view, the mesh vertices in the third full-screen mesh layer are offset to obtain a distorted third full-screen mesh layer.

[0121] S509 . Output the pixels of the dynamic object in the distant scene in the color image of the first real frame to the second color image according to the distorted third full-screen grid layer, to obtain a third color image.

[0122] During the acquisition of the third color image, the full-screen semantic image of the first real frame can be used to determine whether each pixel in the color image of the first real frame belongs to a dynamic object. If so, the scene depth map is further used to determine whether the pixel belongs to a distant image. If so, the pixel is output to the second color image; if not, the pixel is discarded. Specifically, only pixels of dynamic objects in the distant image are output to the second color image, while pixels of other types of objects are discarded.

[0123] The fourth round of distortion processing is for the dynamic objects in the near scene. The fourth round of distortion processing may include the following steps:

[0124] S510. Generate a fourth full-screen grid layer covering the first real frame.

[0125] S511. According to the motion vector information of the dynamic object in the near scene, the mesh vertices in the fourth full-screen mesh layer are offset to obtain a distorted fourth full-screen mesh layer.

[0126] S512. Output the pixels of the dynamic object in the near scene in the color image of the first real frame to the third color image according to the distorted fourth full-screen grid layer, so as to obtain a first target color image.

[0127] During the acquisition of the first target color image, the full-screen semantic image of the first real frame can be used to determine whether each pixel in the color image of the first real frame belongs to a dynamic object. If so, the scene depth map is further used to determine whether the pixel belongs to the near-field image. If so, the pixel is output to the third color image; if not, the pixel is discarded. Specifically, only pixels of dynamic objects in the near-field are output to the third color image, while pixels of other types of objects are discarded.

[0128] Next, the pixel filling algorithm is described with reference to Figures 6 and 7. It should be noted that the principle of the pixel filling algorithm is as follows: starting from the current pixel to be filled in the first target color image of the predicted frame, P_0, the color values ​​of the pixels surrounding the starting point P_0 are sampled multiple times according to a random sampling offset value, and the color values ​​of the pixels to be filled are averaged to obtain the color values ​​of the filled pixels. When the camera angle is small, considering that dynamic objects vary significantly between video frames, while static objects vary less, to ensure pixel filling accuracy, pixels from static objects can be sampled to fill blank areas, while pixels from dynamic objects can be discarded. However, when the camera angle is large, the variation of static objects increases, and sampling only the pixels of static objects may result in the failure to sample valid pixels. Therefore, the second pixel filling algorithm simultaneously samples pixels from both static and dynamic objects to fill blank areas.

[0129] FIG6 is a flow chart of a first pixel filling algorithm according to an embodiment of the present application. The first pixel filling algorithm in FIG6 is applicable to situations where the camera angle is small. Referring to FIG6 , the first pixel filling algorithm includes the following contents.

[0130] S601. Determine the first pixel to be filled, and determine the coordinate point of the first pixel as the starting point P_0.

[0131] The first pixel is any pixel in a blank area in the first target color image.

[0132] S602. Obtain a random sampling offset sequence and sampling times N.

[0133] The random sampling offset sequence is used to provide a random sampling offset value offset, so as to provide the sampling offset step size and sampling direction for each pixel sampling. For example, the sampling offset value can be expressed as (Δx, Δy). The sampling number N is used to specify the number of times the surrounding pixels of the first pixel are sampled. N is an integer greater than 1.

[0134] S603. Sample pixels around the first pixel N times according to the random sampling offset sequence to obtain the sum color of the color values ​​of the pixels sampled N times.

[0135] For example, in the i-th sampling among N samplings, the sampling offset value offset_i may be sequentially obtained from the random sampling sequence to calculate the sampling coordinate P_i=P_0+offset_i, where i is an integer greater than or equal to 1, and 1≤i≤N.

[0136] Sample the first target color image and the full-screen semantic image according to sampling coordinates P_i. If the pixel representation of the sampled full-screen semantic image is static, then add the color value of the pixel sampled from the first target color image to color. If the pixel representation of the sampled full-screen semantic image is dynamic, then do nothing.

[0137] S604: Obtain the mean color' of the color values ​​of the pixels sampled N times.

[0138] The mean color' can be expressed as: color'=color / N.

[0139] S605 . Output the average color value of the pixel sampled N times, color′, as the color value of the first pixel.

[0140] It should be noted that FIG6 is an exemplary illustration of the first pixel filling algorithm. In practice, the steps in FIG6 may be appropriately modified or added or deleted, and the resulting solution still falls within the protection scope of the embodiments of the present application.

[0141] FIG7 is a flow chart of a second pixel filling algorithm according to an embodiment of the present application. FIG7 is suitable for situations where the camera angle is large. Referring to FIG7 , the second pixel filling algorithm includes the following contents.

[0142] S701. Determine the first pixel to be filled, and determine the coordinate point of the first pixel as the starting point P_0.

[0143] The first pixel is any pixel in a blank area in the first target color image.

[0144] S702. Obtain a random sampling offset sequence and sampling times N.

[0145] The random sampling offset sequence is used to provide a random sampling offset value, so as to provide the sampling offset step size and sampling direction for each pixel sampling. For example, the sampling offset value can be expressed as (Δx, Δy). The sampling number N is used to specify the number of times the surrounding pixels of the first pixel are sampled. N is an integer greater than 1.

[0146] S703. Sample pixels around the first pixel N times according to the random sampling offset sequence to obtain the sum color of the color values ​​of the pixels sampled N times, wherein only pixels of static objects are sampled in the N times of sampling.

[0147] For example, in the i-th sampling among N samplings, the sampling offset value offset_i may be sequentially obtained from the random sampling sequence to calculate the sampling coordinate P_i=P_0+offset_i, where i is an integer greater than or equal to 1, and 1≤i≤N.

[0148] Sample the first target color image and the full-screen semantic image according to sampling coordinates P_i. If the pixel representation of the sampled full-screen semantic image is static, then add the color value of the pixel sampled from the first target color image to color. If the pixel representation of the sampled full-screen semantic image is dynamic, then do nothing.

[0149] S704. If color is not equal to 0, output the average color' of the color values ​​of the pixels sampled N times as the color value of the first pixel.

[0150] The mean color' can be expressed as: color'=color / N.

[0151] S705. If color is equal to 0, resample the pixels around the first pixel M times to obtain the sum of the color values ​​of the M samples. Pixels of static objects and dynamic objects can be sampled in the M samples, where M is an integer greater than 1.

[0152] S706. Obtain the mean color' of the color values ​​of the pixels sampled M times according to the sum color of the color values ​​obtained by the M sampling times.

[0153] S707 . Output the mean color′ of the color values ​​of the pixels sampled M times as the color value of the first pixel.

[0154] It should be noted that FIG7 is an exemplary illustration of the second pixel filling algorithm. In practice, the steps in FIG7 may be appropriately modified or added or deleted, and the resulting solution still falls within the protection scope of the embodiments of the present application.

[0155] FIG8 is a flow chart of a rendering process according to an embodiment of the present application. FIG8 takes rendering a display interface in a game application as an example for illustration. As shown in FIG8 , the method includes the following contents.

[0156] S801. The game application sends a rendering instruction stream to the rendering module. The rendering instruction stream is used to instruct the rendering of video frames in the game.

[0157] While processing the rendering instruction stream, the rendering module can obtain user interface (UI) instructions. For example, based on the characteristics of the game rendering instructions, the rendering module can identify instructions for drawing the game UI. UI instructions include, but are not limited to, resource binding instructions, state setting instructions, and drawing instructions. These instructions are then stored in the instruction stream buffer for subsequent use.

[0158] S802. The rendering module calls the GPU to draw the real frame according to the rendering instruction stream.

[0159] S803. In the process of drawing the real frame, obtain the full-screen semantic image, scene depth map, and color map of the two real frames.

[0160] In the process of drawing the real frame, each rendering instruction can be judged as static or dynamic based on the game rendering instruction characteristics, material information, and instruction calling rules. If it is the drawing of a static object, the stencil value of the rendering instruction is set to 0. The stencil value represents the template value. If it is the drawing of an ordinary dynamic object, the stencil value of the rendering instruction is set to 1. If it is the drawing of a dynamic object with skeletal animation, the three-dimensional space coordinate information of the model of two consecutive frames is saved in a buffer for subsequent motion vector calculation, and the stencil value of the rendering instruction is set to 2.

[0161] After the rendering instruction is completed, a full-screen semantic map can be generated, in which the pixel value covered by static objects is 0, the pixel value covered by ordinary dynamic objects is 1, and the pixel value of dynamic objects with skeletal animation is 2.

[0162] S804. Determine a strategy for generating a predicted frame based on the camera angle between the two real frames.

[0163] If the camera angle is less than or equal to the second preset angle, strategy 1 may be executed. If the camera angle is greater than or equal to the third preset angle, strategy 2 may be executed. If the camera angle is between the second preset angle and the third preset angle, strategy 3 may be executed.

[0164] The third preset angle is greater than the second preset angle. The specific values ​​of the second preset angle and the third preset angle can be determined according to practice and are not limited here. As an example, the second preset angle can be 2° and the third preset angle can be 6°.

[0165] Strategy 1 includes steps S805 to S808:

[0166] S805. Obtain a motion vector map between two real frames.

[0167] Optionally, the embodiment of the present application does not limit the specific method of obtaining the motion vector map.

[0168] As an example, in the process of obtaining a motion vector diagram, different calculation methods can be used to obtain the motion vector of a dynamic object with skeletal animation, the motion vector of an ordinary dynamic object, and the motion vector of a static object, and the above three types of motion vectors can be merged together to obtain a motion vector diagram.

[0169] By calculating the motion vectors of different types of objects using different calculation methods, the accuracy of the motion vector of each type of object can be improved, thereby improving the accuracy of the merged motion vector map.

[0170] For example, for static objects, motion vectors can be calculated using the scene depth map and the virtual camera's change matrix. Specifically, the normalized device coordinates (NDC) can be constructed by sampling the depth value of the corresponding position in the scene depth map and the image coordinates of the sampling position. After obtaining the NDC coordinates, the image coordinates of the current pixel's world space coordinates on the predicted frame image are calculated using the camera space matrix and projection matrix between the two real frames, thereby obtaining pixel-by-pixel motion vector data.

[0171] For dynamic objects with skeletal animation, the motion vector can be calculated based on its model. For example, each rendering instruction of the model with skeletal animation can be marked from the rendering instruction stream of the real frame, and each model with skeletal animation has a corresponding transform feedback buffer. Modify the rendering algorithm of the skeletal animation model in the real frame, add the function of saving the NDC coordinates of each model vertex after coordinate transformation into the corresponding transform feedback buffer, and then obtain the image space coordinates of each vertex of the skeletal animation model in the two real frames. Create a new rendering instruction, calculate the motion vector of each vertex of the skeletal animation model, and then use the rasterization function of the GPU to obtain the motion vector corresponding to the pixel by linear interpolation of the vertex motion vector for each pixel in the pixel area covered by the vertex.

[0172] For common dynamic objects, the motion vector can be estimated by sampling the color values ​​of the corresponding position in the color image of the first real frame and the color value distribution of a certain area around it. Then, an iteratively optimized matching algorithm is used to find the image data that most closely matches this distribution on the color image of the second real frame. First, based on the vertex data of the dynamic object with skeletal animation, the coordinates in image space are calculated. The motion vector of each vertex is calculated by interpolation, and then rasterized and interpolated into pixel data, which is rendered on the motion vector image.

[0173] The motion vector is calculated using the color image, semantic image, and depth image. For each pixel in the motion vector image to be calculated, the data at the corresponding position in the semantic image is sampled to obtain semantic information.

[0174] S806. Perform at least one round of distortion processing on the first real frame to obtain a first target color image of the predicted frame.

[0175] For a detailed description of performing at least one round of distortion processing, please refer to the relevant contents in FIG. 3 to FIG. 5 , which will not be described again for the sake of brevity.

[0176] S807. Fill the blank area in the first target color image to obtain a second target color image of the prediction frame. The second target color image is the display image of the prediction frame.

[0177] For the specific process of filling the blank area in the first target color image, please refer to the relevant contents in FIG3 , FIG6 and FIG7 , which will not be described here for the sake of brevity.

[0178] S808. Perform UI rendering to form a display image of the predicted frame.

[0179] As an example, UI rendering includes: using the display image of the predicted frame as a rendering target, executing the UI rendering instruction stream stored in S801 on the rendering target, drawing UI information onto the display image of the predicted frame, and forming a complete video frame that can be used for display.

[0180] Strategy 2 includes steps S809 and S810:

[0181] S809. Obtain a color image of the prediction frame by using image multiplexing or image mixing.

[0182] Image multiplexing refers to outputting any one of the two real frames as the color image of the prediction frame. Image blending refers to blending the images of the two real frames in a semi-transparent manner and outputting the blended image as the color image of the prediction frame.

[0183] It should be understood that Strategy 2 is suitable for situations where the camera angle between the two real frames is large. If the camera angle is greater than or equal to the third preset angle, it indicates that the current game screen is significantly shaky. Therefore, the information required for predictive frame calculation is too much, and the algorithm will generate inaccurate results. Therefore, image multiplexing or image blending can be used to obtain predictive frames.

[0184] S810. Perform UI rendering to form a display image of the predicted frame.

[0185] As an example, UI rendering includes: using the display image of the predicted frame as a rendering target, executing the UI rendering instruction stream stored in S801 on the rendering target, drawing UI information onto the display image of the predicted frame, and forming a complete video frame that can be used for display.

[0186] Strategy 3 includes step S811:

[0187] S811. Terminate the process of generating the prediction frame.

[0188] It should be understood that if the camera angle between the two real frames is within the range between the second angle threshold and the third angle threshold, it indicates that the algorithm cannot generate a predicted frame image with acceptable image quality in the current scenario, so the frame prediction scheme is exited and normal rendering of the game is restored.

[0189] The method for image processing provided in an embodiment of the present application is described above in conjunction with Figures 1 to 8. The device for executing the above method provided in an embodiment of the present application is described below.

[0190] In order to better understand the embodiments of the present application, the structure of the apparatus for image processing according to the embodiments of the present application is introduced below.

[0191] 9 shows a schematic diagram of the hardware structure of an apparatus 900 for image processing. The apparatus 900 for image processing may include a processor 110, an external memory interface 120, an internal memory 121, a sensor module 180, and a display screen 194.

[0192] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the apparatus 900 for image processing. In other embodiments of the present application, the apparatus 900 for image processing may include more or fewer components than illustrated, or may combine or separate certain components, or may have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0193] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a graphics processing unit (GPU), an image signal processor (ISP), a video codec, etc. The different processing units may be independent devices or integrated into one or more processors.

[0194] It should be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is for illustrative purposes only and does not constitute a structural limitation on the image processing apparatus 900. In other embodiments of the present application, the image processing apparatus 900 may also adopt different interface connection methods from those in the above embodiments, or a combination of multiple interface connection methods.

[0195] Device 900 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0196] Display screen 194 is used to display images, videos, and receive sliding operations. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, device 900 may include one or N display screens 194, where N is a positive integer greater than one.

[0197] External memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of apparatus 900 for image processing. The external memory card communicates with processor 110 via external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0198] The internal memory 121 can be used to store computer-executable program code, which includes instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the image processing device 900 (such as audio data, a phone book, etc.).

[0199] The sensor module 180 may include a pressure sensor, a touch sensor, and the like.

[0200] FIG10 is a schematic diagram of the structure of an apparatus 1000 for image processing provided in an embodiment of the present application. The apparatus 1000 may include an electronic device, or may also include a chip or chip system within an electronic device. The apparatus 1000 includes functional units that execute the image processing methods of FIG3 to FIG8.

[0201] As shown in FIG. 10 , the device 1000 includes a twisting unit 1010 and a filling unit 1020 .

[0202] The warping unit 1010 is configured to perform at least one round of warping on the color image of the first real frame based on motion vector information between the first real frame and the second real frame to obtain a first target color image of the predicted frame, wherein the first target color image of the predicted frame includes blank areas caused by the at least one round of warping. Each round of the at least one round of warping includes: generating a full-screen grid layer covering the first real frame, the full-screen grid layer including multiple grids, each grid including multiple pixels; warping at least some of the grids in the full-screen grid layer based on the motion vector information; and outputting a color image corresponding to each round of warping based on the warped full-screen grid layer and the color image of the first real frame.

[0203] The filling unit 1020 is configured to fill the blank area in the first target color image of the prediction frame to obtain a second target color image of the prediction frame, where the second target color image is the display image of the prediction frame.

[0204] Figure 11 is a schematic diagram of the hardware structure of another device 1100 for image processing provided in an embodiment of the present application. As shown in Figure 11, the device 1100 includes a processor 1101, a communication line 1104 and at least one communication interface (the communication interface 1103 is exemplified in Figure 11).

[0205] The processor 1101 may include a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a GPU, or one or more integrated circuits for controlling the execution of the program of the present application.

[0206] Communications link 1104 may include circuitry that transmits information between the aforementioned components.

[0207] The communication interface 1103 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0208] Possibly, the apparatus for image processing may further include a memory 1102 .

[0209] The memory 1102 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a communication line 1104. The memory may also be integrated with the processor.

[0210] The memory 1102 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the computer-executable instructions stored in the memory 1102, thereby implementing the method for video cropping provided by the embodiment of the present application.

[0211] Possibly, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, and the embodiments of the present application do not specifically limit this.

[0212] As shown in FIG11 , in a specific implementation, as an embodiment, the processor 1101 may include one or more CPUs, and may also include one or more GPUs.

[0213] An embodiment of the present application provides an electronic device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the above-mentioned method for image processing.

[0214] The present embodiment provides a chip. The chip includes a processor configured to invoke a computer program stored in a memory to execute the image processing method described in the above embodiment. The implementation principles and technical effects are similar to those of the above-described related embodiments and are not further described here.

[0215] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0216] In one possible implementation, computer-readable media may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium designed to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave are used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and optical disk as used herein include optical disk, laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk, where disks generally reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0217] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0218] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application.It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processing unit of general-purpose computer, special-purpose computer, embedded processing machine or other programmable device to produce a machine, so that the instruction executed by the processing unit of computer or other programmable data processing device produces the device for realizing the function specified in one flow chart flow or multiple flows and / or one block or multiple blocks of block diagram.

[0219] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image processing, characterized in that include: performing at least one round of warping on the color image of the first real frame according to motion vector information between the first real frame and the second real frame to obtain a first target color image of the predicted frame, wherein the first target color image of the predicted frame includes a blank area caused by the at least one round of warping; Each round of the at least one round of warping includes: generating a full-screen grid layer covering the first real frame, the full-screen grid layer including a plurality of grids, each grid including a plurality of pixels; performing a warping process on at least some of the grids in the full-screen grid layer according to the motion vector information; and outputting a color image corresponding to each round of the warping process based on the warped full-screen grid layer and the color image of the first real frame. The blank area in the first target color image of the prediction frame is filled to obtain a second target color image of the prediction frame, where the second target color image is the display image of the prediction frame.

2. The method according to claim 1, wherein The at least one round of distortion processing includes multiple rounds of distortion processing, and the multiple rounds of distortion processing correspond to multiple distortion types respectively, and the multiple distortion types include at least one of the following: Distortion operations performed on static objects; Distortion operations performed on dynamic objects; Distortion operations performed on static objects in the distance; Warping operations performed on dynamic objects in the foreground; Warping operations performed on dynamic objects in the distance; A distortion operation performed on static objects in the foreground.

3. The method according to claim 1 or 2, wherein: The at least one round of distortion processing includes multiple rounds of distortion processing, and resolutions of full-screen grid layers corresponding to at least two rounds of distortion processing in the multiple rounds of distortion processing are different.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: When the object processed in the i-th round of the at least one warping process is a static object in the distant view, setting the resolution of the full-screen grid layer used in the i-th round of the process to a first resolution, where i is an integer greater than or equal to 1; When the object processed in the i-th round of at least one round of distortion processing is a static object or a dynamic object in the foreground, the resolution of the full-screen grid layer used in the i-th round of processing is set to a second resolution, and the first resolution is lower than the second resolution.

5. The method according to any one of claims 1 to 4, characterized in that The at least one round of distortion processing includes multiple rounds of distortion processing, and a first round of distortion processing in the multiple rounds of distortion processing includes: generating a first full-screen grid layer covering the first real frame; offsetting mesh vertices in the first full-screen mesh layer according to motion vector information of the static object to obtain a distorted first full-screen mesh layer; According to the distorted first full-screen grid layer, pixels of static objects in the color image of the first real frame are output to a blank image to obtain a first color image.

6. The method according to claim 5, wherein Outputting the pixels of the static object in the color image of the first real frame to the blank image to obtain the first color image includes: determining, based on the full-screen semantic image of the first real frame, whether a first pixel belongs to a static object or a dynamic object, the first pixel being any pixel in the color image of the first real frame; In the case where the first pixel belongs to a static object, the first pixel is output to the first pixel in the blank image. a position to obtain the first color image, where the first position is a corresponding position of the first pixel in the first full-screen grid layer after the distortion; In a case where the first pixel belongs to a dynamic object, the first pixel is not output to the blank image.

7. The method according to claim 5 or 6, wherein: The second round of distortion processing in the at least one round of distortion processing includes: generating a second full-screen grid layer covering the first real frame; offsetting mesh vertices in the second full-screen mesh according to motion vector information of the dynamic object to obtain a distorted second full-screen mesh layer; According to the distorted second full-screen grid layer, pixels of the dynamic object in the color image of the first real frame are output to the first color image to obtain the first target color image.

8. The method according to claim 7, wherein Outputting pixels of the dynamic object in the color image of the first real frame to the first color image according to the warped second full-screen grid layer to obtain the first target color image includes: determining, based on the full-screen semantic image of the first real frame, whether a second pixel belongs to a static object or a dynamic object, the second pixel being any pixel in the color image of the first real frame; If the second pixel belongs to a dynamic object, output the second pixel to a second position in the first color image to obtain the first target color image, where the second position is a corresponding position of the second pixel in the warped second full-screen grid layer; In a case where the second pixel belongs to a static object, the second pixel is not output to the first color image.

9. The method according to any one of claims 1 to 4, characterized in that The at least one round of distortion processing includes multiple rounds of distortion processing, and a first round of distortion processing in the multiple rounds of distortion processing includes: generating a first full-screen grid layer covering the first real frame; offsetting the mesh vertices in the first full-screen mesh layer according to the motion vector information of the static object in the distant view to obtain a distorted first full-screen mesh layer; According to the distorted first full-screen grid layer, pixels of static objects in the distant scene in the color image of the first real frame are output to a blank image to obtain a first color image.

10. The method according to claim 9, wherein Outputting pixels of static objects in the distant scene in the color image of the first real frame to a blank image according to the distorted first full-screen grid layer to obtain a first color image includes: determining, based on the full-screen semantic image of the first real frame, whether a first pixel belongs to a static object or a dynamic object, the first pixel being any pixel in the color image of the first real frame; In a case where the first pixel belongs to a static object, determining, according to the scene depth map of the first real frame, whether the depth of the first pixel is greater than a preset depth threshold; When the depth of the first pixel is greater than the preset depth threshold D, outputting the first pixel to a first position in the blank image to obtain the first color image, where the first position is a corresponding position of the first pixel in the warped first full-screen grid layer; When the depth of the first pixel is less than the preset depth threshold D, not outputting the first pixel to the blank image; In a case where the first pixel belongs to a dynamic object, the first pixel is not output to the blank image.

11. The method according to claim 9 or 10, wherein: The second round of distortion processing in the multiple rounds of distortion processing includes: generating a second full-screen grid layer covering the first real frame; offsetting the mesh vertices in the second full-screen mesh layer according to the motion vector information of the static object in the near scene to obtain a distorted second full-screen mesh layer; According to the distorted second full-screen grid layer, pixels of static objects in the near scene in the color image of the first real frame are output to the first color image to obtain a second color image.

12. The method according to claim 11, wherein The third round of distortion processing in the multiple rounds of distortion processing includes: generating a third full-screen grid layer covering the first real frame; offsetting the mesh vertices in the third full-screen mesh layer according to motion vector information of the dynamic object in the distant view to obtain a distorted third full-screen mesh layer; According to the distorted third full-screen grid layer, pixels of the dynamic object in the color image of the first real frame are output to the second color image to obtain a third color image.

13. The method according to claim 12, wherein: The fourth round of distortion processing in the multiple rounds of distortion processing includes: generating a fourth full-screen grid layer covering the first real frame; offsetting the mesh vertices in the fourth full-screen mesh layer according to motion vector information of the dynamic object in the near scene to obtain a distorted fourth full-screen mesh layer; According to the distorted fourth full-screen grid layer, pixels of the dynamic object in the color image of the first real frame are output to the third color image to obtain the first target color image.

14. The method according to any one of claims 1 to 13, characterized in that Filling the blank area in the first target color image of the prediction frame to obtain the second target color image of the prediction frame includes: determining a threshold range to which a camera angle between the first real frame and the second real frame belongs; Selecting a target pixel filling algorithm from at least two candidate pixel filling algorithms according to a threshold range to which the camera angle belongs; According to the target pixel filling algorithm, the blank area in the first target color image is filled to obtain the second target color image.

15. The method according to claim 14, wherein The selecting a target pixel filling algorithm from at least two candidate pixel filling algorithms according to a threshold range to which the camera angle belongs includes: When the camera angle is less than a first angle threshold, selecting a first pixel filling algorithm as the target pixel filling algorithm, wherein the first pixel filling algorithm implements pixel filling by sampling pixels of static objects around the pixels to be filled; When the camera angle is greater than the first angle threshold, a second pixel filling algorithm is selected as the target pixel filling algorithm, wherein the second pixel filling algorithm realizes pixel filling by sampling pixels of static objects and dynamic objects around the pixels to be filled.

16. A device for image processing, characterized in that: The device comprises: processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the apparatus performs the method according to any one of claims 1 to 15.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

18. A computer program product, characterized in that The invention comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 15.

Images