Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment. In the method, when a prediction frame is calculated based on a first real frame and a second real frame, firstly, motion vectors corresponding to the first real frame and the second real frame are calculated, an initial mesh corresponding to the second real frame is created, then, subdivision intersection points are identified based on pixel depth values, and the initial mesh is subdivided based on the subdivision intersection points to obtain a target mesh. And according to the target Mesh and the motion vector, carrying out image distortion processing on the second real frame to generate a prediction frame image. Therefore, the problem of local stretching of the image area with discontinuous depth values in the prediction frame can be avoided, continuous graphic information is well reserved, the picture quality of the prediction frame image is improved, and the visual experience of a user is good.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 18, 2022, with application number 202211444045.9 and application name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of smart terminal technology, and in particular to an image processing method and electronic device. Background Art

[0003] Video interpolation technology aims to improve the frame rate and smoothness of videos, making them appear smoother. By inserting a predicted frame (also known as an intermediate frame or transition frame) between adjacent real frames (or original frames), the frame rate of a video can be doubled. The quality of the predicted frame is directly related to the smoothness of the video. The movement direction and speed of each object in the image are calculated from the information of two adjacent frames, and the objects in the image are moved accordingly to generate the predicted frame.

[0004] In current prediction frame generation technology, image warping is a good way to ensure image continuity. However, in some image regions, image warping may cause local image distortion, making the difference between the predicted frame image and the real frame image obvious, greatly reducing the user's visual experience.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems, an embodiment of the present application provides an image processing method and an electronic device. In this method, when the electronic device calculates a predicted frame based on a first real frame and a second real frame, it first calculates the motion vector corresponding to the first real frame and the second real frame, and creates an initial mesh corresponding to the second real frame. Then, based on the pixel depth value, the subdivision intersection is identified, and the initial mesh is subdivided based on the subdivision intersection to obtain the target mesh. Then, according to the target mesh and the motion vector, the second real frame is subjected to image distortion processing to generate a predicted frame image. This can avoid the problem of local stretching in the image area with discontinuous depth values ​​in the predicted frame, so as to better retain continuous graphic information, improve the picture quality of the predicted frame image, and provide the user with a better visual experience.

[0007] In a first aspect, an embodiment of the present application provides an image processing method. The method includes: an electronic device acquiring a first image and a second image, and calculating a motion vector based on the first image and the second image; wherein the first image and the second image are real frame images; the electronic device creating an initial mesh of the second image, and determining an initial geometric figure to be subdivided corresponding to a static object in the initial mesh; wherein the initial geometric figure has two and only two subdivision intersections located on the edges of the initial geometric figure, and each edge of each initial geometric figure has at most one subdivision intersection; the subdivision intersections are determined based on the depth values ​​of pixels in the initial geometric figure and are edge pixels of the static object; the electronic device subdividing the corresponding initial geometric figure based on the two subdivision intersections of each initial geometric figure to obtain a target mesh of the second image; in the target mesh, the line connecting the two subdivision intersections of each initial geometric figure is the edge of a target triangle, or a portion of the line segment on the line is the edge of the target triangle; wherein the target triangle is a geometric figure divided in the target mesh; and the electronic device performing image warping processing on the second image based on the target mesh and the motion vector to generate a predicted frame image corresponding to the first image and the second image.

[0008] The first image and the second image may be two adjacent real frame images in a video source.

[0009] Exemplarily, the first image and the second image may include only image information, or may include both image information and user interface information.

[0010] The subdivision intersection is used to subdivide the initial geometric figures to be subdivided corresponding to the static objects in the initial Mesh, so that the pixel depth information of each geometric figure after subdivision is continuous.

[0011] For example, the initial geometric shape may be a triangle, a quadrilateral, a polygon, etc.

[0012] In this way, the problem of local stretching in image areas with discontinuous depth values ​​in the predicted frame image can be avoided, so as to better retain continuous graphic information, improve the picture quality of the predicted frame image, and provide users with a better visual experience.

[0013] According to the first aspect, the initial Mesh is a quadrilateral Mesh, and the initial geometric figure is a quadrilateral; after the electronic device creates the initial Mesh of the second image, the method further includes: the electronic device normalizing the quadrilateral Mesh to divide each quadrilateral into two triangles.

[0014] In this way, when the initial geometric figures divided when creating the Mesh are non-triangular, the Mesh is standardized and each initial geometric figure is divided into triangles, which is more conducive to graphic calculation processing.

[0015] According to the first aspect, or any implementation of the first aspect above, the subdivision intersection of the quadrilateral to be subdivided is the first subdivision intersection and the second subdivision intersection, and the quadrilateral to be subdivided is divided into the first triangle and the second triangle;

[0016] Among them, in the target Mesh:

[0017] The first triangle is subdivided into a first sub-triangle, a second sub-triangle, and a third sub-triangle; wherein the common side of the first sub-triangle and the second sub-triangle is the line connecting the first subdivision intersection and the second subdivision intersection, and the common side of the second sub-triangle and the third sub-triangle is the line connecting the first subdivision intersection or the second subdivision intersection and the vertex of the first triangle (in this case, refer to the situations shown in (1) and (2) in Figure 13b); or,

[0018] The second triangle is subdivided into a fourth sub-triangle, a fifth sub-triangle, and a sixth sub-triangle; wherein the common side of the fourth sub-triangle and the fifth sub-triangle is the line connecting the first subdivision intersection and the second subdivision intersection, and the common side of the fifth sub-triangle and the sixth sub-triangle is the line connecting the first subdivision intersection or the second subdivision intersection and the vertex of the second triangle (in this case, refer to the situations shown in (7) and (8) in Figure 13b); or,

[0019] The first triangle is subdivided into the seventh sub-triangle, the eighth sub-triangle and the ninth sub-triangle, and the second triangle is subdivided into the tenth sub-triangle, the eleventh sub-triangle and the twelfth sub-triangle; among them, the common side of the seventh sub-triangle and the eighth sub-triangle is the line connecting the first sub-division intersection and the third point, the common side of the eighth sub-triangle and the ninth sub-triangle is the line connecting the first sub-division intersection and the vertex of the first triangle; the common side of the tenth sub-triangle and the eleventh sub-triangle is the line connecting the second sub-division intersection and the third point, and the common side of the eleventh sub-triangle and the twelfth sub-triangle is the line connecting the second sub-division intersection and the vertex of the second triangle; the line connecting the first sub-division intersection and the second sub-division intersection, and the intersection of the common side of the first triangle and the second triangle is the third point (at this time, you can refer to the situations shown in (3) to (6) in Figure 13b).

[0020] The first triangle can be subdivided individually based on the subdivision intersection, or not, or it can be subdivided together with the second triangle based on the subdivision intersection. In this way, in the target mesh, the pixel depth values ​​of each sub-triangle are continuous. This prevents geometric stretching and deformation when performing image warping based on each subdivided sub-triangle. This avoids local stretching in image areas with discontinuous depth values ​​in the predicted frame image, and better preserves continuous graphic information.

[0021] According to the first aspect, or any implementation of the first aspect above, the method further includes: in the target Mesh, the electronic device sets depth information of each triangle.

[0022] In this way, when the electronic device performs image distortion processing, it can combine pixel depth information to ensure the correct depth sequence relationship between different objects in the predicted frame image.

[0023] According to the first aspect, or any implementation method of the first aspect above, the electronic device performs image distortion processing on the second image based on the target Mesh and the motion vector, which may include: for each triangle in the target Mesh, the electronic device determines the motion vector of each vertex of the triangle, and moves the triangle according to the motion vector of each vertex of the triangle; when each triangle moves to the target position, the electronic device determines the coverage relationship between the triangle and other triangles at the target position based on the depth information of the triangle and the depth information of other triangles at the target position.

[0024] In this way, when the electronic device moves a triangle based on the motion vector, it can determine its covering relationship with other triangles according to the depth information of the triangle, so as to ensure the correct depth order relationship between different objects in the predicted frame image.

[0025] According to the first aspect, or any implementation method of the above first aspect, the subdivision intersection of the quadrilateral to be subdivided is the first subdivision intersection and the second subdivision intersection, and the quadrilateral to be subdivided is divided into the first triangle and the second triangle; the electronic device meshes the initial geometric figure according to the two subdivision intersections of the initial geometric figure, which may include: the electronic device meshes the first triangle and / or the second triangle using a matching mesh subdivision strategy based on the positional relationship between the first subdivision intersection and the second subdivision intersection and the first triangle and the second triangle.

[0026] Among them, the electronic device can mesh the first triangle separately, mesh the second triangle separately, or mesh the first triangle and the second triangle together based on the positional relationship between the first subdivision intersection point and the second subdivision intersection point and the first triangle and the second triangle.

[0027] According to the first aspect, or any implementation method of the first aspect above, the electronic device uses a matching mesh subdivision strategy to mesh the first triangle and / or the second triangle based on the positional relationship between the first subdivision intersection and the second subdivision intersection, the first triangle, and the second triangle. It can include: when the horizontal coordinate of the first subdivision intersection is equal to the horizontal coordinates of the first vertex and the second vertex of the first triangle, and the vertical coordinate of the second subdivision intersection is equal to the vertical coordinates of the second vertex and the third vertex of the first triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, and connects the first subdivision intersection and the third vertex of the first triangle or connects the second subdivision intersection and the first vertex of the first triangle, so as to subdivide the first triangle into three triangles separately.

[0028] In this case, the electronic device performs mesh subdivision on the first triangle based on the first subdivision intersection and the second subdivision intersection, but does not perform mesh subdivision on the second triangle. In this case, the situations shown in (1) and (2) in FIG. 13b may be referred to.

[0029] According to the first aspect, or any implementation of the first aspect above, the electronic device subdivides the first triangle and / or the second triangle using a matching mesh subdivision strategy based on a positional relationship between the first subdivision intersection point and the second subdivision intersection point, the first triangle, and the second triangle, and may include:

[0030] When the horizontal coordinate of the first subdivision intersection is equal to the horizontal coordinates of the first vertex and the second vertex of the second triangle, and the vertical coordinate of the second subdivision intersection is equal to the vertical coordinates of the second vertex and the third vertex of the second triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, and connects the first subdivision intersection and the third vertex of the second triangle or connects the second subdivision intersection and the first vertex of the second triangle to subdivide the second triangle into three triangles separately.

[0031] In this case, the electronic device performs mesh subdivision on the second triangle based on the first subdivision intersection and the second subdivision intersection, but does not perform mesh subdivision on the first triangle. In this case, the situations shown in (7) and (8) in FIG. 13b may be referred to.

[0032] According to the first aspect, or any implementation of the first aspect above, the electronic device subdivides the first triangle and / or the second triangle using a matching mesh subdivision strategy based on a positional relationship between the first subdivision intersection point and the second subdivision intersection point, the first triangle, and the second triangle, including:

[0033] When the ordinate of the first subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the first triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the second triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, connects the first subdivision intersection and the first vertex of the first triangle, and connects the second subdivision intersection and the first vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively (in this case, the situation shown in (3) in FIG. 13b may be referred to);

[0034] When the abscissa of the first subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the first triangle, and the abscissa of the second subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the second triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, connects the first subdivision intersection and the third vertex of the first triangle, and connects the second subdivision intersection and the third vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively (in this case, the situation shown in (5) in FIG. 13b may be referred to);

[0035] When the ordinate of the first subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the first triangle, and the abscissa of the second subdivision intersection is equal to the abscissa of the first vertex and the second vertex of the second triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, connects the first subdivision intersection and the first vertex of the first triangle, and connects the second subdivision intersection and the third vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively (in this case, the situation shown in (4) in FIG. 13b may be referred to);

[0036] When the horizontal coordinate of the first subdivision intersection is equal to the horizontal coordinates of the first vertex and the second vertex of the first triangle, and the vertical coordinate of the second subdivision intersection is equal to the vertical coordinates of the second vertex and the third vertex of the second triangle, the electronic device connects the first subdivision intersection and the second subdivision intersection, connects the first subdivision intersection and the third vertex of the first triangle, and connects the second subdivision intersection and the first vertex of the second triangle to subdivide the first triangle and the second triangle into three triangles respectively (at this time, you can refer to the situation shown in (6) in Figure 13b).

[0037] In the above four situations, the electronic device may perform mesh subdivision on the first triangle and the second triangle based on the first subdivision intersection point and the second subdivision intersection point, so that the depth value of the pixel points of each subdivided triangle is continuous.

[0038] According to the first aspect, or any implementation method of the first aspect above, the electronic device determines the initial geometric figures to be subdivided corresponding to the static object in the initial Mesh, which may include: the electronic device determines the initial geometric figures corresponding to the static object in the initial Mesh; the electronic device determines the subdivision intersections of each initial geometric figure; the electronic device uses the initial geometric figure with 2 subdivision intersections and at most one subdivision intersection on each edge as the initial geometric figure to be subdivided.

[0039] According to the first aspect, or any implementation method of the first aspect above, the electronic device determines the subdivision intersection of the initial geometric figure, which may include: the electronic device determines each edge pixel point in the initial geometric figure; wherein the depth value of the edge pixel point is within a preset range, and the difference in depth value between the edge pixel point and the surrounding pixel points exceeds a preset threshold; the electronic device uses the edge pixel point located on the edge of the initial geometric figure as the subdivision intersection of the initial geometric figure.

[0040] In this way, subdivision intersections are determined among the pixel points on the edge of the static object in the image, so that the depth values ​​of the pixel points of the subdivided triangles are all continuous.

[0041] According to the first aspect, or any implementation of the first aspect above, the first image and the second image are rendered images.

[0042] Since the first image and the second image are rendered images, the predicted frame image calculated based on the two is also a rendered image, which can achieve halving the rendering at the same frame rate, thereby significantly reducing the power consumption of electronic devices and reducing the heat generation of electronic devices.

[0043] According to the first aspect, or any implementation method of the above first aspect, the electronic device obtains the first image and the second image, which may include: the electronic device obtains the first resource image, and separates the user interface UI information in the first resource image to obtain the first image; the electronic device obtains the second resource image, and separates the UI information in the second resource image to obtain the second image; after the electronic device generates the predicted frame image corresponding to the first image and the second image, it also includes: the electronic device performs image completion processing on the predicted frame image, and fuses the predicted frame image after the completion processing and the UI information to obtain the target predicted frame image.

[0044] The resource image includes image information and UI information.

[0045] The UI information separated in the first resource image and the UI information separated in the second resource image may be the same as or different from each other.

[0046] When the UI information separated in the first resource image is different from the UI information separated in the second resource image, the electronic device fuses the second predicted frame image with the UI information corresponding to the second resource image to obtain a target predicted frame image, where the second resource image is the current frame resource image and the first resource image is the previous frame resource image.

[0047] In this way, the electronic device separates the UI information from the resource image and then calculates the predicted frame image, which can effectively reduce the image data processing volume and improve the accuracy of the predicted frame image information.

[0048] According to the first aspect, or any implementation of the first aspect above, the first resource image and the second resource image are resource images in a game application.

[0049] In the game interpolation scene, the image scene is relatively complex, there are many static objects (such as various buildings, etc.), and the depth difference of static objects in the scene is large (that is, the depth values ​​of different static objects are discontinuous). In addition, since most games are virtual worlds, when users play mobile games, user operations often cause large-angle shaking of the game scene, unlike the physical world where the movement of objects follows the laws of physics. Therefore, in the case of a relatively complex game scene, large-angle scene shaking may only cause some objects in the image to move more, rather than causing all objects in the image to move more. In other words, in the game scene, the distribution of MV images calculated based on continuous real frame images is uneven, especially in image areas with large differences in depth information, the difference in MV values ​​is also relatively large. In the game interpolation scene, when the electronic device predicts the frame image based on the MV calculation, it is very likely that the edge areas of static objects with discontinuous depth values ​​will be distorted in the predicted frame image.

[0050] In the technical solution provided in the embodiment of the present application, the edge area of ​​the static object in the game image is identified by the pixel depth value, the edge area is meshed based on the pixel depth information, and the image distortion processing is performed using the subdivided mesh geometric figures as units, so as to solve the problem of distortion of the edge area of ​​the static object in the game scene prediction frame image and enhance the user's gaming experience.

[0051] In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the image processing method according to the first aspect and any one of the first aspects.

[0052] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0053] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising a computer program, which, when executed on an electronic device, causes the electronic device to execute the image processing method according to the first aspect and any one of the first aspects.

[0054] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0055] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed, enables a computer to execute an image processing method as in the first aspect or any one of the first aspects.

[0056] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0057] In a fifth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the image processing method according to the first aspect or any one of the first aspects to control the receiving pin to receive a signal and to control the transmitting pin to send a signal.

[0058] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] FIG1a is a schematic diagram showing an exemplary frame insertion mode;

[0060] FIG1b is an exemplary diagram showing a predicted frame interpolation;

[0061] FIG2 is a schematic diagram showing an exemplary process of generating a predicted frame image;

[0062] FIG3 is a schematic diagram of an exemplary image gridding;

[0063] FIG4 is an exemplary depth image;

[0064] FIG5 is a schematic diagram illustrating an exemplary application scenario;

[0065] FIG6 is a schematic diagram showing an exemplary image grid distortion;

[0066] FIG7 is a schematic diagram illustrating a hardware structure of an electronic device;

[0067] FIG8 is a schematic diagram illustrating a software structure of an electronic device;

[0068] FIG9 is a schematic diagram illustrating the interaction of modules involved in an exemplary image processing method;

[0069] FIG10 is a schematic diagram illustrating an exemplary image distortion process;

[0070] FIG11 is a schematic diagram showing an exemplary grid standardization;

[0071] FIG12 is an exemplary flowchart of determining subdivided intersection points;

[0072] FIG13a is a schematic diagram showing an exemplary distribution of subdivided intersection points;

[0073] FIG13 b is a schematic diagram showing an exemplary grid subdivision;

[0074] FIG14 is a schematic diagram showing an exemplary grid movement;

[0075] FIG15a is a schematic diagram of one of the exemplary application scenarios;

[0076] FIG15 b is a schematic diagram showing one of the exemplary application scenarios. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0078] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0079] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0080] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0081] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0082] With the advancement of electronic technology, the refresh rates of smart device screens are increasing, and the frame rates of video sources are also increasing. This has led to an increase in memory usage and rendering power consumption in smart devices. However, due to the power consumption of smart devices or the limitations of their CPU and GPU capabilities, high refresh and frame rates often cause smart devices to overheat or experience lag, thus affecting the user experience.

[0083] Take mobile gaming as an example. With the development and popularity of large-scale mobile games, the rendering pipelines of modern mobile games are becoming increasingly complex, and the resource load is also increasing. At the same time, the computing power of modern smart terminals has not been able to meet the needs of modern large-scale mobile games, and smart terminals are limited by limited battery capacity and heat dissipation capabilities. Therefore, developers focus on how to reduce unnecessary rendering overhead, improve the limited frame rate and smoothness, and reduce heat generation without significantly affecting game quality.

[0084] To improve the user experience, developers often lower the frame rate of the video source and then increase it through video interpolation technology, thus halving the rendering time at the same frame rate. This significantly reduces the power consumption of smart terminals and reduces the heat generation of smart terminals.

[0085] Video interpolation technology aims to improve the frame rate and smoothness of videos, making them appear smoother. By inserting a predicted frame (also known as an intermediate frame or transition frame) between adjacent real frames (or original frames, etc.), the frame rate of the video can be doubled. The image quality of the predicted frame is directly related to the smoothness of the video. The image information of two adjacent real frames is used to calculate the movement direction and speed of each object in the image, and the objects in the image are moved accordingly to obtain the predicted frame.

[0086] As the name implies, real frames refer to the original image frames in the video source, not the image frames generated by prediction methods. For example, real frames can be image frames drawn by developers when developing applications such as games, or image frames captured by image acquisition devices (such as cameras) during video production.

[0087] As shown in FIG1a , the video interpolation method can be divided into predicted frame interpolation and predicted frame extrapolation. In the predicted frame interpolation method, as shown in FIG1a (1), a predicted frame image is calculated based on two adjacent real frame images, and the predicted frame image is interpolated between the two real frame images, thereby improving the video frame rate. In the predicted frame extrapolation method, as shown in FIG1a (2), a predicted frame image is calculated based on two adjacent real frame images, and the predicted frame image is interpolated after the two real frame images as the next frame image, thereby improving the video frame rate.

[0088] Taking the predicted frame interpolation method as an example, as shown in Figure 1b: predicted frame 1 is calculated based on real frame 1 and real frame 2, and is inserted between real frame 1 and real frame 2; predicted frame 2 is calculated based on real frame 2 and real frame 3, and is inserted between real frame 2 and real frame 3; predicted frame 3 is calculated based on real frame 3 and real frame 4, and is inserted between real frame 3 and real frame 4; and so on. In this way, predicted frame interpolation increases the frame rate of the video, making the transition between static and dynamic objects smoother and the picture more fluid.

[0089] The following briefly describes the process of generating a prediction frame using a game scene as an example. It should be noted that the relevant drawings shown in this embodiment are all shown in grayscale form.

[0090] As shown in Figure 2, the first real frame and the second real frame are two adjacent image frames in a game application. Assume that the second real frame is the current image frame and the first real frame is the image frame immediately preceding the second real frame. Therefore, the smart terminal can calculate a predicted frame based on the first and second real frames. Using predicted frame interpolation as an example, the smart terminal can insert the calculated predicted frame between the two real frames for display, thereby increasing the frame rate of the game video and achieving half the rendering speed at the same frame rate.

[0091] In this game application, the real frame includes image information and UI (User Interface) information. Among them, image information can be understood as the picture information in the video source, and UI information refers to the overall design information of the software's human-computer interaction, operation logic, and interface aesthetics. When the smart terminal calculates the predicted frame based on adjacent real frames, since the interface UI information will not change, the smart terminal can first separate the image information and UI information, and calculate the predicted frame only based on the image information, thereby reducing the amount of image data processing and improving the accuracy of the predicted frame. Therefore, as shown in Figure 2, when calculating the predicted frame, the smart terminal can first extract the first image 101 and the first UI 102 in the first real frame, and extract the second image 103 and the second UI 104 in the second real frame, and then perform an image prediction operation based on the first image 101 and the second image frame to obtain a predicted frame image 107.

[0092] 2 , when calculating the predicted frame image 107, the intelligent terminal may first calculate the motion vector (MV) 105 of the first image 101 and the second image 103, and perform image warping on the second image 103 based on the motion vector 105 to obtain the predicted frame image 106. Image warping refers to changing the position of image pixels through certain transformations, such as translation, rotation, scaling, affine, perspective, and cylindrical transformations.

[0093] Image warping can effectively ensure the continuous movement of objects in the image. However, due to the different movement speeds of objects in the image, image information may be overlapped or missing in the predicted frame image 106 obtained through image warping. Therefore, the smart terminal needs to perform image completion (Blur) on the predicted frame image 106 to obtain the predicted frame image 107. At this point, the smart terminal fuses the second UI 104 with the predicted frame image 107 to obtain the predicted frame 108 calculated based on the first and second real frames.

[0094] It should be noted that both the image distortion processing and the fusion UI information processing can be based on the information in the current real frame. Of course, the image distortion processing and the fusion UI information processing can also be based on the information of the previous frame of the current real frame, which is not limited in this embodiment.

[0095] Among them, in computer graphics, mesh geometry is usually used as the basic unit of image processing. Similarly, in the image distortion processing stage mentioned above, the smart terminal also performs image distortion processing based on mesh geometry as the basic unit. That is to say, the smart terminal performs image distortion processing by forming a geometric group of triangles or quadrilaterals in the image. Taking the quadrilateral mesh as an example, as shown in Figure 3, the image frame 20 is a global workgroup. A plurality of quadrilaterals 201 can be divided in the image frame 20, and each quadrilateral 201 includes a plurality of pixel points 2011. Among them, each quadrilateral 201 is a local workgroup, which is a basic unit for the smart terminal to perform image distortion processing.

[0096] However, each object displayed in the image has depth information, so the depth value of each pixel in the image is not the same. Among them, the depth value of a pixel refers to the vertical distance (depth) from the image collector (such as a camera, etc.) to this point in the image scene, and the value range is a floating point number of 0-1. Figure 4 shows an example of a depth image. A depth image refers to an image that uses the vertical distance (depth) from the image collector to each point in the scene as a pixel value. Since the depth value is just a floating point number, each pixel of the depth map only needs to store the floating point value of the r (red) channel, so the depth map we see is red and black (Figure 4 is shown in grayscale). For example, as shown in Figure 4, the closer the object is to the image collector, the redder the corresponding area on the depth map (the depth value is close to 1), and the farther it is, the darker it is (the depth value is close to 0). In other words, the depth value from 0 to 1 represents the distance from the image collector to the object from far to near.

[0097] In image areas where pixel depth values ​​are discontinuous, such as discontinuous depth values ​​for pixels within the same geometric shape in a mesh, using that geometric shape as a basic unit for image distortion can lead to localized image stretching or even abnormal object depth information, distorting the predicted frame. This can lead to significant differences between the predicted frame and the real frame, resulting in a poor user visual experience. For example, if the depth difference between a pixel and its surrounding pixels within a geometric shape in an image mesh exceeds a set threshold, the pixel depth value within that shape is discontinuous.

[0098] FIG5 exemplarily shows a comparison diagram of a real frame and a predicted frame. In the image scene shown in FIG5 , the edge area of ​​the pillar in the image is an area with discontinuous depth values, and the depth values ​​of the pixels in the pillar area are significantly greater than the depth values ​​of the pixels in the sky area. In the real frame shown in FIG5 (1), the edge of the pillar in the image is straight, while in the predicted frame shown in FIG5 (2), a local stretching problem occurs at the edge of the pillar in the image, and the depth values ​​of some pixels in the sky area are greater than the depth values ​​of the pixels in the pillar area (that is, the sky is closer to the image collector than the pillar, and the object depth information is abnormal). In this way, the predicted frame image has a distortion problem and is quite different from the real frame.

[0099] As a result, in the predicted frames generated after image distortion processing, image areas with discontinuous depth values ​​will have local stretching problems, or even abnormal object depth information problems, which are quite different from the real scene, resulting in a poor visual experience for users.

[0100] To address the above issues, an embodiment of the present application provides an image processing method. In this method, for image areas with discontinuous depth values, the intelligent terminal subdivides the image mesh based on the depth information of the pixels and performs image distortion processing using the subdivided image mesh geometry as units. This solves the problem of local stretching in image areas with discontinuous depth values ​​in the predicted frame image, better preserves continuous graphic information, and thus improves the picture quality of the predicted frame image, providing the user with a better visual experience.

[0101] In addition, when the smart terminal performs image distortion processing using the subdivided image Mesh geometry as a unit, it also combines the depth information of the pixel points for related processing to ensure that the objects in the predicted frame image can maintain the correct depth relationship and avoid the problem of abnormal object depth information.

[0102] Among them, the image processing method provided by the embodiment of the present application is suitable for image interpolation scenarios with large differences in depth information, such as game interpolation scenarios.

[0103] As we all know, in mobile games, the image scenes are relatively complex, there are many static objects (such as various buildings, etc.), and the depth differences of static objects in the scene are relatively large (that is, the depth values ​​of different static objects are discontinuous). In addition, since most games are virtual worlds, when users play mobile games, user operations often cause large-angle shaking of the game scene, unlike the physical world where the movement of objects follows the laws of physics. Therefore, in the case of complex game scenes, large-angle scene shaking may only cause part of the objects in the image to move more, rather than causing all objects in the image to move more. In other words, in the game scene, the distribution of MV images calculated based on continuous real frame images is uneven, especially in image areas with large differences in depth information, the difference in MV values ​​is also relatively large.

[0104] Therefore, in the game interpolation scenario, when the electronic device calculates the predicted frame image based on the MV, the predicted frame image is very likely to have distortion in the edge area of ​​the static object with discontinuous depth values.

[0105] FIG6 (1) shows an example of a grayscale image of a motion vector frame calculated based on a game image. In the grayscale image shown in FIG6 (1), the lighter the grayscale (the closer the grayscale value is to 1), the smaller the motion vector is, and the darker the grayscale (the closer the grayscale value is to 0), the larger the motion vector is. Continuing to refer to FIG6 (1), the motion vectors on the left and right sides of the edge area of ​​the static object in the image frame vary greatly. Combining FIG6 (1) and (2), taking area 301 as an example, sub-area 3011 corresponds to the column area in the image, and sub-area 3012 corresponds to the sky area in the image. The motion vector corresponding to the column is smaller than the motion vector corresponding to the sky. In other words, the motion vector values ​​on the left and right sides of the column edge area vary greatly. In this case, once the motion vectors corresponding to different pixel areas in the same image mesh geometry differ greatly, image distortion processing of the mesh geometry based on the motion vector will inevitably lead to serious deformation of the mesh geometry, and even the problem of overlapping of some mesh geometry, and the accuracy of the depth information of the object in the image cannot be guaranteed. As shown in (3) of Figure 6, among the four vertices of Mesh geometry 201, the motion vectors corresponding to V0 and V1 are significantly different from the motion vectors corresponding to V2 and V3. After moving each vertex according to its corresponding motion vector, Mesh geometry 201 is distorted into Mesh geometry 201'. Not only is the Mesh geometry severely deformed, but it may also cover other Mesh geometry. This will cause distortion in some areas of the predicted frame image, especially the edge areas of static objects (i.e., pixel areas with discontinuous depth values), as shown in (4) of Figure 6.

[0106] In the technical solution provided in the embodiment of the present application, the edge area of ​​the static object in the game image is identified by the pixel depth value, the edge area is meshed based on the pixel depth information, and the image distortion processing is performed using the subdivided mesh geometric figures as units, so as to solve the problem of distortion of the edge area of ​​the static object in the game scene prediction frame image and enhance the user's gaming experience.

[0107] As shown in Figure 7, a schematic diagram of the structure of the electronic device 100 is shown. Optionally, the electronic device 100 can be a terminal, which can also be called a terminal device, a smart terminal, etc. The terminal can be a device such as a cellular phone or a tablet computer, which is not limited in this application. It should be understood that the electronic device 100 shown in Figure 7 is only an example of an electronic device, and the electronic device 100 can have more or fewer components than shown in the figure, can combine two or more components, or can have different component configurations. The various components shown in Figure 6 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0108] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0109] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0110] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0111] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0112] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0113] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0114] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0115] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0116] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0117] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0118] The wireless communication module 160 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0119] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0120] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0121] In embodiments of the present application, when generating a predicted frame image, the GPU can subdivide the image grid based on the image processing method provided in embodiments of the present application, and perform image warping based on the subdivided image grid to generate the predicted frame image. The GPU can also perform image completion processing on the predicted frame to achieve the effect of quickly completing the predicted frame image.

[0122] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0123] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0124] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0125] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0126] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0127] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the image processing method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0128] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0129] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0130] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through speaker 170A. In some embodiments, electronic device 100 can be provided with multiple speakers 170A.

[0131] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0132] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0133] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0134] The pressure sensor is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor.

[0135] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor.

[0136] The accelerometer can detect the magnitude of acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the accelerometer can detect the magnitude and direction of gravity. The accelerometer can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0137] A touch sensor, also known as a "touch panel," can be mounted on display screen 194. The touch sensor and display screen 194 together form a touch screen, also known as a "touch screen." The touch sensor detects touch operations applied to or near the touch sensor. The touch sensor can communicate the detected touch operations to the application processor to determine the type of touch event.

[0138] Keys 190 include a power button (also known as a power button), a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. Electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100.

[0139] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or touch vibration feedback. For example, touch operations on different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects.

[0140] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0141] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0142] FIG8 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0143] The layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0144] The application layer can include a series of application packages.

[0145] As shown in FIG8 , the application package may include applications such as games, videos, cameras, gallery, WLAN, Bluetooth, calls, calendars, maps, and navigation.

[0146] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0147] As shown in FIG8 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0148] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0149] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0150] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0151] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0152] The Notification Manager allows applications to display notifications in the status bar. These messages can be used to convey notifications and disappear automatically after a short pause, without requiring user interaction. For example, notifications are used to notify the completion of downloads, message reminders, etc. Notifications can also appear in the form of icons or scrolling text in the top status bar of the system, such as notifications from applications running in the background, or in the form of dialog windows on the screen. Notifications can also be text messages displayed in the status bar, beeps, vibrations, or flashing indicator lights emitted by electronic devices.

[0153] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0154] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0155] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0156] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0157] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), an image extraction module, an image synthesis module, a predicted frame generation module, and a predicted frame completion module.

[0158] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0159] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0160] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0161] A 2D graphics engine is a drawing engine for 2D drawings.

[0162] In an embodiment of the present application, the prediction frame generation module can be used to generate a prediction frame image based on information of two adjacent frames of image. As shown in FIG8 , the prediction frame generation module can include an MV calculation submodule and an image distortion submodule.

[0163] The MV calculation submodule may be used to calculate the MV based on two consecutive image frames.

[0164] The image distortion submodule can be used to perform image distortion processing on the image according to the MV. In an embodiment of the present application, the image distortion submodule can perform image distortion processing on static objects and dynamic objects in the image respectively. When performing image distortion processing on static objects, the image distortion submodule can subdivide the image Mesh based on the depth information of the pixel points, and perform image distortion processing on the subdivided image Mesh geometry as a unit. Moreover, when the image distortion submodule performs image distortion processing on the subdivided image Mesh geometry as a unit, it will also perform relevant processing in combination with the depth information of the pixel points, and overlay the image Mesh geometry with a high pixel depth value on the image Mesh geometry with a low pixel depth value, so that each object in the predicted frame image can maintain the correct depth relationship.

[0165] The prediction frame completion module can be used to perform completion processing on the blank areas in the prediction frame image generated by the prediction frame generation module, such as fuzzy completion processing, to obtain the prediction frame image after image completion processing.

[0166] The image extraction module can be used to separate image information and UI information in the resource image frame to obtain an image frame, which is used for intermediate frame prediction.

[0167] The image extraction module may also be integrated into a three-dimensional graphics processing library, which is not limited in this embodiment.

[0168] The image synthesis module can be used to fuse the predicted frame image after image completion processing and UI information to obtain the predicted frame to be displayed.

[0169] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, audio drivers, Wi-Fi drivers, sensor drivers, etc. Hardware includes at least processors, displays, Wi-Fi modules, sensors, etc.

[0170] It is understood that the layers in the software structure shown in FIG8 and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not limit this.

[0171] It is understandable that, in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0172] FIG9 is a schematic diagram of the interaction between the modules. Referring to FIG9 , in the game interpolation scenario, the image processing method provided by the embodiment of the present application specifically includes:

[0173] S401, the game APP sends a resource image to the 3D graphics processing library.

[0174] Resource images refer to the image data in the game app, which are intended to be displayed in the game app interface. Resource images include image information and UI information.

[0175] Among them, UI information refers to the overall design information of the software's human-computer interaction, operating logic, and interface aesthetics.

[0176] The image information includes but is not limited to the GRB (Green, Red, Blue) value and depth value of each pixel.

[0177] S402, the three-dimensional image processing library renders the resource images of the game APP.

[0178] The process of rendering the resource image by the three-dimensional image processing library can refer to the existing technology and will not be described in detail here.

[0179] S403: The image extraction module intercepts the rendered resource image, extracts the real frame image and UI information from the resource image, and stores the real frame image and UI information in correspondence.

[0180] Among them, in most cases, UI information does not change, and the display position in each frame is also fixed and does not change. For example, in a game application interface, UI information may include user operation controls and the like.

[0181] When an electronic device predicts an intermediate frame image based on adjacent real frame images, since the interface UI information will not change, the image information and UI information can be first separated through the image extraction module, and the intermediate frame prediction can be performed only based on the image information, thereby reducing the amount of image data processing and improving the accuracy of the intermediate frame image prediction.

[0182] In some special circumstances, the UI information may change. The electronic device may also separate the image information and the UI information through the image extraction module, and perform intermediate frame prediction based on the image information. After obtaining the intermediate frame image, it can be merged with the UI information of the current real frame image. This embodiment does not limit this.

[0183] The image extraction module identifies the real frame image (ie, the game scene image) and UI information in the rendered resource image through features, and draws the real frame image into a private image frame cache.

[0184] At the same time, the image extraction module also stores the matched UI information. For example, the UI information may include but is not limited to the UI image content and the display position information of the UI image.

[0185] S404: The MV calculation submodule calculates a motion vector based on two adjacent real frame images.

[0186] The two adjacent real frame images may be a current real frame image and a real frame image preceding the current real frame image, such as the real frame image 103 and the real frame image 101 shown in FIG. 2 .

[0187] The MV calculation submodule reads two adjacent real frame images in the image frame buffer and calculates the motion vectors corresponding to the two real frame images.

[0188] Among them, the MV calculation submodule can select a matching calculation method to calculate the motion vector corresponding to each image area according to the object attributes (such as static objects, dynamic objects) pre-annotated in each image area in the real frame image, so as to obtain the motion vector corresponding to the current frame image, such as the motion vector frame 105 shown in Figure 2.

[0189] In an optional embodiment, some objects in the image (such as people, etc.) are pre-labeled with dynamic object (i.e., objects in motion) attributes, and other unlabeled objects in the image (such as the sky, pillars, etc.) are static objects.

[0190] For example, for image regions corresponding to static objects, the MV calculation submodule can use a reprojection method to calculate the corresponding motion vectors; for image regions corresponding to dynamic objects, the MV calculation submodule can use an optical flow method to calculate the corresponding motion vectors. The steps for calculating motion vectors using the reprojection and optical flow methods can be referenced in existing technologies and will not be further described here.

[0191] S405 , the image warping submodule performs image warping on the current frame image based on the motion vector to obtain a first predicted frame image, and sends the first predicted frame image to the predicted frame completion module.

[0192] After the MV calculation submodule calculates the motion vector corresponding to the current frame image based on the current frame image and the previous frame image of the current frame image, the image distortion submodule performs image distortion on the current frame image based on the motion vector to obtain a first predicted frame image.

[0193] It should be noted that due to the varying movement speeds of various objects in the video source, the predicted frame image calculated by the image warping submodule based on the motion vectors may contain overlapping or missing image information. Specifically, the first predicted frame image is the predicted frame image to be completed, such as predicted frame image 106 shown in FIG2 .

[0194] Among them, when the image distortion submodule distorts the current frame image based on the motion vector, it first performs image distortion processing on the image area corresponding to the static object in the current frame image based on the motion vector, and then performs image distortion processing on the image area corresponding to the dynamic object in the current frame image based on the motion vector, so as to obtain the first predicted frame image.

[0195] As an optional implementation, S405 may specifically include:

[0196] S4051, the image warping submodule creates and subdivides the mesh of the current frame image, and performs image warping processing on the static objects in the current frame image based on the motion vector using the subdivided mesh geometry as a processing unit.

[0197] S4052: The image warping submodule performs image warping processing on the dynamic object in the current frame image based on the motion vector using the created and standardized Mesh geometry as a processing unit.

[0198] After the image warping submodule creates the mesh for the current frame image, it can be normalized to facilitate geometric processing. Therefore, the image warping submodule uses the normalized mesh geometry as a processing unit based on motion vectors to warp dynamic objects in the current frame image. Details on image mesh creation and normalization are provided below and will not be elaborated here.

[0199] Because image regions with discontinuous depth values ​​are typically located at the edges of static objects in the image, the image warping submodule first subdivides the mesh geometry corresponding to the static object in the current frame image based on the motion vector, subdividing the mesh geometry based on the depth information of the pixels. This ensures that the depth values ​​of the pixels corresponding to the subdivided mesh geometry are continuous. This means that within each subdivided mesh geometry, the depth values ​​of the pixels are continuous, or the depth values ​​of the pixels have minimal variance.

[0200] When the image warping submodule performs image warping on the subdivided grid based on the motion vector, it can better preserve the continuous graphic information of the static object to avoid the distortion problem in the edge area of ​​the static object.

[0201] It should be noted that motion vector-based mesh warping involves moving each mesh vertex according to the motion vector corresponding to that vertex. Whether moving static objects using refined meshes or moving dynamic objects using unrefined meshes, the processing is the same; the only difference lies in the degree of mesh refinement.

[0202] As an optional implementation, S4051 can be implemented as S501 to S509. As shown in FIG10 , the image distortion submodule performs image distortion processing on a static object in the current frame image, specifically including:

[0203] S501: The image warping submodule creates a grid of the current image frame.

[0204] The image warping submodule creates a mesh of the current image frame, which can also be called meshing the current image frame, in order to divide the current frame image into geometric shapes that are easier to process, such as polygons, triangles or quadrilaterals.

[0205] Among them, the image distortion submodule creates a Mesh of the current image frame with the same size as the current image frame, so that each graphic filled in the Mesh can cover a pixel area of ​​the current image frame.

[0206] For example, in the embodiment of the present application, the image warping submodule may create a quadrilateral grid of the current image frame, that is, grid the current image frame according to the size of the quadrilateral, wherein the quadrilateral may be a square.

[0207] S502: The image distortion submodule performs a standardization process on the grid.

[0208] Among them, if the image distortion submodule creates a non-triangular mesh for the current image frame, such as a quadrilateral mesh, the image distortion submodule can standardize the non-triangular mesh so that each non-triangle in the mesh is divided into multiple triangles to facilitate graphics calculation.

[0209] Exemplarily, the image warping submodule creates a quadrilateral mesh of the current image frame, and the image warping submodule may further divide each quadrilateral in the quadrilateral mesh into two triangles to achieve mesh standardization.

[0210] Each quadrilateral or triangle in the quadrilateral mesh can be indexed by vertex. For example, to facilitate computational processing, the vertices of each quadrilateral or triangle can be sorted in a clockwise direction to normalize the vertex index of the image mesh.

[0211] As shown in (1) of FIG11 , each quadrilateral 201 in the current image frame grid can be divided into triangle 201_1 and triangle 201_2. As shown in (2) of FIG11 , taking quadrilateral Square (V0-V1-V2-V3) as an example, the quadrilateral can be divided into triangle Tringle1 (V0-V1-V2) and triangle Tringle2 (V2-V3-V0). In triangle Tringle1, V0 is the starting vertex and V2 is the ending vertex; in triangle Tringle2, V2 is the starting vertex and V0 is the ending vertex.

[0212] It should be noted that in the process of standardizing the quadrilateral Square (V0-V1-V2-V3), the starting vertex V0 of the first triangle (i.e., Tringle1 (V0-V1-V2)) is the starting vertex V0 of the quadrilateral Square (V0-V1-V2-V3), and the starting side V0-V1 of the first triangle is the starting side V0-V1 of the quadrilateral. At the same time, the ending vertex V2 of the first triangle is the starting vertex V2 of the second triangle (i.e., Tringle2 (V2-V3-V0)), and the ending vertex V0 of the second triangle is the starting vertex V0 of the first triangle.

[0213] S503: The image distortion submodule sequentially uses each geometric figure before the Mesh normalization process as a current processing figure (or referred to as a current processing Mesh geometric figure).

[0214] The image warping submodule uses each geometric figure created before the normalization process as a processing unit to determine whether mesh subdivision is required and how to perform mesh subdivision and image warping. In other words, the image warping submodule sequentially determines whether mesh subdivision is required for each quadrilateral before the normalization process and how to perform mesh subdivision and image warping. In the mesh division example shown in FIG11 (1), the image warping submodule sequentially determines whether mesh subdivision is required for each quadrilateral and how to perform mesh subdivision and image warping.

[0215] It should be noted that the image warping submodule can also concurrently determine whether each geometric figure before normalization needs to be meshed and how to perform meshing and image warping. For each geometric figure, the determination and processing flow of the image warping submodule can refer to S504 to S508.

[0216] S504, the image distortion submodule determines whether the currently processed graphic corresponds to a static object in the current image frame, if so, execute S505, if not, execute S509.

[0217] In the Mesh of the current image frame, this embodiment only needs to subdivide the geometric figures corresponding to the static objects so that the pixel depth values ​​in each geometric figure corresponding to the static object are continuous. Therefore, the image distortion submodule needs to determine whether the currently processed figure corresponds to a static object.

[0218] The image warping submodule determines whether the currently processed shape corresponds to a static object based on pre-annotated image information. If not, it continues to sequentially determine the next geometric shape in the mesh until all geometric shapes in the mesh of the current image frame have been processed. If so, the image warping submodule continues to determine whether the currently processed shape meets the mesh subdivision requirements.

[0219] S505: The image distortion submodule determines the subdivision intersection of the currently processed graphic.

[0220] When determining whether the current processed graphic meets the grid subdivision condition, the image distortion submodule determines whether there are subdivision intersections corresponding to the current processed graphic, and further determines whether the current processed graphic meets the grid subdivision condition based on the number of subdivision intersections.

[0221] The subdivision intersection is the pixel point used to subdivide the geometric shape in the mesh. This pixel point is located on the edge of the geometric shape. For example, in a quadrilateral mesh, the subdivision intersection needs to be located on any edge of the quadrilateral.

[0222] FIG12 exemplarily shows the process of determining the subdivision intersection. As shown in FIG12 , the specific process of the image distortion submodule determining the subdivision intersection of the current processing image includes:

[0223] S5051: The image distortion submodule sequentially uses each pixel point in the current processing image as the current processing pixel point.

[0224] The image distortion submodule determines whether each pixel point in the current processing graph is a subdivision intersection point of the current processing graph.

[0225] It should be noted that the image distortion submodule can also determine in parallel whether each pixel point in the current processing graph is a subdivision intersection point of the current processing graph. For each pixel point in each current processing graph, the judgment process of the image distortion submodule can refer to S5052 to S5056.

[0226] S5052, the image distortion submodule determines whether the depth value of the currently processed pixel is within a preset depth range, if so, executes S5053, if not, executes S5055.

[0227] The preset depth range may be understood as a depth range of interest, within which edges of static objects are likely to be distorted when the image is distorted.

[0228] S5053, the image distortion submodule determines whether the depth difference between the currently processed pixel and the surrounding pixels exceeds a preset threshold. If so, execute S5054; if not, execute S5055.

[0229] If the depth difference between the currently processed pixel and the surrounding pixels exceeds a preset threshold, it indicates that the depth value of the currently processed pixel is significantly different from that of its surrounding pixels, and the currently processed pixel may be located on the edge of a static object.

[0230] S5054: The image distortion submodule marks the currently processed pixel as an edge point corresponding to the currently processed graphic.

[0231] Among them, edge points, also called static object edge points, are pixel points on the edge of a static object.

[0232] S5055: The image distortion submodule determines whether all pixels in the currently processed image have been processed.

[0233] The image distortion submodule first identifies each edge point among all the pixels of the currently processed image, and then determines whether there is a subdivision intersection of the currently processed image among each edge point.

[0234] For any pixel in the currently processed image, if its depth value is within a preset depth range and the depth difference between it and any surrounding pixel values ​​exceeds a preset threshold, the image distortion submodule determines that the pixel is a static object edge point.

[0235] S5056: The image distortion submodule determines, from the edge points of the static objects corresponding to the current processing graphic, pixel points located on the edge of the current processing graphic as subdivision intersection points of the current processing graphic.

[0236] In the current processing graph, after the image distortion submodule identifies each static object edge point, it determines whether each static object edge point is located on the edge of the current processing graph. If so, the static object edge point is used as a subdivision intersection point of the current processing graph.

[0237] For example, the image distortion submodule can determine whether each static object edge point is located on the edge of the current processing graphic based on a compute shader calculation. Detailed explanations of the compute shader can be found in existing technologies and will not be repeated here.

[0238] In a possible implementation, the image warping submodule may also determine the subdivision intersection points corresponding to each geometric figure after determining the edge points of the static objects corresponding to each geometric figure.

[0239] Exemplarily, after determining each static object edge point corresponding to each geometric figure, the image warping submodule may generate a mask image of the static object edge points. In the mask image of the static object edge points, the grayscale value of the static object edge points is 1, and the grayscale values ​​of the remaining pixels are 0. Furthermore, based on the created Mesh of the current image frame, the image warping submodule may also determine a mask image of the geometric figure edges in the Mesh. In the mask image of the geometric figure edges, the grayscale values ​​of the pixels on the geometric figure edges are 1, and the grayscale values ​​of the remaining pixels are 0.

[0240] The image warping submodule calculates the static object edge points located on the edges of the geometric figure, also known as the segmentation focus, based on the mask image of the geometric figure and the calculated mask image of the static object edge points. The image warping submodule ANDs the mask image of the static object edge points with the mask image of the geometric figure edge to obtain the mask image of the segmentation intersection. In the mask image of the segmentation intersection, the grayscale value of the segmentation intersection is 1, while the grayscale values ​​of the remaining pixels are 0. This allows the image warping submodule to record the coordinates of the segmentation intersection corresponding to each geometric figure in the mesh.

[0241] It should be pointed out that, due to different image scenes, the number of subdivision intersections of different geometric figures in the Mesh is not necessarily the same. Taking the quadrilateral in the quadrilateral Mesh as an example, the number of its subdivision intersections can be any value of 1, 2, 3, or 4. As shown in Figure 13a, (1), (2), and (3) in Figure 13a respectively illustrate the case where the number of subdivision intersections of the quadrilateral in the Mesh is 2, (4) in Figure 13a illustrates the case where the number of subdivision intersections of the quadrilateral in the Mesh is 1, (5) in Figure 13a illustrates the case where the number of subdivision intersections of the quadrilateral in the Mesh is 4, and (6) in Figure 13a illustrates the case where the number of subdivision intersections of the quadrilateral in the Mesh is 4. For some special scenes, the number of subdivision intersections of the geometric figures in the Mesh can also be other values, and the number of subdivision intersections on each edge is not necessarily only one.

[0242] S506, the image distortion submodule determines whether the currently processed graphic meets the grid subdivision condition according to the number of subdivision intersections, and if so, executes S507, and if not, executes S508.

[0243] In this embodiment of the present application, the mesh subdivision condition can be that the number of subdivision intersections of the geometric figure in the mesh is two, and the two subdivision intersections are located on different edges of the geometric figure. In other words, the mesh subdivision condition can be that the geometric figure has exactly two subdivision intersections, and each edge of the geometric figure has at most one subdivision intersection.

[0244] S507 , the image distortion submodule subdivides the current processing graph according to the subdivided intersection points, and performs image distortion processing on the subdivided current processing graph according to the motion vector.

[0245] If the number of subdivision intersections of the current processing graphic is 2, and the two subdivision intersections are located on different edges of the current processing graphic, the current processing graphic meets the grid subdivision conditions. The image distortion submodule subdivides the current processing graphic according to the two subdivision intersections and performs image distortion processing on the standardized and subdivided current processing graphic.

[0246] When the image warping submodule subdivides the current processing graph according to the two subdivision intersection points, it can select a matching subdivision strategy to subdivide the corresponding triangle according to the coordinate relationship between the two subdivision focal points and each triangle vertex of the current processing graph after normalization.

[0247] The following explains various mesh subdivision scenarios in conjunction with the standardized mesh shown in (2) of Figure 11. In the standardized mesh shown in (2) of Figure 11, the quadrilateral Square (V0-V1-V2-V3) is divided into the first triangle (i.e., triangle Tringle1 (V0-V1-V2)) and the second triangle (i.e., triangle Tringle2 (V2-V3-V0)).

[0248] Scenario 1

[0249] In this case, only the first triangle needs to be meshed separately, and the second triangle does not need to be meshed.

[0250] Of the two subdivision intersection points, the abscissa of the first subdivision intersection point is equal to the abscissas of the first vertex (or starting vertex) and the second vertex (or intermediate vertex) of the first triangle, and the ordinate of the second subdivision intersection point is equal to the ordinates of the second vertex and the third vertex (or ending vertex) of the first triangle. In this case, the image warping submodule needs to perform mesh subdivision on the first triangle based on these two subdivision intersection points, without performing mesh subdivision on the second triangle.

[0251] In one embodiment, the image warping submodule may connect the first subdivision intersection point and the second subdivision intersection point, and connect the first subdivision intersection point and the third vertex of the first triangle to perform mesh subdivision on the first triangle to obtain a plurality of subdivided triangles.

[0252] As shown in (1) of Figure 13b, the horizontal coordinate of the first subdivision intersection point Point2 is equal to the horizontal coordinates of the first vertex V0 and the second vertex V1 of the first triangle, and the vertical coordinate of the second subdivision intersection point Point1 is equal to the vertical coordinates of the second vertex V1 and the third vertex V2 of the first triangle. In this case, the image warping submodule needs to mesh the first triangle based on the subdivision intersection points Point2 and Point1, without meshing the second triangle.

[0253] Continuing with reference to (1) in FIG13b , the image warping submodule connects the first subdivision intersection Point2 with the second subdivision intersection Point1, and connects the first subdivision intersection Point2 with the third vertex V2 of the first triangle to subdivide the first triangle Tringle1 (V0-V1-V2) into: a first subdivision triangle S-Tringle1 (V0-Point2-V2), a second subdivision triangle S-Tringle2 (Point2-Point1-V2) and a third subdivision triangle S-Tringle3 (Point2-V1-Point1).

[0254] In another embodiment, the image warping submodule may also connect the first subdivision intersection point and the second subdivision intersection point, and connect the second subdivision intersection point and the first vertex of the first triangle to perform mesh subdivision on the first triangle to obtain multiple subdivided triangles.

[0255] As shown in (2) in Figure 13b, the image warping submodule connects the first subdivision intersection Point2 and the second subdivision intersection Point1, and connects the second subdivision intersection Point1 and the first vertex V0 of the first triangle to subdivide the first triangle Tringle1 (V0-V1-V2) into: the first subdivision triangle S-Tringle1 (V0-Point2-Point1), the second subdivision triangle S-Tringle2 (Point2-V1-Point1) and the third subdivision triangle S-Tringle3 (Point1-V2-V0).

[0256] The vertex indices of each subdivided triangle are also sorted clockwise. The same is true for the following situations, which will not be described in detail below.

[0257] In the subdivision method shown in (1) and (2) in Figure 13b, the starting edge (V0-Point2) of the first subdivided triangle S-Tringle1 is located on the starting edge (V0-V1) of the first triangle, that is, on the starting edge (V0-V1) of the quadrilateral, which does not affect the geometric processing order of the first triangle and facilitates developers to perform subdivision programming.

[0258] Scenario 2

[0259] In this case, both the first triangle and the second triangle need to be meshed together.

[0260] (1) Of the two subdivision intersections, the ordinate of the first subdivision intersection is equal to the ordinates of the second and third vertices of the first triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second and third vertices of the second triangle. In this case, the image warping submodule needs to perform mesh subdivision on both the first and second triangles based on the two subdivision intersections.

[0261] Among them, the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, and connect the first subdivision intersection and the first vertex of the first triangle, and connect the second subdivision intersection and the first vertex of the second triangle to mesh the first triangle and the second triangle to obtain multiple subdivided triangles.

[0262] It should be pointed out that when the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, the line connecting the first subdivision intersection and the second subdivision intersection intersects with the terminal edge of the first triangle (that is, the terminal edge of the second triangle), and the first triangle and the second triangle can be meshed at the same time.

[0263] As shown in (3) of Figure 13b, the ordinate of the first subdivision intersection point Point1 is equal to the ordinates of the second vertex V1 and the third vertex V2 of the first triangle, and the ordinate of the second subdivision intersection point Point2 is equal to the ordinates of the second vertex V3 and the third vertex V0 of the second triangle. At this point, the image warping submodule needs to perform mesh subdivision on the first and second triangles based on the subdivision intersection points Point1 and Point2.

[0264] Continuing with reference to (3) in FIG13b , the image warping submodule connects the first subdivision intersection Point1 to the second subdivision intersection Point2, connects the first subdivision intersection Point1 to the first vertex V0 of the first triangle, and connects the second subdivision intersection Point2 to the first vertex V2 of the second triangle. The line connecting the first subdivision intersection Point1 and the second subdivision intersection Point2 intersects the terminal edge (V2-V0) of the first triangle (that is, the terminal edge (V0-V2) of the second triangle) at PointInter. Thus, the first triangle Tringle1 (V0-V1-V2) is subdivided into: the first subdivision triangle S-Tringle1 (V0-V1-Point1), the second subdivision triangle S-Tringle2 (V0-Point1-PointInter), and the third subdivision triangle S-Tringle3 (PointInter-Point1-V2). The second triangle Tringle2 (V2-V3-V0) is subdivided into a first subdivided triangle S-Tringle1 (V2-V3-Point2), a second subdivided triangle S-Tringle2 (V2-Point2-PointInter), and a third subdivided triangle S-Tringle3 (PointInter-Point2-V0).

[0265] (2) Of the two subdivision intersections, the ordinate of the first subdivision intersection is equal to the ordinates of the second and third vertices of the first triangle, and the abscissa of the second subdivision intersection is equal to the abscissas of the first and second vertices of the second triangle. In this case, the image warping submodule needs to perform mesh subdivision on both the first and second triangles based on the two subdivision intersections.

[0266] Among them, the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, and connect the first subdivision intersection and the first vertex of the first triangle, and connect the second subdivision intersection and the third vertex of the second triangle to mesh the first triangle and the second triangle to obtain multiple subdivided triangles.

[0267] It should be pointed out that when the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, the line connecting the first subdivision intersection and the second subdivision intersection intersects with the terminal edge of the first triangle (that is, the terminal edge of the second triangle), and the first triangle and the second triangle can be meshed at the same time.

[0268] As shown in (4) of Figure 13b, the ordinate of the first subdivision intersection point Point1 is equal to the ordinates of the second vertex V1 and the third vertex V2 of the first triangle, and the abscissa of the second subdivision intersection point Point2 is equal to the abscissas of the first vertex V2 and the second vertex V3 of the second triangle. At this point, the image warping submodule needs to perform mesh subdivision on the first and second triangles based on the subdivision intersection points Point1 and Point2.

[0269] Continuing with reference to (4) in FIG13b , the image warping submodule connects the first subdivision intersection Point1 to the second subdivision intersection Point2, connects the first subdivision intersection Point1 to the first vertex V0 of the first triangle, and connects the second subdivision intersection Point2 to the third vertex V0 of the second triangle. The line connecting the first subdivision intersection Point1 and the second subdivision intersection Point2 intersects the terminal edge (V2-V0) of the first triangle (that is, the terminal edge (V0-V2) of the second triangle) at PointInter. Thus, the first triangle Tringle1 (V0-V1-V2) is subdivided into: the first subdivision triangle S-Tringle1 (V0-V1-Point1), the second subdivision triangle S-Tringle2 (V0-Point1-PointInter), and the third subdivision triangle S-Tringle3 (PointInter-Point1-V2). The second triangle Tringle2 (V2-V3-V0) is subdivided into a first subdivided triangle S-Tringle1 (V2-Point2-PointInter), a second subdivided triangle S-Tringle2 (Point2-V0-PointInter), and a third subdivided triangle S-Tringle3 (Point2-V3-V0).

[0270] (3) Of the two subdivision intersections, the horizontal coordinate of the first subdivision intersection is equal to the horizontal coordinates of the first and second vertices of the first triangle, and the horizontal coordinate of the second subdivision intersection is equal to the horizontal coordinates of the first and second vertices of the second triangle. In this case, the image warping submodule needs to perform mesh subdivision on the first and second triangles based on the two subdivision intersections.

[0271] Among them, the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, and connect the first subdivision intersection and the third vertex of the first triangle, and connect the second subdivision intersection and the third vertex of the second triangle to mesh the first triangle and the second triangle to obtain multiple subdivided triangles.

[0272] It should be pointed out that when the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, the line connecting the first subdivision intersection and the second subdivision intersection intersects with the terminal edge of the first triangle (that is, the terminal edge of the second triangle), and the first triangle and the second triangle can be meshed at the same time.

[0273] As shown in (5) of Figure 13b, the horizontal coordinate of the first subdivision intersection point Point1 is equal to the horizontal coordinates of the first vertex V0 and the second vertex V1 of the first triangle, and the horizontal coordinate of the second subdivision intersection point Point2 is equal to the horizontal coordinates of the first vertex V2 and the second vertex V3 of the second triangle. At this time, the image warping submodule needs to mesh the first triangle and the second triangle based on the subdivision intersection points Point1 and Point2.

[0274] Continuing with reference to (5) in FIG13b , the image warping submodule connects the first subdivision intersection Point1 to the second subdivision intersection Point2, connects the first subdivision intersection Point1 to the third vertex V2 of the first triangle, and connects the second subdivision intersection Point2 to the third vertex V0 of the second triangle. The line connecting the first subdivision intersection Point1 and the second subdivision intersection Point2 intersects the terminal edge (V2-V0) of the first triangle (that is, the terminal edge (V0-V2) of the second triangle) at PointInter. Thus, the first triangle Tringle1 (V0-V1-V2) is subdivided into: the first subdivision triangle S-Tringle1 (V0-Point1-PointInter), the second subdivision triangle S-Tringle2 (Point1-V2-PointInter), and the third subdivision triangle S-Tringle3 (Point1-V1-V2). The second triangle Tringle2 (V2-V3-V0) is subdivided into a first subdivided triangle S-Tringle1 (V2-Point2-PointInter), a second subdivided triangle S-Tringle2 (Point2-V0-PointInter), and a third subdivided triangle S-Tringle3 (Point2-V3-V0).

[0275] (4) Of the two subdivision intersections, the horizontal coordinate of the first subdivision intersection is equal to the horizontal coordinates of the first and second vertices of the first triangle, and the vertical coordinate of the second subdivision intersection is equal to the vertical coordinates of the second and third vertices of the second triangle. In this case, the image warping submodule needs to perform mesh subdivision on the first and second triangles based on the two subdivision intersections.

[0276] Among them, the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, and connect the first subdivision intersection and the third vertex of the first triangle, and connect the second subdivision intersection and the first vertex of the second triangle to mesh the first triangle and the second triangle to obtain multiple subdivided triangles.

[0277] It should be pointed out that when the image distortion submodule can connect the first subdivision intersection and the second subdivision intersection, the line connecting the first subdivision intersection and the second subdivision intersection intersects with the terminal edge of the first triangle (that is, the terminal edge of the second triangle), and the first triangle and the second triangle can be meshed at the same time.

[0278] As shown in (6) of Figure 13b, the horizontal coordinate of the first subdivision intersection point Point1 is equal to the horizontal coordinates of the first vertex V0 and the second vertex V1 of the first triangle, and the vertical coordinate of the second subdivision intersection point Point2 is equal to the vertical coordinates of the second vertex V3 and the third vertex V0 of the second triangle. At this time, the image warping submodule needs to mesh the first triangle and the second triangle based on the subdivision intersection points Point1 and Point2.

[0279] Continuing with reference to (6) in FIG13b , the image warping submodule connects the first subdivision intersection Point1 to the second subdivision intersection Point2, connects the first subdivision intersection Point1 to the third vertex V2 of the first triangle, and connects the second subdivision intersection Point2 to the first vertex V2 of the second triangle. The line connecting the first subdivision intersection Point1 and the second subdivision intersection Point2 intersects the terminal edge (V2-V0) of the first triangle (that is, the terminal edge (V0-V2) of the second triangle) at PointInter. Thus, the first triangle Tringle1 (V0-V1-V2) is subdivided into: the first subdivision triangle S-Tringle1 (V0-Point1-PointInter), the second subdivision triangle S-Tringle2 (Point1-V2-PointInter), and the third subdivision triangle S-Tringle3 (Point1-V1-V2). The second triangle Tringle2 (V2-V3-V0) is subdivided into: a first subdivided triangle S-Tringle1 (V2-V3-Point2), a second subdivided triangle S-Tringle2 (Point2-PointInter-V2) and a third subdivided triangle S-Tringle3 (Point2-V0-PointInter).

[0280] In the subdivision method shown in (3) to (6) in Figure 13b, two subdivision intersections jointly subdivide two triangles. The starting edge of the first subdivided triangle obtained by subdividing the first triangle is located on the starting edge (V0-V1) of the first triangle, that is, on the starting edge (V0-V1) of the quadrilateral; the starting edge of the first subdivided triangle obtained by subdividing the second triangle is also located on the starting edge (V0-V1) of the second triangle. This mesh subdivision does not affect the geometric processing order of the triangles before subdivision, making it easier for developers to perform subdivision programming.

[0281] Scenario 3

[0282] In this case, there is no need to mesh the first triangle, only the second triangle needs to be meshed separately.

[0283] Of the two subdivision intersection points, the abscissa of the first subdivision intersection point is equal to the abscissas of the first and second vertices of the second triangle, and the ordinate of the second subdivision intersection point is equal to the ordinates of the second and third vertices of the first triangle. In this case, the image warping submodule needs to perform mesh subdivision on the second triangle based on these two subdivision intersection points, without performing mesh subdivision on the first triangle.

[0284] In one embodiment, the image warping submodule may connect the first subdivision intersection point and the second subdivision intersection point, and connect the first subdivision intersection point and the third vertex of the second triangle to perform mesh subdivision on the second triangle to obtain a plurality of subdivided triangles.

[0285] As shown in (7) of Figure 13b, the horizontal coordinate of the first subdivision intersection point Point1 is equal to the horizontal coordinates of the first vertex V2 and the second vertex V3 of the second triangle, and the vertical coordinate of the second subdivision intersection point Point2 is equal to the vertical coordinates of the second vertex V3 and the third vertex V0 of the second triangle. In this case, the image warping submodule needs to perform mesh subdivision on the second triangle based on the subdivision intersection points Point2 and Point1, without performing mesh subdivision on the first triangle.

[0286] Continuing with reference to (7) in FIG13b , the image warping submodule connects the first subdivision intersection Point1 with the second subdivision intersection Point2, and connects the first subdivision intersection Point1 with the third vertex V0 of the second triangle to subdivide the second triangle Tringle1 (V2-V3-V0) into: a first subdivision triangle S-Tringle1 (V2-Point1-V0), a second subdivision triangle S-Tringle2 (Point1-Point2-V0) and a third subdivision triangle S-Tringle3 (Point1-V3-Point2).

[0287] In another embodiment, the image warping submodule may also connect the first subdivision intersection point and the second subdivision intersection point, and connect the second subdivision intersection point and the first vertex of the second triangle to perform mesh subdivision on the second triangle to obtain multiple subdivision triangles.

[0288] As shown in (8) in Figure 13b, the image warping submodule connects the first subdivision intersection Point1 and the second subdivision intersection Point2, and connects the second subdivision intersection Point2 and the first vertex V2 of the second triangle to subdivide the second triangle Tringle1 (V2-V3-V0) into: the first subdivision triangle S-Tringle1 (V2-Point1-Point2), the second subdivision triangle S-Tringle2 (Point1-V3-Point2) and the third subdivision triangle S-Tringle3 (Point2-V0-V2).

[0289] In the subdivision method shown in (7) and (8) in Figure 13b, the starting edge (V2-Point1) of the first subdivided triangle S-Tringle1 is located on the starting edge (V2-V3) of the second triangle, which does not affect the geometric processing order of the second triangle and facilitates developers to perform subdivision programming.

[0290] In this way, by subdividing the triangles according to the subdivision strategy described above, a smaller control unit can be obtained, making the image distortion operation more refined. Moreover, since the image distortion submodule meshes the triangles according to the subdivision intersections, the geometric figures with discontinuous depth information in the Mesh are subdivided, and the depth information in the geometric figures obtained after the subdivision process does not differ too much. Furthermore, when the image distortion submodule moves these subdivided geometric figures, it can avoid the problem of severe deformation of the geometric figures due to excessive differences in mesh depth information, ensure the correctness of the image content, and thus improve the picture instructions, making the predicted frame image closer to the real frame image.

[0291] After the image warping submodule subdivides the current processing image based on the subdivided intersections, it then performs image warping on the normalized and subdivided current processing image based on the motion vectors. For each subdivided triangle, the image warping submodule moves each vertex of the triangle to the corresponding position according to the corresponding motion vector during the image warping process, thereby obtaining the distorted triangle.

[0292] At the same time, for each triangle (whether subdivided or not), the image warping submodule can also set the depth information of the triangle based on the depth values ​​of the triangle vertices. Exemplarily, the image warping submodule performs interpolation processing based on the depth values ​​of each triangle vertex to obtain the depth value of each pixel within the triangle as the depth information of the triangle.

[0293] Thus, when the image warping submodule moves a triangle based on a motion vector, it can determine its overlapping relationship with other triangles based on the depth information of the triangle. For example, if the depth information of triangle 1 is greater than the depth information of triangle 2, then triangle 1 overlaps triangle 2. Here, triangle 2 is the triangle at the target position when triangle 1 is moved according to the corresponding motion vector.

[0294] Therefore, the image warping submodule refers to the pixel depth information when moving geometric shapes in the Mesh, which can ensure the correct depth order relationship between different objects.

[0295] Figure 14 illustrates an example scene for image warping based on subdivided geometry. As shown in Figure 14 , within the quadrilateral Square (V0-V1-V2-V3), the first triangle Tringle1 (V0-V1-V2) is subdivided into: subdivided triangle S-Tringle1 (V0-V1-Point1), subdivided triangle S-Tringle2 (V0-Point1-PointInter), and subdivided triangle S-Tringle3 (PointInter-Point1-V2). The second triangle Tringle2 (V2-V3-V0) is subdivided into: subdivided triangle S-Tringle1' (V2-V3-Point2), subdivided triangle S-Tringle2' (V2-Point2-PointInter), and subdivided triangle S-Tringle3' (PointInter-Point2-V0).

[0296] Among them, the depth information of S-Tringle1 (V0-V1-Point1), S-Tringle2 (V0-Point1-PointInter) and S-Tringle3' (PointInter-Point2-V0) are relatively consistent, and the depth information of S-Tringle3 (PointInter-Point1-V2), S-Tringle1' (V2-V3-Point2) and S-Tringle2' (V2-Point2-PointInter) are relatively consistent.

[0297] When performing image warping processing, subdivided triangles S-Tringle1, S-Tringle2, S-Tringle3, S-Tringle1', S-Tringle2', and S-Tringle3' are moved according to motion vectors corresponding to their respective vertices. For example, S-Tringle1(V0-V1-Point1) moves to S-Tringle1(V0'-V1'-Point1'), S-Tringle2(V0-Point1-PointInter) moves to S-Tringle2(V0'-Point1'-PointInter'), and S-Tringle3(PointInter-Point1-V2) moves to S-Tringle3(PointInter"-Point1"-V2"). Similarly, S-Tringle1'(V2-V3-Point2) moves to S-Tringle1'(V2"-V3"-Point2"), S-Tringle2'(V2-Point2-PointInter) moves to S-Tringle2'(V2"-Point2"-PointInter"), and S-Tringle3'(PointInter-Point2-V0) moves to S-Tringle3'(PointInter'-Point2'-V0').

[0298] Figures (1) to (3) in Figure 14 illustrate different image distortion scenarios. In the scenarios shown in Figures (1) to (3), triangles with consistent depth information move in relatively consistent directions, which can avoid the problem of severe triangle deformation caused by excessive differences in depth information within the triangle.

[0299] Among them, when moving a triangle in Mesh, it is necessary to determine its coverage relationship with other triangles based on the depth information of the triangle. Taking (3) in Figure 14 as an example, compared with S-Tringle3(PointInter-Point1-V2), S-Tringle1'(V2-V3-Point2) and S-Tringle2'(V2-Point2-PointInter), the depth values ​​of S-Tringle1(V0-V1-Point1), S-Tringle2(V0-Point1-PointInter) and S-Tringle3'(PointInter-Point2-V0) are larger. Therefore, when moving S-Tringle1(V0-V1-Point1), S-Tringle2(V0-Point1-PointInter) and S-Tringle3'(PointInter-Point2-V0), and moving S-Tringle3(PointInter-Point1-V2), S-Tringle1' (V2-V3-Point2) and S-Tringle2'(V2-Point2-PointInter), if there is an overlapping area in the moving target position, S-Tringle1(V0'-V1'-Point1'), S-Tringle2(V0'-Point1'-PointInter') and S-Tringle3'(PointInter'-Point2'-V0') will overwrite S-Tringle3(PointInter"-Point1"-V2"), S-Tringle1'(V2"-V3"-Point2") and S-Tringle2'(V2"-Point2"-PointInter") to ensure the correct depth order of objects in the image area.

[0300] S508: The image distortion submodule performs image distortion processing on the current processing image according to the motion vector.

[0301] If the number of subdivision intersections of the currently processed graphic is not 2, or if the number of subdivision intersections of the currently processed graphic is 2 but the two subdivision intersections are located on the same edge of the currently processed graphic, the currently processed graphic does not meet the mesh subdivision conditions, and the image warping submodule directly performs image warping on the standardized currently processed graphic. For the currently processed graphic, the image warping submodule performs image warping on each triangle obtained by standardization. When performing image warping on a triangle, the image warping submodule moves each vertex of the triangle to the corresponding position according to the corresponding motion vector, thereby obtaining a distorted triangle.

[0302] Similarly, for each unsubdivided triangle, the image warping submodule can also set the depth information of the triangle based on the depth values ​​of the vertices. Exemplarily, the image warping submodule performs interpolation processing based on the depth values ​​of each triangle vertex to obtain the depth value of each pixel within the triangle as the depth information of the triangle.

[0303] In this way, when the image warping submodule moves a triangle based on the motion vector, it can determine the overlapping relationship between the triangle and other triangles according to the depth information of the triangle, so as to ensure the correct depth order relationship between different objects.

[0304] S509: Check whether the image distortion submodule has processed all geometric figures in the current image frame Mesh. If so, execute S4052; if not, execute S504.

[0305] At this point, the image warping submodule completes the image warping operation for the geometric figures corresponding to the static objects in the current image frame Mesh.

[0306] The image warping submodule then performs the image warping operation on the triangles corresponding to the dynamic objects in the current image frame mesh. At this point, within the created and standardized current image frame mesh, the image warping submodule searches for each triangle corresponding to the dynamic object and performs the image warping operation on each triangle corresponding to the dynamic object.

[0307] Similarly, for each triangle corresponding to the dynamic object, each vertex of the triangle is moved to a corresponding position according to the corresponding motion vector, so as to obtain a triangle after image distortion.

[0308] At the same time, for each triangle corresponding to the dynamic object, the image warping submodule can also set the depth information of the triangle based on the depth values ​​of the vertices. Exemplarily, the image warping submodule performs interpolation processing based on the depth values ​​of each triangle vertex to obtain the depth value of each pixel within the triangle as the depth information of the triangle.

[0309] In this way, when the image warping submodule moves a triangle based on the motion vector, it can determine the overlapping relationship between the triangle and other triangles according to the depth information of the triangle, so as to ensure the correct depth order relationship between different objects.

[0310] At this point, the image warping submodule has completed the image warping operation for the mesh geometry corresponding to the dynamic object in the current image frame. In other words, the image warping submodule has completed the image warping operation for the current frame image. After the image warping phase is complete, the predicted frame generation module continues with the rasterization and pixel processing phases to generate the first predicted frame image. Regarding the predicted frame image generation process, any details not fully explained in this embodiment can be referred to existing technologies and will not be elaborated upon here.

[0311] In another optional embodiment, the image distortion submodule can also create a triangle mesh for the current frame image. In this case, the image distortion submodule can determine whether each triangle in the triangle mesh needs to be meshed, and how to perform mesh subdivision and image distortion. The judgment and processing flow for each triangle can also refer to the above and will not be repeated here. It should be noted that if the number of subdivision intersections of a triangle is 2, then the triangle meets the mesh subdivision conditions and can be meshed with reference to the subdivision method of Case 1 or Case 3.

[0312] In another optional embodiment, for the current frame image, the image distortion submodule can also create a polygonal mesh and perform normalization processing. In this case, the image distortion submodule can determine whether each triangle in the polygonal mesh needs to be meshed, and how to perform mesh subdivision and image distortion. The judgment and processing flow for each triangle can also refer to the above and will not be repeated here. It should be pointed out that if the number of subdivision intersections of a polygon in the polygonal mesh is 2, then the polygon meets the mesh subdivision conditions and can be meshed with the polygon according to the subdivision methods of Cases 1 to 3 above.

[0313] Figure 15a exemplarily shows an example of a predicted frame image. In the predicted frame image shown in Figure 15a, region 601 adopts the existing technical solution for image distortion processing, and region 602 adopts the solution provided by this embodiment for image distortion processing. Among them, region 601 and region 602 are only regions where the depth information of static objects is discontinuous. By comparing region 601 and region 602, it can be seen that the image in region 601 is distorted, especially the edge area of ​​the static object is severely deformed, while the picture effect of region 602 is better, the edge area of ​​the static object is clear, and there is no deformation of the edge area of ​​the static object.

[0314] FIG15b exemplarily shows a comparative schematic diagram of a predicted frame image. Among them, the predicted frame image shown in (1) in FIG15b is generated by performing image distortion processing using the solution provided in this embodiment, while the predicted frame image shown in (2) in FIG15b is generated by performing image distortion processing using the existing technical solution. Region 603 in (1) in FIG15b and region 604 in (2) in FIG15b are the same image regions, both of which are regions where the depth information of static objects is discontinuous. By comparing (1) and (2) in FIG15b , it can be seen that the image in region 604 is distorted, especially the edge region of the static object is severely deformed, while the picture effect of region 603 is better, the edge region of the static object is clear, and there is no deformation of the edge region of the static object.

[0315] S406 , the prediction frame completion module performs image completion on the first prediction frame image to obtain a second prediction frame image, and sends the second prediction frame image to the image synthesis module.

[0316] Since the moving speeds of various objects in the original video are different, the first predicted frame image calculated by the electronic device based on the motion vector may have image information overlap or missing. In other words, the first predicted frame image is the predicted frame image to be completed.

[0317] Therefore, the prediction frame completion module needs to perform image completion on the first prediction frame image to obtain a second prediction frame image, wherein the second prediction frame image is the prediction frame image obtained after completing the image completion process on the first prediction frame image.

[0318] S407 , the image synthesis module synthesizes the second predicted frame image with the UI information to obtain a target predicted frame image, and sends the target predicted frame image for display.

[0319] The image synthesis module synthesizes the second predicted frame image and the UI information corresponding to the current real frame image to obtain a target predicted frame image.

[0320] In this way, after completing the complement processing of the predicted frame image, it is synthesized with the UI information to obtain the target predicted frame image, and then the target predicted frame image can be output to the electronic device display screen for display.

[0321] It should be noted that the above process is explained using the example of processing a single predicted frame image. The process for processing each predicted frame image is also similar and will not be repeated here. The order in which the target predicted frame images are displayed is related to the method of interpolating the predicted frame images. Reference can be made to existing technologies and will not be repeated here. Any areas not fully explained in this process can also be referred to existing technologies and will not be repeated here.

[0322] High-refresh-rate games on mobile platforms are often prone to overheating or lag due to power consumption or CPU and GPU limitations, thus affecting the user experience. To enhance the smoothness of the game experience, game developers often lower the original refresh rate and then generate predicted frames in the image space, reducing the number of rendering operations by half at the same frame rate, thereby significantly reducing the power consumption of smart terminals and reducing the heat generation of smart terminals.

[0323] The image processing method provided by the embodiments of this application can effectively address image distortion in predicted frame images, especially the deformation of static object edges in areas with discontinuous depth information, thereby improving the image quality of the predicted frame images. This ensures that the game's smoothness remains unchanged while the number of rendered frames is halved, reducing power consumption and heat generation in smart terminals, thereby improving the user experience.

[0324] In other interpolation application scenarios, if there are areas in the image with discontinuous depth information (or large depth differences), the image processing method provided in the embodiment of the present application can also be used to perform image distortion processing to improve the image quality of the predicted frame image, which will not be elaborated here.

[0325] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0326] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0327] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0328] Among them, the electronic devices (such as mobile phones, etc.), computer storage media, computer program products or chips provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0329] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0330] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0331] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image and a second image, and calculate a motion vector based on the first image and the second image; wherein the first image and the second image are real frame images; Creating an initial mesh of the second image, and determining in the initial mesh initial geometric figures to be subdivided corresponding to the static object; wherein the initial geometric figures have only two subdivision intersections located on edges of the initial geometric figures, and each edge of each initial geometric figure has at most one subdivision intersection; the subdivision intersections are determined based on depth values ​​of pixels in the initial geometric figures and are edge pixels of the static object; Subdividing the corresponding initial geometric figures according to the two subdivision intersection points of each of the initial geometric figures to obtain a target mesh of the second image; in the target mesh, a line connecting the two subdivision intersection points of each of the initial geometric figures is a side of a target triangle, or a portion of the line segments on the line is a side of the target triangle; wherein the target triangle is a geometric figure divided in the target mesh; Perform image warping processing on the second image according to the target Mesh and the motion vector to generate a predicted frame image corresponding to the first image and the second image.

2. The method according to claim 1, characterized in that The initial Mesh is a quadrilateral Mesh, and the initial geometric figure is a quadrilateral; After creating the initial Mesh of the second image, the following steps are further included: The quadrilateral mesh is normalized to divide each quadrilateral into two triangles.

3. The method according to claim 2, characterized in that The subdivision intersection points of the quadrilateral to be subdivided are the first subdivision intersection point and the second subdivision intersection point, and the quadrilateral to be subdivided is divided into the first triangle and the second triangle; Among them, in the target Mesh: The first triangle is subdivided into a first sub-triangle, a second sub-triangle, and a third sub-triangle; wherein the common side of the first sub-triangle and the second sub-triangle is a line connecting the first sub-division intersection point and the second sub-division intersection point, and the common side of the second sub-triangle and the third sub-triangle is a line connecting the first sub-division intersection point or the second sub-division intersection point and a vertex of the first triangle; or The second triangle is subdivided into a fourth sub-triangle, a fifth sub-triangle, and a sixth sub-triangle; wherein the common side of the fourth sub-triangle and the fifth sub-triangle is a line connecting the first subdivision intersection point and the second subdivision intersection point, and the common side of the fifth sub-triangle and the sixth sub-triangle is a line connecting the first subdivision intersection point or the second subdivision intersection point and a vertex of the second triangle; or, The first triangle is subdivided into a seventh sub-triangle, an eighth sub-triangle and a ninth sub-triangle, and the second triangle is subdivided into a tenth sub-triangle, an eleventh sub-triangle and a twelfth sub-triangle; wherein the common side of the seventh sub-triangle and the eighth sub-triangle is the line connecting the first subdivision intersection and the third point, the common side of the eighth sub-triangle and the ninth sub-triangle is the line connecting the first subdivision intersection and the vertex of the first triangle; the common side of the tenth sub-triangle and the eleventh sub-triangle is the line connecting the second subdivision intersection and the third point, the common side of the eleventh sub-triangle and the tenth sub-triangle is the line connecting the second subdivision intersection and the third point, The common side of the two sub-triangles is the line connecting the second subdivision intersection and the vertex of the second triangle; the intersection of the line connecting the first subdivision intersection and the second subdivision intersection and the common side of the first triangle and the second triangle is the third point.

4. The method according to claim 1, wherein Also includes: In the target Mesh, the depth information of each triangle is set.

5. The method according to claim 4, characterized in that Performing image warping processing on the second image according to the target Mesh and the motion vector, including: For each triangle in the target Mesh, determine the motion vectors of each vertex of the triangle, and move the triangle according to the motion vectors of each vertex of the triangle; When each triangle moves to a target position, an overlapping relationship between the triangle and other triangles at the target position is determined based on the depth information of the triangle and the depth information of other triangles at the target position.

6. The method according to claim 2, characterized in that The subdivision intersection points of the quadrilateral to be subdivided are the first subdivision intersection point and the second subdivision intersection point, and the quadrilateral to be subdivided is divided into the first triangle and the second triangle; Subdividing the initial geometric figure into a mesh according to two subdivision intersection points of the initial geometric figure, comprising: According to the positional relationship between the first subdivision intersection point and the second subdivision intersection point, the first triangle, and the second triangle, a matching mesh subdivision strategy is used to perform mesh subdivision on the first triangle and / or the second triangle.

7. The method according to claim 6, characterized in that Subdividing the first triangle and / or the second triangle using a matching mesh subdivision strategy according to the positional relationship between the first subdivision intersection point and the second subdivision intersection point and the first triangle and the second triangle, including: When the abscissa of the first subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the first triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the first triangle, the first subdivision intersection and the second subdivision intersection are connected, and the first subdivision intersection and the third vertex of the first triangle are connected, or the second subdivision intersection and the first vertex of the first triangle are connected to separately subdivide the first triangle into three triangles.

8. The method according to claim 6, characterized in that Subdividing the first triangle and / or the second triangle using a matching mesh subdivision strategy according to the positional relationship between the first subdivision intersection point and the second subdivision intersection point and the first triangle and the second triangle, including: When the abscissa of the first subdivision intersection is equal to the abscissas of the first and second vertices of the second triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second and third vertices of the second triangle, the first subdivision intersection and the second subdivision intersection are connected, and the first subdivision intersection and the third vertex of the second triangle are connected, or the second subdivision intersection and the first vertex of the second triangle are connected, so as to subdivide the second triangle into three triangles separately.

9. The method according to claim 6, characterized in that Subdividing the first triangle and / or the second triangle using a matching mesh subdivision strategy according to the positional relationship between the first subdivision intersection point and the second subdivision intersection point and the first triangle and the second triangle, including: When the ordinate of the first subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the first triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the second triangle, connecting the first subdivision intersection and the second subdivision intersection, connecting the first subdivision intersection and the first vertex of the first triangle, and connecting the second subdivision intersection and the first vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively; When the abscissa of the first subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the first triangle, and the abscissa of the second subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the second triangle, connecting the first subdivision intersection and the second subdivision intersection, connecting the first subdivision intersection and the third vertex of the first triangle, and connecting the second subdivision intersection and the third vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively; When the ordinate of the first subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the first triangle, and the abscissa of the second subdivision intersection is equal to the abscissa of the first vertex and the second vertex of the second triangle, connecting the first subdivision intersection and the second subdivision intersection, connecting the first subdivision intersection and the first vertex of the first triangle, and connecting the second subdivision intersection and the third vertex of the second triangle, so as to subdivide the first triangle and the second triangle into three triangles respectively; When the abscissa of the first subdivision intersection is equal to the abscissas of the first vertex and the second vertex of the first triangle, and the ordinate of the second subdivision intersection is equal to the ordinates of the second vertex and the third vertex of the second triangle, the first subdivision intersection is connected to the second subdivision intersection, the first subdivision intersection is connected to the third vertex of the first triangle, and the second subdivision intersection is connected to the first vertex of the second triangle to subdivide the first triangle and the second triangle into three triangles respectively.

10. The method according to claim 1, characterized in that Determining initial geometric figures to be subdivided corresponding to static objects in the initial Mesh, including: Determining an initial geometric figure corresponding to a static object in the initial Mesh; determining a subdivision intersection point of each of the initial geometric figures; An initial geometric figure with 2 subdivision intersections and at most one subdivision intersection on each edge is used as the initial geometric figure to be subdivided.

11. The method according to claim 10, characterized in that Determining the subdivision intersection of the initial geometric figure, including: Determining each edge pixel point in the initial geometric figure; wherein the depth value of the edge pixel point is within a preset range, and the difference between the depth value of the edge pixel point and the surrounding pixel points exceeds a preset threshold; The edge pixel points located on the edge of the initial geometric figure are used as subdivision intersection points of the initial geometric figure.

12. The method according to any one of claims 1 to 11, characterized in that The first image and the second image are rendered images.

13. The method according to claim 12, characterized in that The acquiring of the first image and the second image comprises: Acquire a first resource image; separate user interface UI information from the first resource image to obtain the first image; Acquire a second resource image; separate UI information from the second resource image to obtain the second image; After generating the predicted frame image corresponding to the first image and the second image, the method further includes: The predicted frame image is subjected to image completion processing, and the completed predicted frame image and UI information are fused to obtain a target predicted frame image.

14. The method according to claim 13, characterized in that The first resource image and the second resource image are resource images in a game application.

15. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the image processing method according to any one of claims 1 to 14.

16. A computer-readable storage medium comprising a computer program, characterized in that When the computer program is run on an electronic device, the electronic device is enabled to execute the image processing method according to any one of claims 1 to 14.

Images