Image prediction method and device

Abstract

The invention provides an image processing method and electronic equipment, and the method comprises the steps: obtaining a depth offset map based on a depth map of an image frame, and enabling the depth offset map to be used for indicating the correct shielding relation between pixels. And moving pixels in the image frame based on the depth offset map to obtain a prediction frame. Based on the depth of the pixel of the image frame, the image authenticity of the prediction frame can be improved in a mode of obtaining the prediction frame, and the problem of image distortion is avoided.

Description

[0001] This application is a divisional application. The application number of the original application is 202310886771.4, and the application date of the original application is July 17, 2023. The entire content of the original application is incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and in particular, to an image processing method and apparatus. Background Art

[0003] With the continuous improvement of the quality of mobile games, higher requirements are put forward for mobile phone chips and mobile phone designs. Due to limitations such as chip computing power and mobile phone heat dissipation, it is impossible to have both high image quality and high frame rate in the experience of mobile games. Currently, in order to achieve a high frame rate, manufacturers have launched a predicted frame scheme, that is, adjacent two frames are used to generate an intermediate predicted frame in the case of low frame rate and high image quality, so as to achieve the goal of high image quality and high frame rate. However, due to the imperfect processing of the predicted frame in the existing frame interpolation scheme, there is a problem of image distortion, which reduces the game experience of users. Summary of the Invention

[0004] This application provides an image processing method and an electronic device. In this method, the electronic device can generate a predicted frame based on the depth of each pixel in the image frame to improve the picture authenticity of the predicted frame.

[0005] In a first aspect, this application provides an image processing method. Applied to an electronic device, the method includes: obtaining a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame. Moving all pixels in the first depth map to obtain a first depth offset map, where the first depth offset map is used to indicate the occlusion relationship of all pixels in the first depth map. Moving all pixels in the second depth map to obtain a second depth offset map, where the second depth offset map is used to indicate the occlusion relationship of all pixels in the second depth map. Obtaining a target depth offset map based on the first depth offset map and the second depth offset map. Moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target predicted frame. In this way, based on the depth of each pixel in the real frame, this application can obtain the correct occlusion relationship of each pixel in the predicted frame, effectively avoid the image distortion problem in the predicted frame, and improve the user's visual experience.

[0006] Exemplarily, the predicted frame is an image frame between two real frames.

[0007] Exemplarily, the electronic device may be a terminal, a server, etc.

[0008] Exemplarily, the first image frame is the current frame, and the second image frame is the previous frame.

[0009] In a possible implementation, moving all pixels in the first depth map to obtain a first depth offset map includes: determining a first position of a first pixel in the first depth map in the first depth offset map based on a first motion vector (MV) map of the first image frame; if the first depth value of the first pixel in the first depth map is greater than the depth value of the first position, writing the first depth value to the first position; determining a second position of a second pixel in the first depth map in the first depth map based on the first MV map; if the first position is the same as the second position, and the second depth value of the second pixel in the first depth map is greater than the first depth value, writing the second depth value to the first position. In this way, through the MV map, the present application can obtain the target positions of the pixels of the real frame, and can obtain the depth offset map based on the depth of each pixel, so as to determine the correct occlusion relationship between the pixels in the predicted frame based on the depth offset map.

[0010] In a possible implementation, moving all pixels in the second depth map to obtain a second depth offset map includes: determining a third position of a third pixel in the second depth map in the second depth offset map based on a second MV map of the second image frame; if the third depth value of the third pixel in the second depth map is greater than the depth value of the third position, writing the third depth value to the third position; determining a fourth position of a fourth pixel in the second depth map in the second depth map based on the second MV map; if the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, writing the fourth depth value to the third position. In this way, through the MV map, the present application can obtain the target positions of the pixels of the real frame, and can obtain the depth offset map based on the depth of each pixel, so as to determine the correct occlusion relationship between the pixels in the predicted frame based on the depth offset map.

[0011] In a possible implementation, obtaining a target depth offset map based on the first depth offset map and the second depth offset map includes: writing the depth values corresponding to all positions in the first offset map to the target depth offset map, where for the positions in the target depth offset map where the depth values are not written, obtaining the depth values from the corresponding positions in the second depth offset map and writing them. In this way, the present application can combine the previous frame and the current frame to obtain the depth values of all points in the depth offset map as much as possible.

[0012] In a possible implementation, moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: obtaining a coordinate mapping map based on the target depth offset map, the first image frame, and the second image frame; wherein the coordinate mapping map is used to indicate pixel points corresponding to all positions in the target depth offset map in the first image frame or the second image frame. In this way, the present application can obtain a coordinate mapping map based on the correct occlusion relationship in the depth offset map, so as to obtain the mapping relationship between each point between the real frame and the depth offset map.

[0013] In a possible implementation, moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: obtaining a to-be-completed prediction frame based on the coordinate mapping map, the first image frame, and the second image frame. In this way, in the present application, pixels in the first image frame and the second image frame can be moved based on the coordinate mapping map, so as to obtain a to-be-completed prediction frame. Among them, the coordinate mapping map is obtained based on a depth offset map that can reflect the correct occlusion relationship. Therefore, the occlusion relationship between pixel points in the to-be-completed prediction frame is accurate, and there will be no problem of image distortion.

[0014] In a possible implementation, moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: finding void points in the to-be-completed prediction frame; determining a fifth pixel point from among a plurality of pixel points around the void point, the fifth pixel point being a non-void point; based on the coordinate mapping map, finding a sixth pixel corresponding to the fifth pixel point in the first image frame or the second image frame; based on a first relative position between the void point and the fifth pixel, determining a seventh pixel in the first image frame or the second image frame, the second relative position between the seventh pixel and the sixth pixel being the same as the first relative position; obtaining a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel based on the first depth map or the second depth map; writing the color of the pixel corresponding to the smaller depth value among the seventh depth value and the sixth depth value into the void point. In this way, based on the mapping relationship and depth between pixel points in the real frame in the coordinate mapping map, the present application can find the point with the depth closest to that of the void point and fill the color of this point into the void point, thereby avoiding the problem of image distortion.

[0015] Second aspect, the present application provides an electronic device. The electronic device includes: one or more processors, a memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when the computer programs are executed by the one or more processors, the electronic device is caused to perform the following steps: obtaining a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame; moving all pixels in the first depth map to obtain a first depth offset map, where the first depth offset map is used to indicate the occlusion relationship of all pixels in the first depth map; moving all pixels in the second depth map to obtain a second depth offset map, where the second depth offset map is used to indicate the occlusion relationship of all pixels in the second depth map; obtaining a target depth offset map based on the first depth offset map and the second depth offset map; and moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame.

[0016] In a possible implementation manner, when the computer programs are executed by the one or more processors, the electronic device is caused to perform the following steps: determining a first position of a first pixel in the first depth map in the first depth offset map based on a first motion vector (MV) map of the first image frame; if the first depth value of the first pixel in the first depth map is greater than the depth value of the first position, writing the first depth value into the first position; determining a second position of a second pixel in the first depth map in the first depth map based on the first MV map; if the first position is the same as the second position, and the second depth value of the second pixel in the first depth map is greater than the first depth value, writing the second depth value into the first position.

[0017] In a possible implementation manner, when the computer programs are executed by the one or more processors, the electronic device is caused to perform the following steps: determining a third position of a third pixel in the second depth map in the second depth offset map based on a second MV map of the second image frame; if the third depth value of the third pixel in the second depth map is greater than the depth value of the third position, writing the third depth value into the third position; determining a fourth position of a fourth pixel in the second depth map in the second depth map based on the second MV map; if the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, writing the fourth depth value into the third position.

[0018] In a possible implementation, when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps: write the depth values corresponding to all positions in the first offset map into the target depth offset map, where for positions in the target depth offset map where depth values have not been written, obtain the depth values from the corresponding positions in the second depth offset map and write them in.

[0019] In a possible implementation, when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps: obtain a coordinate mapping map based on the target depth offset map, the first image frame, and the second image frame; where the coordinate mapping map is used to indicate the pixel points in the first image frame or the second image frame corresponding to all positions in the target depth offset map.

[0020] In a possible implementation, when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps: obtain a to-be-completed prediction frame based on the coordinate mapping map, the first image frame, and the second image frame.

[0021] In a possible implementation, when the computer program is executed by the one or more processors, the electronic device is caused to perform the following steps: find the void points in the to-be-completed prediction frame; determine a fifth pixel point from among a plurality of pixel points around the void points, where the fifth pixel point is a non-void point; based on the coordinate mapping map, find a sixth pixel in the first image frame or the second image frame corresponding to the fifth pixel point; based on a first relative position between the void point and the fifth pixel, determine a seventh pixel in the first image frame or the second image frame, where a second relative position between the seventh pixel and the sixth pixel is the same as the first relative position; based on the first depth map or the second depth map, obtain a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel; write the color of the pixel corresponding to the smaller of the seventh depth value and the sixth depth value into the void point.

[0022] In a third aspect, the present application provides a computer-readable medium for storing a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation manner of the first aspect.

[0023] In a fourth aspect, the present application provides a computer program, the computer program including instructions for performing the method in the first aspect or any possible implementation manner of the first aspect.

[0024] Fifth aspect, the present application provides a chip, which includes a processing circuit and transceiver pins. Among them, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation manner of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. Description of the Drawings

[0025] Figure 1 Schematic diagram of a predicted frame shown exemplarily;

[0026] Figure 2 Schematic diagram of the display of a predicted frame shown exemplarily;

[0027] Figure 3 Schematic diagram of the principle of an image processing method shown exemplarily;

[0028] Figure 4 Schematic diagram of the flow of an image processing method shown exemplarily;

[0029] Figure 5 Schematic diagram of the acquisition process of a depth map and an MV map shown exemplarily;

[0030] Figure 6 Schematic diagram of the acquisition of a depth offset map shown exemplarily;

[0031] Figure 7 Schematic diagram of a depth map shown exemplarily;

[0032] Figure 8 Schematic diagram of the acquisition of a depth offset map shown exemplarily;

[0033] Figure 9 Schematic diagram of the acquisition of a depth offset map shown exemplarily;

[0034] Figure 10 Schematic diagram of the acquisition of a depth offset map shown exemplarily;

[0035] Figure 11 Schematic diagram of the acquisition process of a coordinate mapping map shown exemplarily;

[0036] Figure 12 Schematic diagram of the acquisition of a coordinate mapping map shown exemplarily;

[0037] Figure 13 Schematic diagram of the acquisition of a predicted frame to be completed shown exemplarily;

[0038] Figure 14 Schematic diagram of a predicted frame to be completed shown exemplarily;

[0039] Figure 15 Schematic diagram of the acquisition process of a predicted frame shown exemplarily;

[0040] Figure 16 Schematic diagram of pixel search shown by way of example;

[0041] Figure 17 Schematic diagram of obtaining a predicted frame shown by way of example;

[0042] Figure 18 Schematic diagram of obtaining a predicted frame shown by way of example;

[0043] Figure 19 Schematic diagram of the structure of the device shown by way of example. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0045] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0046] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe the specific order of the target objects.

[0047] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0048] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0049] The solution in the embodiments of the present application is applied to a cloud server, hereinafter referred to as the cloud server for short. Among them, the cloud server may be a server cluster composed of one or more servers, which is not limited in the present application. Specifically, the cloud server generates image frames of a game in the cloud and sends the image frames to the game side (such as a terminal, a mobile phone, a wearable device, etc.), and the game side receives and displays the image frames. Exemplarily, only a game scenario is taken as an example in the embodiments of the present application. In other embodiments, the image processing method in the embodiments of the present application can also be applied to other video stream scenarios, which is not limited in the present application.

[0050] Exemplarily, to improve the frame rate of the game, game manufacturers provide a predicted frame solution, that is, in the case of low frame rate and high image quality, adjacent two frames are used to generate an intermediate predicted frame, and while ensuring high image quality, the goal of high frame rate is achieved. Figure 1 For a schematic diagram of the predicted frame shown exemplarily, please refer to Figure 1 In the embodiments of the present application, the cloud server inserts a predicted frame between two image frames to improve the frame rate. To better illustrate the embodiments of the present application, in the present application, the image frame before the predicted frame is called the previous frame, and the frame after the predicted frame is called the current frame. That is to say, in the embodiments of the present application, a predicted frame can be generated based on the previous frame and the current frame, and the predicted frame is inserted between the previous frame and the current frame. Optionally, the previous frame can also be called the first frame (or the first image frame), and the current frame can also be called the second frame (or the second image frame), which is not limited in the present application.

[0051] Figure 2 For a schematic diagram of the predicted frame display shown exemplarily, please refer to Figure 2 Exemplarily, the predicted frame is generated based on the previous frame and the current frame. As Figure 2 (1) in is the previous frame 100. Due to the movement of the camera, the object moves in the picture. For example, the image 101 shifts to the left side of the picture. That is, the picture to be displayed by the predicted frame is as shown in the image frame 110 in (2) of Figure 2 . Among them, the image 101 shifts to the left compared with the previous frame, and the part that was originally blocked by the image 101 in the previous frame 100 will be displayed in the image frame. However, since the previous frame 100 does not include the blocked part, the cloud server needs to repair the blank image 102 to avoid blank areas in the image of the predicted frame.

[0052] As Figure 2 (2) in shows, the cloud server may supplement the image of the image 101 into the blank image 102, and supplement the original image part of the pillar into the area that should originally be the ground, which causes the pillar in the predicted image frame to be distorted. From the user's perspective, the pillar in the image displayed by the predicted frame 130 suddenly becomes wider in the image 102 part, resulting in distortion and affecting the user experience.

[0053] The present invention provides a scheme for generating a prediction frame pixel by pixel, which can effectively solve the problem of image distortion in the drawing of the prediction frame. Figure 3 For a schematic diagram showing the principle of the image processing method in the embodiments of the present application shown by way of example, please refer to Figure 3 , specifically including but not limited to:

[0054] S301, Global depth pixel movement.

[0055] Exemplarily, the cloud server can move the pixels in the depth map corresponding to the image frame based on the depth map and the MV map (Motion Vector) of the previous frame.

[0056] S302, Obtain a depth offset map, which is used to indicate the occlusion relationship.

[0057] The cloud server can obtain a depth offset map with a correct occlusion relationship based on the depth of each pixel after movement.

[0058] S303, Move the original image based on the depth offset map.

[0059] Exemplarily, the cloud server moves the pixels of the previous frame and the current frame based on the depth offset map, and then obtains a high-quality prediction frame. Since the correct image shape and mapping rules are maintained, the picture of the prediction frame is basically the same as the real rendered frame, and high image quality is maintained while the number of frames is doubled.

[0060] The following combines Figure 4 The flow schematic diagram of the image processing method shown, and details of the steps in Figure 2 are described in detail. Please refer to Figure 4 , specifically including but not limited to the following steps:

[0061] S401, Obtain the depth map and the MV map of the current frame and the previous frame.

[0062] Figure 5 For a schematic diagram showing the acquisition process of the depth map and the MV map shown by way of example, please refer to Figure 5 , exemplarily, the processing of the image frame by the cloud server generally includes: receiving a game rendering instruction stream, resource preparation, game rendering, and obtaining a real frame (i.e., an image frame, such as the previous frame or the current frame) stage.

[0063] In the process of the cloud server executing the above stages to obtain the previous frame or the current frame, the cloud server can obtain corresponding resources from the stages of resource preparation, game rendering, and obtaining the real frame, and obtain the MV map or matrix information of the real frame, the depth map of the real frame, and the real frame from the obtained resources. Among them, the real frame includes the previous frame and the current frame, that is, two adjacent image frames, which can also be referred to as the first image frame and the second image frame. That is to say, in Figure 5 In the shown process, the cloud server can obtain the matrix information or MV map of the previous frame (which can also be referred to as the previous image frame and will not be repeated hereinafter), the depth map of the previous frame, and the previous frame (that is, the image of the previous frame, which can also be referred to as the rendered image of the previous frame); and, the cloud server also obtains the matrix information or MV map of the current frame (which can also be referred to as the current image frame and will not be repeated hereinafter), the depth map of the current frame, and the current frame (that is, the image of the current frame, which can also be referred to as the rendered image of the current frame).

[0064] Exemplarily, during the rendering process of the real frame, since coordinate mapping is required, the matrix used to calculate the vertex position of the object is the matrix information in the embodiments of the present application. That is to say, both the matrix information and the MV map are used to indicate the moving direction and moving distance of the camera. In one example, the cloud server can directly obtain the MV map corresponding to the real frame from the resources. In another example, the cloud server can also obtain the matrix information corresponding to the real frame and obtain the corresponding MV map based on the matrix information. The specific acquisition method can refer to the existing technology, and the present application does not make any limitations.

[0065] Exemplarily, the depth map of the real frame is used to indicate the depth corresponding to each pixel in the image frame. Among them, the depth of the pixel closer to the screen is greater, while the depth of the pixel farther from the screen is smaller. Optionally, in other embodiments, the depth value of the pixel closer to the screen can be smaller, and conversely, the depth value of the pixel farther from the screen can be larger. The present application does not make any limitations.

[0066] S402. Obtain a depth offset map based on the depth maps and MV maps of the current frame and the previous frame.

[0067] The cloud server can obtain a depth offset map with a correct occlusion relationship based on the depth of each pixel after movement. That is, in the embodiments of the present application, the depth offset map can be used to indicate the occlusion relationship between pixels.

[0068] Figure 6 For the schematic diagram of obtaining the depth offset map shown exemplarily, please refer to Figure 6 , specifically including but not limited to:

[0069] S601-1. Obtain the depth map S1 of the current frame.

[0070] Figure 7 For an exemplary schematic diagram of a depth map, please refer to Figure 7 , the original image of the current frame can also be referred to as the rendered image of the current frame. Among them, the rendered image of the current frame includes a plurality of pixels, and each pixel includes coordinates and a color. Among them, the coordinates are optionally in a coordinate system with the upper left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis, and the color can also be referred to as a color value.

[0071] As described above, the cloud server can obtain the depth map 70 of the current frame, denoted as S1, where the depth map includes the depth values of each pixel. For example, the depth of pixel 701 is A, and the depth of pixel 702 is B, where A is greater than B, that is, the distance between pixel 701 and the screen is less than the distance between pixel 702 and the screen. That is to say, pixel 701 is closer to the screen, and pixel 702 is farther from the screen than pixel 701.

[0072] S602-1, obtain the MV map of the current frame.

[0073] Exemplarily, the cloud server can obtain the MV map of the current frame based on the Figure 5 method in. Optionally, the cloud server can directly obtain the MV map, or it can obtain the matrix information corresponding to the current frame and obtain the corresponding MV map based on the matrix information. This application does not make any limitations.

[0074] S603-1, based on the depth D1 of the pixel in the depth map S1, obtain the target position P1 of the offset of the pixel.

[0075] Exemplarily, the cloud server can move the pixels in the depth map of the current frame based on the MV map of the current frame to move the pixels to the target position.

[0076] Figure 8 For an exemplary schematic diagram of obtaining a depth offset map, please refer to Figure 8 , exemplarily, the cloud server obtains the current frame depth map 70, which is S1, and the cloud server obtains the current frame MV map 71. Taking the pixel X in the current frame depth map as an example, the depth of the pixel X is Dx. The cloud server moves the pixel X based on the offset amount △X corresponding to the pixel X in the MV map to determine the target position P1 of the pixel X in the depth offset map 72 (i.e., T1) corresponding to the current frame. Among them, △X is used to indicate the moving direction and moving distance of the pixel X.

[0077] S604-1, read the depth d1 of P1.

[0078] Optionally, due to camera movement, multiple pixels in the current frame may move to the same target position. In the embodiments of the present application, the depth of the target position is the maximum depth among the multiple pixels. For example, after pixels A, B, and C move, their target position is point P. The depths of pixels A, B, and C from small to large are depth A, depth B, and depth C respectively. Therefore, the depth of the target position is the depth of pixel C, which is depth C. That is to say, due to camera movement, the pixels in the current frame move accordingly. In the case where different pixels move to the same position, the occlusion relationship can be determined based on the depth. That is, in the above example, pixels A, B, and C will all move to the target position P, and the depth of pixel C is the largest. Therefore, pixel C is the closest to the screen and will occlude pixels B and C. It can be understood that in the predicted frame, pixel C will be displayed at position P, while pixels B and A will not be displayed.

[0079] Exemplarily, the cloud server reads the depth value d1 of the target position P1. Optionally, the initial depth of the target position P1 can be 0 (which can be set according to actual needs and is not limited in this application). After the depth value of the pixel is written to the target position P1, the depth value of the target position P1 changes accordingly.

[0080] Still referring to Figure 8 , in this example, it is assumed that the current depth d1 of the target position P1 is 0, that is, the depth of the pixel has not been written yet.

[0081] S605-1. Determine the depth D of P1 based on D1 and d1.

[0082] Exemplarily, the cloud server compares the current depth d1 of the target position P1 with the depth D1 of the pixel. In one example, if D1 is less than d1, that is, the depth of the pixel moving to the target position P1 is less than the current depth of the target position P1, the depth of the target position P1 is not updated. In another example, if D1 is greater than d1, that is, the depth of the pixel moving to the target position P1 is greater than the current depth of the target position P1, the depth of the pixel is written.

[0083] Still referring to Figure 8 , exemplarily, the cloud server obtains the current depth d1 of the target position P1. It is assumed that the current depth of the target position is 0. The cloud server compares d1 with the depth Dx of pixel X. In this example, the depth Dx of pixel X is greater than the current depth d1 of P1 (which is 0). The cloud server writes the depth Dx of pixel X to the target position P1, that is, the depth D of the target position P1 is updated to Dx.

[0084] Exemplarily, the cloud server repeatedly executes S603-1 to S605-1. For example, Figure 9As shown, the cloud server traverses to the pixel Y in the current frame depth map S1, and the depth of pixel Y is Dy. Based on the MV map, the cloud server moves pixel Y and determines that the target position of pixel Y in the depth offset map T1 is also P1. The cloud server compares the current depth d1 (i.e., Dx) of the target position P1 with the depth Dy of pixel Y. In one example, if Dy is less than d1, the depth of the target position P1 is not updated, that is, the depth of the target position P1 remains Dx. In this example, since Dy is greater than d1, the cloud server updates the depth of the target position P1, and the updated depth D of the target position P1 is Dy.

[0085] Exemplarily, the cloud server traverses all pixels in the current frame depth map S1 in a loop and obtains the depth offset map T1.

[0086] S601-2. Obtain the depth map S2 of the previous frame.

[0087] S602-2. Obtain the MV map of the previous frame.

[0088] S603-2. Based on the depth D2 of the pixel in the depth map S2, obtain the offset target position P2 of the pixel.

[0089] S604-2. Read the depth d2 of P2.

[0090] S605-2. Based on D2 and d2, determine the depth D of P2.

[0091] Exemplarily, the cloud server performs the same processing on the depth map and the MV map of the previous frame, and obtains the depth offset map T2 corresponding to the previous frame. The specific implementation can refer to S601-1 to S605-1, which will not be elaborated here.

[0092] Exemplarily, after the cloud server moves all pixels in the depth map of the current frame, there may be some blank areas in the depth offset map. For example, after the edge pixels in the depth map of the current frame move to the left, there will be blanks at the original positions of the edge pixels. As Figure 10 shown, the depth offset map 72 includes a blank area 72-1 and a blank area 72-2. Similarly, the depth offset map 73 corresponding to the previous frame may include a blank area 73-1 and a blank area 73-2.

[0093] It should be noted that the positions of each pixel and the blank areas in the embodiments of the present application are only for illustrative purposes, and the present application does not make any limitations.

[0094] Still referring to Figure 10 , exemplarily, the cloud server can obtain the depth offset map 74 (denoted as T) based on the depth offset map 72 (i.e., T1) corresponding to the current frame and the depth offset map 73 (i.e., T2) corresponding to the previous frame.

[0095] Specifically, the cloud server can compare the depths of the same positions in the depth offset maps T1 and T2. In one example, if the pixel at a position in the depth offset map T1 is 0, the depth at the same position in the depth offset map T2 can be obtained. If the depth at the same position in the depth offset map T2 is not 0, the depth value can be written to this position in the depth offset map T1. If the depth at this position in the depth offset map T2 is also 0, the depth at this position in T1 remains 0. That is, this position in both the depth offset maps T1 and T2 is blank. For example Figure 10 the blank area 74-1 in the depth offset map 74 in. In another example, if the depth value of a pixel in the depth offset map T2 is greater than the depth of the pixel at the same position in the depth offset map T1, and the difference is greater than a preset value (for example, 500), the cloud server writes the depth of the pixel at this position in the depth offset map T2 to this position in the depth offset map T1. Exemplarily, the cloud server traverses all pixels one by one based on the above method to obtain the depth offset map 74, denoted as T. Among them, the depth offset map T includes but is not limited to positions P1, P2, P3, and P4. Among them, the positions and quantities are only for illustrative purposes, and the present application does not make limitations.

[0096] S403. Obtain a coordinate mapping map based on the depth offset map, the current frame, and the previous frame.

[0097] Figure 11 For the schematic diagram of the acquisition process of the coordinate mapping map shown exemplarily, please refer to Figure 11 and specifically includes but is not limited to:

[0098] S1101-1. Obtain the depth map of the current frame.

[0099] S1102-1. Obtain the MV map of the current frame.

[0100] S1101-1 to 1102-1 can refer to the above, and will not be elaborated here.

[0101] S1103-1. Calculate the target position Dp1 based on the depth D1 of P1.

[0102] Exemplarily, the cloud server can obtain the depth of any pixel and the corresponding target position Dp1 based on the depth map and the MV map of the current frame.

[0103] Illustrate with an example. As Figure 12 shown, the cloud server can traverse to point A (equivalent to Figure 11The P1 point described in []. The cloud server determines that the depth of point A is D1, and the actual coordinates of pixel A in the current depth map are (x1, y1). The cloud server can determine, based on the depth map and the MV map of the current frame, that the target position after the movement of A is Dp1, where the actual coordinates of Dp1 in the depth offset map are (x2, y2). The acquisition method can refer to the above, and will not be elaborated here.

[0104] S1104-1, Read the depth d1 at the position Dp1 of the depth offset map.

[0105] Exemplarily, the cloud server reads the depth d1 at the position Dp1 (i.e., the coordinates are (x2, y2)) in the depth offset map. For example, still referring to Figure 12 , the cloud server obtains that the depth at the position Dp1 in the depth offset map 74 (i.e., T) is d1.

[0106] S1105-1, Determine whether D1 is equal to d1.

[0107] Exemplarily, the cloud server compares D1 and d1. In one example, if D1 = d1, it can be determined that the depth at the position Dp1 is the depth of P1. It can also be understood that the depth at the position Dp1 is obtained by writing the depth of pixel point A to the position Dp1 in the Figure 6 process shown in [].

[0108] As described above, in the Figure 6 process, multiple pixels may move to the same target position, and the depth of the target position in the depth offset map is the depth corresponding to the pixel with the largest depth value among multiple pixels. Correspondingly, in this step of S1105-1, the cloud server compares the depth in the current frame depth map with the depth in the depth offset map, and can determine that the depth of Dp1 is obtained after the movement of A.

[0109] For another example, still referring to Figure 12 , assuming point B in the current frame depth map, based on the MV map, it can be determined that it also moves to the position Dp1, that is, the target position is Dp1. In this example, since the depth of point B is not equal to the depth of Dp1, S1106-1 will not be executed for point B, and the next position in the current frame depth map will continue to be traversed.

[0110] S1106-1, Write the coordinates of P1 to Dp1.

[0111] Exemplarily, the cloud server determines that the depth D1 of P1 is equal to the depth d1 of Dp1, then it can write the coordinates of P1 to Dp1 as the coordinates of Dp1.

[0112] For example, still referring to Figure 12, the cloud server determines that the depth D1 of A is equal to the depth d1 of Dp1, and the cloud server updates the coordinates of Dp1, that is, writes the actual coordinates (x1, y1) of point A in the current frame depth map to the position of Dp1. Moreover, the cloud server marks that the coordinates of Dp1 are from the current frame, for example, the update source identifier is set to 1.

[0113] In the embodiment of the present application, as Figure 12 shown, the written coordinates of Dp1 are the actual coordinates of point A in the current frame depth map: (x1, y1). Moreover, the actual coordinates of Dp1 in the coordinate mapping map are: (x2, y2). That is to say, the written coordinates corresponding to each position in the coordinate mapping map, which are used to indicate the depth of Dp1 in the depth offset map, are obtained after moving the points corresponding to the written coordinates in the depth map.

[0114] Optionally, the cloud server can generate an initial coordinate mapping map, and the source identifier corresponding to each position in the initial coordinate mapping map is set to 0.5 (which can be set according to actual needs). The cloud server can update the coordinates of each position in the initial coordinate mapping map and update the corresponding source identifier based on the Figure 11 process. Among them, if the coordinate source is the current frame, the identifier is 1, and if the coordinate source is the previous frame, the identifier is 0.

[0115] Exemplarily, as described above, the depth offset map may optionally include one or more blank positions, such as the blank area 74-1. Correspondingly, the coordinate offset map also includes one or more positions (or points) where the coordinates are not written. For example, the points in the area 75-1 are all written coordinates, and the sources of these points are still the initial values, such as 0.5.

[0116] S1101-2, obtain the depth map of the previous frame.

[0117] S1102-2, obtain the MV map of the previous frame.

[0118] S1103-2, calculate the target position Dp2 based on the depth D2 of P2.

[0119] S1104-2, read the depth d2 at the position of Dp2 in the depth offset map.

[0120] S1105-2, determine whether D2 is equal to d2.

[0121] S1106-2, write the coordinates of P2 to Dp2.

[0122] S1107, obtain the coordinate mapping map.

[0123] Exemplarily, the cloud server can also execute the above method on the depth map of the previous frame to update the initial coordinate mapping map, so as to obtain the coordinate mapping map.

[0124] Optionally, for a position whose coordinates have been updated based on the depth map of the current frame, the cloud server may no longer update the coordinates of this position based on the previous frame, thereby reducing the computational complexity.

[0125] Exemplarily, as described above, there may be blank areas in the depth offsets of the current frame and the previous frame. Based on the depth maps of the current frame and the previous frame, the cloud server can as much as possible improve the coordinate mapping map to update the coordinates corresponding to each position in the coordinate mapping map, and obtain the coordinate mapping map.

[0126] S404. Obtain a predicted frame to be completed based on the real frame and the coordinate mapping map.

[0127] Exemplarily, the cloud server can obtain the predicted frame to be completed based on the real frame (including the current frame or the previous frame) and the coordinate mapping map. Specifically, the cloud server can move the pixels in the current frame or the previous frame based on the coordinate mapping map to obtain the predicted frame to be completed. It can be understood that the depth offset map is used to indicate the correct occlusion relationship after the pixel movement. The coordinate mapping map is obtained based on the depth offset map, and its coordinates are used to indicate the actual coordinates of the points corresponding to the correct occlusion relationship after the movement in the real frame. Therefore, the cloud server can move the corresponding points in the real frame based on the coordinate mapping map to obtain the predicted frame to be completed.

[0128] Exemplarily, as described above, the information of each position in the coordinate mapping map includes the real coordinates, the written coordinates, and the source identifier. The cloud server can then obtain the color from the corresponding position in the corresponding real frame (the current frame or the previous frame) based on the written coordinates and the source identifier of each position, and fill it into the predicted frame to be completed.

[0129] For example, Figure 13 For an exemplary schematic diagram of obtaining the predicted frame to be completed, please refer to Figure 13 , the cloud server traverses the position Dp1 in the coordinate mapping map 75, and obtains that the written coordinates of Dp1 are (x1, y1), and the coordinate source is the current frame (i.e., the source identifier is 1).

[0130] The cloud server can search for the pixel corresponding to the coordinates (x1, y1) in the current frame 130 based on the written coordinates (x1, y1) of Dp1, for example, it is pixel A, and obtain the color value (such as the RGB value) of this pixel A.

[0131] The cloud server writes the color value of pixel A into Dp1 of the to-be-completed prediction frame. That is, in the to-be-completed prediction frame, the actual coordinates of pixel Dp1 (i.e., the coordinates in the coordinate system of the to-be-completed prediction frame) are (x1, y1), and the color is the color of pixel A in the current frame. Among them, the coordinates of Dp1 in the to-be-completed prediction frame are the actual coordinates of this position in the to-be-completed prediction frame, which can also be understood as the actual coordinates of Dp1 in the coordinate mapping diagram. And the coordinates of Dp1 recorded in the coordinate mapping diagram can actually be understood as the actual coordinates of the pixel moved to this position in the original image (such as the current frame).

[0132] Exemplarily, as Figure 10 described in, there may optionally be a blank area 74-1 in the depth offset map. Correspondingly, there is also a corresponding blank area 75-1 in the coordinate mapping diagram. Similarly, there is also a blank area 131-1 in the to-be-completed prediction frame 131. The source identifier corresponding to each pixel in the blank area 131-1 is 0.5. Optionally, each pixel in the blank area 131-1 has actual coordinates, and the actual coordinates are the actual coordinates of the pixel in the coordinate system of the to-be-completed prediction frame 131.

[0133] Figure 14 For the schematic diagram of the to-be-completed prediction frame shown exemplarily, please refer to Figure 14 , in the to-be-completed prediction frame 140, it includes pixel 140-2, pixel 140-1, and pixel 140-3. Among them, both pixel 140-2 and pixel 140-3 can obtain the corresponding occlusion relationship based on the current frame or the previous frame and the depth offset map, and perform rendering. And pixel 140-1 is blank. For the specific description, please refer to Figure 1 , which will not be elaborated here.

[0134] S405, update the to-be-completed prediction frame based on the depth offset map, the coordinate mapping diagram, and the real frame to obtain the prediction frame.

[0135] Exemplarily, the cloud server can complete the blank area in the to-be-completed prediction frame based on the depth offset map, the coordinate mapping diagram, and the real frame to obtain the prediction frame.

[0136] Optionally, the cloud server can also generate an initial prediction frame. The cloud server traverses each pixel point in the to-be-completed prediction frame one by one. If the pixel point is non-gap (i.e., non-blank), write the pixel point to the corresponding position in the initial prediction frame. If the pixel point is a gap (i.e., blank), search for non-gap points around the pixel point in the to-be-completed prediction frame, and based on the depth values corresponding to the non-gap points, select a suitable pixel point to write the color of the pixel point in the real frame into the initial prediction frame.

[0137] Figure 15For an exemplary schematic diagram of the acquisition process of the prediction frame, please refer to Figure 15 , which specifically includes but is not limited to the following steps:

[0138] S1501, determine whether point P is a void.

[0139] Exemplarily, the cloud server traverses each pixel point in the prediction frame to be completed one by one, and determines whether point P is a void (i.e., blank). In one example, if the source corresponding to point P is 0.5 (it can also be determined based on the color of point P, which is not limited in this application), it can be determined that point P is a void point, and S1502 is executed. In another example, if the source corresponding to point P is 1 (such as the current frame) or 0 (such as the previous frame), it can be determined that point P is a non-void point, and the color of point P can be written to the corresponding position in the prediction frame. Correspondingly, the coordinate of point P in the prediction frame is the actual coordinate of point P in the prediction frame to be completed, and the color of point P is the color of point P in the prediction frame to be completed.

[0140] S1502, circularly search for surrounding pixels.

[0141] Exemplarily, Figure 16 For an exemplary schematic diagram of pixel search, please refer to Figure 16 , the prediction frame to be completed includes Dp1, Dp2, Dp3, and point P. The cloud server traverses to point P and detects that the source of point P is 0.5, that is, it is located in the blank area 131-1, and determines that point P is a void point.

[0142] The cloud server traverses the pixel points around point P with a step size of 1. The step size of 1 is used to indicate that the interval between the surrounding pixel points and point P is 1 pixel point. As Figure 16 shown, the cloud server traverses the pixel points around point P, including P1 to P8.

[0143] S1503, determine whether P1 is a void.

[0144] Exemplarily, the cloud server traverses the pixel points around point P one by one and determines whether the pixel points are void points. For example, still referring to Figure 16 , the cloud server traverses to point P1 and detects that the source of point P1 is 1 (or 0), and it can be determined that point P1 is a non-void point, and S1504 can be executed for point P1.

[0145] Optionally, if all points P1 to P8 are void points, the cloud server expands the search range and sets the step size to 2 to traverse more pixel points. Optionally, if no non-void points are still detected, the growth rate of the step size can be increased, such as setting the step size to 4 (or 8) to traverse more pixel points.

[0146] In one example, if P1 is a non-gap point, the depth value t1 corresponding to P1 can be obtained from the depth offset map. In another example, if P1 is a gap point, the next pixel point is traversed until a non-gap point pixel is found.

[0147] S1504. Based on the coordinate mapping map, obtain the source of P1.

[0148] S1505. Determine whether P1 is from the current frame.

[0149] Exemplarily, as described above, the source identifier corresponding to each point position is recorded in the coordinate mapping map. For example, the identifier 1 is used to indicate being from the current frame, and the identifier 0 is used to indicate being from the previous frame. The cloud server can determine whether P1 is from the current frame based on the source identifier of P1. In one example, if P1 is from the current frame (for example, the source identifier is 1), then execute S1506-1. In another example, if P1 is from the previous frame (for example, the source identifier is 0), then execute S1506-2.

[0150] S1506-1. Obtain the depths d1 and t1 of point P1 and point P2 in S1 respectively.

[0151] Exemplarily, as Figure 17 shown, the cloud server queries the write coordinates of point P1 with the actual coordinates (x1, y1) in the to-be-completed prediction frame based on the actual coordinates of point P1. For example, the write coordinates of point P1 are (x2, y2). That is to say, point P1 in the to-be-completed prediction frame is obtained by moving the pixel with the actual coordinates (x2, y2) in the current frame, and point P1 in the coordinate mapping map is obtained by moving the point with the actual coordinates (x2, y2) in the current frame depth map.

[0152] The cloud server can find the corresponding point in the current depth map based on the write coordinates of point P1, that is, the point with the actual coordinates (x2, y2), for example, point X.

[0153] The cloud server, based on the relative position between point P1 and point P in the to-be-completed prediction frame (for example, the vector from point P1 to point P), finds the point in the current frame depth map whose relative position with point X is equal to the relative position between point P1 and point P, for example, point W. The actual coordinates of point W in the current frame depth map are (x3, y3).

[0154] The cloud server obtains the depth d1 of point X in the current depth map, and the depth t1 of point W in the current depth map. Among them, the depth d1 of point X is the depth of point P1 in S1, and point W can be understood as Figure 15 the point P2 described in

[0155] S1507-1, Compare d1 and t1.

[0156] S1508-1, Write the color of the point with the smaller depth value to point P.

[0157] Exemplarily, the cloud server compares d1 and t1. In one example, if d1 is greater than t1, the color of the point corresponding to t1 (such as point W) is written to point P. In another example, if d1 is less than t1, the color of the point corresponding to d1 (such as point X) is written to point P.

[0158] For illustration, please refer to Figure 18 , assuming d1 is less than t1, the cloud server can determine to write the color of the pixel point corresponding to point W to point P in the to-be-completed prediction frame. Exemplarily, the cloud server can, based on the actual coordinates (x3, y3) of W in the current frame depth map, the pixel point W in the current frame, and the actual coordinates of pixel point W in the current frame are (x3, y3). The cloud server obtains the color value (i.e., color) of pixel point W, and the cloud server writes the color value of pixel point W to point P in the to-be-completed prediction frame.

[0159] Still referring to Figure 14 , exemplarily, the cloud server, based on Figure 15 the process in, can write the color of the pixel of pixel S140-3 to pixel S140-1, that is, write the color of the pixel whose depth is closer to the blank pixel to the blank pixel to avoid the problem of image distortion.

[0160] S1506-2, Obtain the depths d1 and t1 of point P1 and point P2 in S1 respectively.

[0161] S1507-2, Compare d1 and t1.

[0162] S1508-2, Write the color of the point with the smaller depth value to point P.

[0163] S1506-2 to S1508-2 can refer to S1501-1 to S1508-1, which will not be elaborated here.

[0164] S1509, Obtain the prediction frame.

[0165] Exemplarily, the cloud server executes S1501 to S1508-1 (or S1508-2) on all pixel points in the to-be-completed prediction frame to fill all the gap points in the to-be-completed prediction, thereby obtaining a complete prediction frame.

[0166] It can be understood that, in order to implement the above functions, the electronic device includes corresponding hardware and / or software modules for executing each function. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0167] In one example, Figure 19 FIG. shows a schematic block diagram of a device 1900 according to an embodiment of the present application. The device 1900 may include: a processor 1901 and a transceiver / transceiver pin 1902. Optionally, it may further include a memory 1903.

[0168] Each component of the device 1900 is coupled together through a bus 1904. Among them, the bus 1904 includes, in addition to the data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, all various buses are referred to as the bus 1904 in the figure.

[0169] Optionally, the memory 1903 may be used for the instructions in the foregoing method embodiments. The processor 1901 may be used to execute the instructions in the memory 1903, control the receiving pin to receive signals, and control the transmitting pin to transmit signals.

[0170] The device 1900 may be the electronic device or the chip of the electronic device in the foregoing method embodiments.

[0171] Among them, all relevant contents of each step involved in the foregoing method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated herein.

[0172] This embodiment also provides a computer storage medium. Computer instructions are stored in the computer storage medium. When the computer instructions run on an electronic device, the electronic device is caused to execute the above-related method steps to implement the method in the above embodiments.

[0173] This embodiment also provides a computer program product. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the method in the above embodiments.

[0174] In addition, an embodiment of the present application further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer-executable instructions. When the device runs, the processor may execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0175] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0176] As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image prediction method, which is executed by an electronic device, characterized in that, it includes: Obtain the depth map and motion vector map of the image frame; Move the pixels in the depth map according to the motion vector map to obtain an offset map; Write the color in the image frame according to the offset map to obtain a predicted image frame; Wherein, the moving the pixels in the depth map according to the motion vector map includes: Move the first pixel in the depth map according to the first motion vector in the motion vector map, wherein, after the first pixel is moved, it is in the first position, and the depth of the first pixel in the depth map is the first depth; Write the first depth to the first position; Move the second pixel in the depth map according to the second motion vector in the motion vector map, wherein, after the second pixel is moved, it is in the first position, and the depth of the second pixel in the depth map is the second depth; After the first depth is written to the first position, when the second depth is less than the first depth, write the second depth to the first position.

2. The method according to claim 1, characterized in that, The moving the pixels in the depth map according to the motion vector map further includes: Move the third pixel in the depth map according to the third motion vector in the motion vector map, wherein, after the third pixel is moved, it is in the first position, and the depth of the third pixel in the depth map is the third depth; After the first depth is written to the first position, when the third depth is greater than the first depth, the third depth is not written to the first position.

3. The method according to claim 2, characterized in that, The distance between the third pixel and the screen of the electronic device is greater than the distance between the first pixel and the screen of the electronic device.

4. The method according to any one of claims 1-3, characterized in that, The distance between the second pixel and the screen of the electronic device is less than the distance between the first pixel and the screen of the electronic device.

5. The method according to any one of claims 1-4, characterized in that, The image frame is a single frame of image.

5. The method according to any one of claims 1-4, characterized in that, The image frame is a single frame of image.

6. The method according to any one of claims 1-5, characterized in that, The writing the color in the image frame according to the offset map includes: Write the color of the pixel at the fifth position in the image frame to the fourth position, wherein, the fifth position is determined according to the offset map and the image frame, and the fourth position is in the predicted image frame.

7. The method according to any one of claims 1-6, characterized in that, Before obtaining the depth map and motion vector map of the image frame, the method further includes: Run a game application, and the screen of the game application includes the image frame.

8. An electronic device, characterized in that, it includes: One or more processors, a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, cause the electronic device to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that it includes computer instructions, and when the computer instructions run on an electronic device, cause the electronic device to perform the method according to any one of claims 1-7.

10. A computer program product, characterized in that when the computer program product runs on a computer, cause the computer to perform the method according to any one of claims 1-7.

11. A chip system, characterized in that it includes one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor, and the signal includes computer instructions stored in the memory; when the processor executes the computer instructions, cause the electronic device to perform the method according to any one of claims 1-7.

Images