Image processing method and device

Abstract

The invention provides an image processing method and electronic equipment, and the method comprises the steps: obtaining a depth offset map based on a depth map of an image frame, and enabling the depth offset map to be used for indicating the correct shielding relation between pixels. And moving pixels in the image frame based on the depth offset map to obtain a prediction frame. Based on the depth of the pixel of the image frame, the image authenticity of the prediction frame can be improved in a mode of obtaining the prediction frame, and the problem of image distortion is avoided.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and device. Background Art

[0002] As the quality of mobile games continues to improve, higher demands are placed on mobile phone chips and mobile phone designs. Due to limitations such as chip computing power and mobile phone heat dissipation, the mobile game experience cannot have both high image quality and high frame rate. At present, manufacturers have introduced a prediction frame solution to achieve a high frame rate, that is, using two adjacent frames to generate an intermediate prediction frame under the condition of low frame rate and high image quality, thereby achieving the goal of high image quality and high frame rate. However, the existing interpolation solution has the problem of image distortion due to imperfect processing of prediction frames, which reduces the user's gaming experience. Summary of the invention

[0003] The present application provides an image processing method and an electronic device. In the method, the electronic device can generate a prediction frame based on the depth of each pixel in the image frame to improve the picture authenticity of the prediction frame.

[0004] In a first aspect, the present application provides an image processing method. Applied to an electronic device, the method includes: obtaining a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame. All pixels in the first depth map are moved to obtain a first depth offset map, and the first depth offset map is used to indicate the occlusion relationship of all pixels in the first depth map. All pixels in the second depth map are moved to obtain a second depth offset map, and the second depth offset map is used to indicate the occlusion relationship of all pixels in the second depth map. Based on the first depth offset map and the second depth offset map, a target depth offset map is obtained. Based on the target depth offset map, all pixels in the first image frame or the second image frame are moved to obtain a target predicted frame. In this way, the present application can obtain the correct occlusion relationship of each pixel in the predicted frame based on the depth of each pixel in the real frame, which can effectively avoid the image distortion problem in the predicted frame and improve the user's visual experience.

[0005] Exemplarily, the predicted frame is an image frame between two real frames.

[0006] Illustratively, the electronic device may be a terminal, a server, etc.

[0007] Exemplarily, the first image frame is the current frame, and the second image frame is the previous frame.

[0008] In a possible implementation, the moving of all pixels in the first depth map to obtain a first depth offset map includes: determining the first position of the first pixel in the first depth map in the first depth offset map based on the first motion vector MV map of the first image frame; if the first depth value of the first pixel in the first depth map is greater than the depth value of the first position, writing the first depth value to the first position; determining the second position of the second pixel in the first depth map in the first depth map based on the first MV map; if the first position is the same as the second position, and the second depth value of the second pixel in the first depth map is greater than the first depth value, writing the second depth value to the first position. In this way, the present application can obtain the target position of each pixel of the real frame through the MV map, and can obtain the depth offset map based on the depth of each pixel, so as to determine the correct occlusion relationship between each pixel in the predicted frame based on the depth offset map.

[0009] In a possible implementation, the moving of all pixels in the second depth map to obtain a second depth offset map includes: determining the third position of the third pixel in the second depth map in the second depth offset map based on the second MV map of the second image frame; if the third depth value of the third pixel in the second depth map is greater than the depth value of the third position, writing the third depth value to the third position; determining the fourth position of the fourth pixel in the second depth map in the second depth map based on the second MV map; if the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, writing the fourth depth value to the third position. In this way, the present application can obtain the target position of each pixel of the real frame through the MV map, and can obtain the depth offset map based on the depth of each pixel, so as to determine the correct occlusion relationship between the pixels in the predicted frame based on the depth offset map.

[0010] In a possible implementation, the obtaining of the target depth offset map based on the first depth offset map and the second depth offset map includes: writing the depth values ​​corresponding to all positions in the first offset map into the target depth offset map, wherein, for positions in the target depth offset map where no depth values ​​are written, the depth values ​​are obtained from the corresponding positions in the second depth offset map and written. In this way, the present application can combine the previous frame and the current frame to obtain the depth values ​​of all points in the depth offset map as much as possible.

[0011] In a possible implementation, the method of moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target predicted frame includes: obtaining a coordinate mapping map based on the target depth offset map, the first image frame, and the second image frame; wherein the coordinate mapping map is used to indicate the pixel points corresponding to all positions in the target depth offset map in the first image frame or the second image frame. In this way, the present application can obtain a coordinate mapping map based on the correct occlusion relationship in the depth offset map, thereby obtaining the mapping relationship between each point between the real frame and the depth offset map.

[0012] In one possible implementation, based on the target depth offset map, all pixels in the first image frame or the second image frame are moved to obtain a target prediction frame, including: based on the coordinate mapping map, the first image frame and the second image frame, obtaining the prediction frame to be completed. In this way, in the present application, the pixels in the first image frame and the second image frame can be moved based on the coordinate mapping map to obtain the prediction frame to be completed. Among them, the coordinate mapping map is obtained based on the depth offset map that can reflect the correct occlusion relationship. Therefore, the occlusion relationship between each pixel in the prediction frame to be completed is accurate, and there will be no image distortion problem.

[0013] In a possible implementation, the method of moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: searching for gap points in the prediction frame to be completed; determining a fifth pixel from a plurality of pixels around the gap point, wherein the fifth pixel is a non-gap point; searching for a sixth pixel corresponding to the fifth pixel in the first image frame or the second image frame based on the coordinate mapping map; determining a seventh pixel in the first image frame or the second image frame based on a first relative position between the gap point and the fifth pixel, wherein a second relative position between the seventh pixel and the sixth pixel is the same as the first relative position; obtaining a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel based on the first depth map or the second depth map; and writing the color of the pixel corresponding to the smaller depth value between the seventh depth value and the sixth depth value into the gap point. In this way, the present application can find the point closest to the depth of the gap point based on the mapping relationship and depth between the pixels in the real frame and the coordinate mapping map, and fill the gap point with the color of the point, thereby avoiding the problem of image distortion.

[0014] In a second aspect, the present application provides an electronic device. It includes: one or more processors, a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: obtaining a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame; moving all pixels in the first depth map to obtain a first depth offset map, the first depth offset map is used to indicate the occlusion relationship of all pixels in the first depth map; moving all pixels in the second depth map to obtain a second depth offset map, the second depth offset map is used to indicate the occlusion relationship of all pixels in the second depth map; based on the first depth offset map and the second depth offset map, obtaining a target depth offset map; based on the target depth offset map, moving all pixels in the first image frame or the second image frame to obtain a target prediction frame.

[0015] In one possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: based on the first motion vector MV map of the first image frame, determining a first position of a first pixel in a first depth map in the first depth offset map; if a first depth value of the first pixel in the first depth map is greater than the depth value of the first position, writing the first depth value to the first position; based on the first MV map, determining a second position of a second pixel in the first depth map in the first depth map; if the first position is the same as the second position, and the second depth value of the second pixel in the first depth map is greater than the first depth value, writing the second depth value to the first position.

[0016] In one possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: based on the second MV map of the second image frame, determining the third position of the third pixel in the second depth map in the second depth offset map; if the third depth value of the third pixel in the second depth map is greater than the depth value of the third position, writing the third depth value to the third position; based on the second MV map, determining the fourth position of the fourth pixel in the second depth map in the second depth map; if the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, writing the fourth depth value to the third position.

[0017] In one possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: writing the depth values ​​corresponding to all positions in the first offset map into the target depth offset map, wherein, for positions in the target depth offset map where depth values ​​are not written, depth values ​​are obtained from corresponding positions in the second depth offset map and written.

[0018] In one possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: acquiring a coordinate mapping map based on the target depth offset map, the first image frame, and the second image frame; wherein the coordinate mapping map is used to indicate the pixel points corresponding to all positions in the target depth offset map in the first image frame or the second image frame.

[0019] In a possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: acquiring a prediction frame to be completed based on the coordinate mapping diagram, the first image frame, and the second image frame.

[0020] In one possible implementation, when the computer program is executed by the one or more processors, the electronic device performs the following steps: searching for gap points in the prediction frame to be completed; determining a fifth pixel point from a plurality of pixel points around the gap point, wherein the fifth pixel point is a non-gap point; searching for a sixth pixel corresponding to the fifth pixel point in the first image frame or the second image frame based on the coordinate mapping diagram; determining a seventh pixel in the first image frame or the second image frame based on a first relative position between the gap point and the fifth pixel, wherein a second relative position between the seventh pixel and the sixth pixel is the same as the first relative position; obtaining a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel based on the first depth map or the second depth map; and writing the color of the pixel corresponding to the smaller depth value between the seventh depth value and the sixth depth value into the gap point.

[0021] In a third aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program, comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0023] In a fifth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of an exemplary prediction frame;

[0025] Figure 2 A schematic diagram showing an exemplary prediction frame display;

[0026] Figure 3 is a schematic diagram showing the principle of an image processing method;

[0027] Figure 4 is a flowchart of an exemplary image processing method;

[0028] Figure 5 The following is a schematic diagram of the process of obtaining a depth map and a MV map as shown in an exemplary manner;

[0029] Figure 6 is a schematic diagram of obtaining a depth offset map shown exemplarily;

[0030] Figure 7 is a schematic diagram of a depth map shown as an example;

[0031] Figure 8 is a schematic diagram of obtaining a depth offset map shown exemplarily;

[0032] Fig. 9 is a schematic diagram of obtaining a depth offset map shown exemplarily;

[0033] Fig.10 is a schematic diagram of obtaining a depth offset map shown exemplarily;

[0034] Fig.11 The figure is a schematic diagram of the process of obtaining a coordinate mapping diagram shown as an example;

[0035] Fig.12 is a schematic diagram for obtaining a coordinate mapping diagram shown exemplarily;

[0036] Fig.13 is a schematic diagram of obtaining a prediction frame to be completed as an example;

[0037] Fig.14 is a schematic diagram of an exemplary prediction frame to be completed;

[0038] Fig.15 is a schematic diagram of an exemplary prediction frame acquisition process;

[0039] Fig.16 is a schematic diagram of an exemplary pixel search;

[0040] Fig.17 The figure is a schematic diagram of obtaining a prediction frame shown as an example;

[0041] Fig.18 The figure is a schematic diagram of obtaining a prediction frame shown as an example;

[0042] Fig.19 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0045] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0046] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0047] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0048] The solution in the embodiment of the present application is applied to a cloud server, hereinafter referred to as a cloud server. Among them, the cloud server can be a server cluster composed of one or more servers, which is not limited in this application. Specifically, the cloud server generates image frames of the game in the cloud, and sends the image frames to the game terminal (such as a terminal, a mobile phone, a wearable device, etc.), and the game terminal receives and displays the image frames. Exemplarily, in the embodiment of the present application, only the game scene is used as an example for explanation. In other embodiments, the image processing method in the embodiment of the present application can also be applied to other video streaming scenes, which is not limited in this application.

[0049] For example, in order to improve the frame rate of the game, the game manufacturer provides a prediction frame solution, that is, under the condition of low frame rate and high image quality, two adjacent frames are used to generate an intermediate prediction frame, thereby achieving the goal of high frame rate while ensuring high image quality. Figure 1 For a schematic diagram of an exemplary prediction frame, please refer to Figure 1 In an embodiment of the present application, the cloud server inserts a prediction frame between two image frames to increase the frame rate. To better illustrate the embodiment of the present application, the image frame before the prediction frame is referred to as the previous frame, and the frame after the prediction frame is referred to as the current frame. That is to say, in an embodiment of the present application, a prediction frame can be generated based on the previous frame and the current frame, and the prediction frame can be inserted between the previous frame and the current frame. Optionally, the previous frame can also be referred to as the first frame (or the first image frame), and the current frame can also be referred to as the second frame (or the second image frame), which is not limited in this application.

[0050] Figure 2 For an exemplary diagram of the predicted frame display, please refer to Figure 2 ,Exemplarily, the predicted frame is generated based on the previous frame and the current frame, e.g. Figure 2 (1) is the previous frame 100. Due to the movement of the camera, the object moves in the picture. For example, image 101 shifts to the left side of the picture. That is, the picture to be displayed by the predicted frame is as follows: Figure 2 As shown in the image frame 110 in (2), the image 101 is offset to the left compared to the previous frame, and the portion originally blocked by the image 101 in the previous frame 100 will be displayed in the image frame. However, since the previous frame 100 does not include the blocked portion, the cloud server needs to patch the blank image 102 to avoid a blank area in the image of the predicted frame.

[0051] like Figure 2 As shown in (2), the cloud server may add the image of image 101 to the blank image 102, and add the original image part of the pillar to the area that should have been the ground, which causes the pillar in the predicted image frame to be distorted. From the user's perspective, the pillar in the image 102 of the image displayed in the predicted frame 130 suddenly becomes wider, resulting in distortion, which affects the user experience.

[0052] The present invention provides a solution for generating a prediction frame pixel by pixel, which can effectively solve the image distortion problem in prediction frame drawing. Figure 3 For an exemplary schematic diagram of the principle of the image processing method in the embodiment of the present application, please refer to Figure 3 , including but not limited to:

[0053] S301, global depth pixel movement.

[0054] Exemplarily, the cloud server may move pixels in the depth map corresponding to the image frame based on the depth map and MV map (Motion Vector) of the previous frame.

[0055] S302: Obtain a depth offset map, where the depth offset map is used to indicate an occlusion relationship.

[0056] The cloud server can obtain a depth offset map with correct occlusion relationship based on the depth of each pixel after the movement.

[0057] S303: Move the original image based on the depth offset image.

[0058] For example, the cloud server moves the pixels of the previous frame and the current frame based on the depth offset map to obtain a high-quality predicted frame. Since the correct image shape and mapping rules are maintained, the predicted frame is basically the same as the real rendered frame, and the high image quality is maintained while doubling the number of frames.

[0059] Combine the following Figure 4 The flowchart of the image processing method shown in FIG. Figure 2 For detailed instructions on the steps in Figure 4 , including but not limited to the following steps:

[0060] S401, obtaining the depth map and MV map of the current frame and the previous frame.

[0061] Figure 5 For an exemplary schematic diagram of the acquisition process of the depth map and MV map, please refer to Figure 5 ,Exemplarily, the processing of image frames by a cloud server usually includes: receiving game rendering instruction stream, resource preparation, game rendering, and obtaining real frames (i.e., image frames, such as the previous frame or the current frame) stages.

[0062] When the cloud server executes the above stages to obtain the previous frame or the current frame, the cloud server can obtain the corresponding resources from the stages of resource preparation, game rendering, and obtaining the real frame, and obtain the MV map or matrix information of the real frame, the depth map of the real frame, and the real frame from the obtained resources. The real frame includes the previous frame and the current frame, that is, two adjacent image frames, which can also be called the first image frame and the second image frame. That is to say, in Figure 5 In the process shown, the cloud server can obtain the matrix information or MV map of the previous frame (also called the previous image frame, which will not be repeated below), the depth map of the previous frame, and the previous frame (that is, the image of the previous frame, which can also be called the rendering of the previous frame); and the cloud server also obtains the matrix information or MV map of the current frame (also called the current image frame, which will not be repeated below), the depth map of the current frame and the current frame (that is, the image of the current frame, which can also be called the rendering of the current frame).

[0063] Exemplarily, in the rendering process of the real frame, since coordinate mapping is required, the matrix used to calculate the vertex position of the object is the matrix information in the embodiment of the present application, that is, the matrix information and the MV diagram are both used to indicate the movement direction and movement distance of the camera. In one example, the cloud server can directly obtain the MV diagram corresponding to the real frame from the resource. In another example, the cloud server can also obtain the matrix information corresponding to the real frame, and obtain the corresponding MV diagram based on the matrix information. The specific acquisition method can refer to the existing technology, and this application is not limited.

[0064] Exemplarily, the depth map of the real frame is used to indicate the depth corresponding to each pixel in the image frame. Among them, the depth of the pixel closer to the screen is greater, and the depth of the pixel farther from the screen is smaller. Optionally, in other embodiments, the depth value of the pixel closer to the screen can be smaller, and conversely, the depth value of the pixel farther from the screen can be larger, which is not limited in this application.

[0065] S402, obtaining a depth offset map based on the depth map and MV map of the current frame and the previous frame.

[0066] The cloud server can obtain a depth offset map with a correct occlusion relationship based on the depth of each pixel after the movement, that is, in an embodiment of the present application, the depth offset map can be used to indicate the occlusion relationship between pixels.

[0067] Figure 6 For an exemplary diagram of obtaining a depth offset map, please refer to Figure 6 , including but not limited to:

[0068] S601-1, obtain the depth map S1 of the current frame.

[0069] Figure 7 For an exemplary depth map diagram, please refer to Figure 7 , the current frame original image can also be called the current frame rendering image. The current frame rendering image includes multiple pixels, each pixel includes coordinates and colors. The coordinates can optionally be a coordinate system with the upper left corner of the image as the origin, the horizontal direction as the X axis, and the vertical direction as the Y axis, and the color can also be called a color value.

[0070] As described above, the cloud server can obtain the depth map 70 of the current frame, denoted as S1, wherein the depth map includes the depth value of each pixel. For example, the depth of pixel 701 is A, and the depth of pixel 702 is B, wherein A is greater than B, that is, the distance between pixel 701 and the screen is less than the distance between pixel 702 and the screen, that is, pixel 701 is closer to the screen, and pixel 702 is farther from the screen than pixel 701.

[0071] S602-1, obtain the MV image of the current frame.

[0072] For example, the cloud server can be based on Figure 5 The MV diagram of the current frame is obtained in the manner described in . Optionally, the cloud server may directly obtain the MV diagram, or may obtain the matrix information corresponding to the current frame, and obtain the corresponding MV diagram based on the matrix information, which is not limited in this application.

[0073] S603 - 1 , based on the depth D1 of the pixel in the depth map S1 , obtain the offset target position P1 of the pixel.

[0074] Exemplarily, the cloud server may move pixels in the depth map of the current frame based on the MV map of the current frame to move the pixels to a target position.

[0075] Figure 8 For an exemplary diagram of obtaining a depth offset map, please refer to Figure 8 , exemplarily, the cloud server obtains the current frame depth map 70, namely S1, and the cloud server obtains the current frame MV map 71. Taking pixel X in the current frame depth map as an example, the depth of pixel X is Dx. The cloud server moves pixel X based on the offset △X corresponding to pixel X in the MV map to determine the target position P1 of pixel X in the depth offset map 72 (namely T1) corresponding to the current frame. Among them, △X is used to indicate the moving direction and moving distance of pixel X.

[0076] S604-1, read the depth d1 of P1.

[0077] Optionally, due to the movement of the camera, multiple pixels of the current frame may move to the same target position. In the embodiment of the present application, the depth of the target position is the maximum depth of the multiple pixels. For example, after pixel A, pixel B and pixel C move, their target positions are all point P. The depths of pixel A, pixel B and pixel C are depth A, depth B and depth C from small to large, respectively. Therefore, the depth of the target position is the depth of pixel C, that is, depth C. In other words, due to the movement of the camera, the pixels of the current frame move with the movement. When different pixels move to the same position, the occlusion relationship can be determined based on the depth. That is, in the above example, pixel A, pixel B and pixel C will all move to the target position P, and the depth of pixel C is the largest. Therefore, pixel C is closest to the screen and will block pixel B and pixel C. It can be understood that in the predicted frame, pixel C will be displayed at the position P, while pixel B and pixel A will not be displayed.

[0078] Exemplarily, the cloud server reads the depth value d1 of the target position P1. Optionally, the initial depth of the target position P1 can be 0 (can be set according to actual needs, not limited by this application). When the depth value of the pixel is written into the target position P1, the depth value of the target position P1 changes accordingly.

[0079] Still refer to Figure 8 In this example, it is assumed that the current depth d1 of the target position P1 is 0, that is, the depth at which no pixel has been written.

[0080] S605 - 1 , based on D1 and d1 , determine the depth D of P1 .

[0081] Exemplarily, the cloud server compares the current depth d1 of the target position P1 with the depth D1 of the pixel. In one example, if D1 is less than d1, that is, the depth of the pixel moved to the target position P1 is less than the current depth of the target position P1, the depth of the target position P1 is not updated. In another example, if D1 is greater than d1, that is, the depth of the pixel moved to the target position P1 is greater than the current depth of the target position P1, the depth of the pixel is written.

[0082] Still refer to Figure 8 , exemplarily, the cloud server obtains the current depth d1 of the target position P1, assuming that the current depth of the target position is 0, and the cloud server compares d1 with the depth Dx of pixel X. In this example, the depth Dx of pixel X is greater than the current depth d1 of P1 (that is, 0), and the cloud server writes the depth Dx of pixel X to the target position P1, that is, the depth D of the target position P1 is updated to Dx.

[0083] Exemplarily, the cloud server repeatedly executes S603-1 to S605-1. For example, Fig. 9As shown, the cloud server traverses to pixel Y in the current frame depth map S1, and the depth of pixel Y is Dy. Based on the MV map, the cloud server moves pixel Y and determines that the target position of pixel Y in the depth offset map T1 is also P1. The cloud server compares the current depth d1 (that is, Dx) of the target position P1 with the depth Dy of pixel Y. In one example, if Dy is less than d1, the depth of the target position P1 is not updated, that is, the depth of the target position P1 remains Dx. In this example, Dy is greater than d1, then the cloud server updates the depth of the target position P1, and the updated depth D of the target position P1 is Dy.

[0084] Exemplarily, the cloud server loops through all pixels in the current frame depth map S1 and obtains the depth offset map T1.

[0085] S601-2, obtain the depth map S2 of the previous frame.

[0086] S602-2, obtain the MV image of the previous frame.

[0087] S603 - 2 , based on the depth D2 of the pixel in the depth map S2 , obtain the offset target position P2 of the pixel.

[0088] S604-2, read the depth d2 of P2.

[0089] S605-2, based on D2 and d2, determine the depth D of P2.

[0090] Exemplarily, the cloud server performs the same processing on the depth map and MV map of the previous frame to obtain the depth offset map T2 corresponding to the previous frame. The specific implementation can refer to S601-1 to S605-1, which will not be repeated here.

[0091] For example, after the cloud server moves all pixels in the depth map of the current frame, some blank areas may appear in the depth offset map. For example, after the edge pixels in the depth map of the current frame move to the left, the original positions of the edge pixels will appear blank. Fig.10 As shown, the depth offset map 72 includes a blank area 72-1 and a blank area 72-2. Similarly, the depth offset map 73 corresponding to the previous frame may include a blank area 73-1 and a blank area 73-2.

[0092] It should be noted that the positions of the pixels and blank areas in the embodiments of the present application are only for illustrative purposes and are not limited in this application.

[0093] Still refer to Fig.10 For example, the cloud server may obtain a depth offset map 74 (denoted as T) based on a depth offset map 72 (ie, T1) corresponding to a current frame and a depth offset map 73 (ie, T2) corresponding to a previous frame.

[0094] Specifically, the cloud server can compare the depths of the same position in the depth offset map T1 and T2. In one example, if the pixel of a position in the depth offset map T1 is 0, the depth of the same position in the depth offset map T2 can be obtained. If the depth of the same position in the depth offset map T2 is not 0, the depth value can be written to the position in the depth offset map T1. If the depth of the position in the depth offset map T2 is also 0, the depth of the position in T1 is still 0. In other words, the position in the depth offset map T1 and T2 is blank. For example Fig.10 In another example, if the depth value of a pixel in the depth offset map T2 is greater than the depth of the pixel at the same position in the depth offset map T1, and the difference is greater than a preset value (for example, 500), the cloud server writes the depth of the pixel at that position in the depth offset map T2 to that position in the depth offset map T1. Exemplarily, the cloud server traverses all pixels one by one based on the above method to obtain a depth offset map 74, denoted as T. The depth offset map T includes but is not limited to position P1, position P2, position P3 and position P4. The positions and quantities are only illustrative examples and are not limited in this application.

[0095] S403, acquiring a coordinate mapping map based on the depth offset map and the current frame and the previous frame.

[0096] Fig.11 For an exemplary schematic diagram of the acquisition process of the coordinate mapping diagram, please refer to Fig.11 , including but not limited to:

[0097] S1101-1, obtain the depth map of the current frame.

[0098] S1102-1, obtain the MV image of the current frame.

[0099] S1101-1~1102-1 can be referred to above and will not be repeated here.

[0100] S1103-1, based on the depth D1 of P1, calculate the target position Dp1.

[0101] Exemplarily, the cloud server may obtain the depth of any pixel and the corresponding target position Dp1 based on the depth map and MV map of the current frame.

[0102] For example, Fig.12 As shown, the cloud server can traverse to point A (equivalent to Fig.11The cloud server determines the depth of point A as D1, and the actual coordinates of pixel A in the current depth map are (x1, y1). The cloud server can determine that the target position after A moves is Dp1 based on the depth map and MV map of the current frame, where the actual coordinates of Dp1 in the depth offset map are (x2, y2). The acquisition method can refer to the above, and will not be repeated here.

[0103] S1104-1, read the depth d1 of the position Dp1 in the depth offset map.

[0104] For example, the cloud server reads the depth d1 at the position Dp1 (i.e., the coordinates are (x2, y2)) in the depth offset map. Fig.12 , the cloud server obtains the depth of position Dp1 in the depth offset map 74 (ie, T) as d1.

[0105] S1105-1, determine whether D1 is equal to d1.

[0106] Exemplarily, the cloud server compares D1 and d1. In one example, if D1=d1, it can be determined that the depth of the position Dp1 is the depth of P1. It can also be understood that the depth of the position Dp1 is Figure 6 In the process shown in , the depth of pixel point A is written to the position Dp1.

[0107] As mentioned above, in Figure 6 In the process, multiple pixels may move to the same target position, and the depth of the target position in the depth offset map is the depth corresponding to the pixel with the largest depth value among the multiple pixels. Accordingly, in step S1105-1, the cloud server compares the depth in the current frame depth map with the depth in the depth offset map to determine that the depth of Dp1 is obtained after A moves.

[0108] Let’s take another example, still referring to Fig.12 , assuming that point B in the current frame depth map is also moved to position Dp1 based on the MV map, that is, the target position is Dp1. In this example, the depth of point B is not equal to the depth of Dp1, then S1106-1 will not be executed for point B, and the next position in the current frame depth map will continue to be traversed.

[0109] S1106-1, the coordinates of P1 are written into Dp1.

[0110] Exemplarily, the cloud server determines that the depth D1 of P1 is equal to the depth d1 of Dp1, and then the coordinates of P1 may be written into Dp1 as the coordinates of Dp1.

[0111] For example, refer to Fig.12, the cloud server determines that the depth D1 of A is equal to the depth d1 of Dp1, and the cloud server updates the coordinates of Dp1, that is, writes the actual coordinates (x1, y1) of point A in the current frame depth map to the position of Dp1. In addition, the cloud server marks the coordinates of Dp1 as coming from the current frame, for example, updating the source identifier to 1.

[0112] In the embodiments of the present application, Fig.12 As shown, the written coordinates of Dp1 are the actual coordinates of point A in the current frame depth map: (x1, y1). And the actual coordinates of Dp1 in the coordinate mapping map are: (x2, y2). In other words, the written coordinates corresponding to each position in the coordinate mapping map are used to indicate the depth of Dp1 in the depth offset map, which is obtained by moving the corresponding points of the written coordinates in the depth map.

[0113] Optionally, the cloud server may generate an initial coordinate map, and the source identifier corresponding to each position in the initial coordinate map is set to 0.5 (which may be set according to actual needs). Fig.11 According to the process in , the coordinates of each position in the initial coordinate map are updated, and the corresponding source identifier is updated. If the coordinate source is the current frame, the identifier is 1, and if the coordinate source is the previous frame, the identifier is 0.

[0114] Exemplarily, as described above, the depth offset map may optionally include one or more blank positions, such as blank area 74-1. Correspondingly, the coordinate offset map also includes one or more positions (or points) where coordinates are not written, such as points in area 75-1 where coordinates are written, and the sources of these points are still initial values, such as 0.5.

[0115] S1101-2, obtain the depth map of the previous frame.

[0116] S1102-2, obtain the MV image of the previous frame.

[0117] S1103-2, based on the depth D2 of P2, calculate the target position Dp2.

[0118] S1104-2, read the depth d2 of the depth offset map Dp2.

[0119] S1105-2, determine whether D2 is equal to d2.

[0120] S1106-2, the coordinates of P2 are written into Dp2.

[0121] S1107, obtaining a coordinate mapping diagram.

[0122] Exemplarily, the cloud server may also perform the above method on the depth map of the previous frame to update the initial coordinate mapping map, thereby obtaining a coordinate mapping map.

[0123] Optionally, for a position whose coordinates have been updated based on the depth map of the current frame, the cloud server may no longer update the coordinates of the position based on the previous frame, thereby reducing computational complexity.

[0124] For example, as described above, the depth offsets of the current frame and the previous frame may have blank areas. Based on the depth maps of the current frame and the previous frame, the cloud server can improve the coordinate mapping map as much as possible to update the coordinates corresponding to each position in the coordinate mapping map to obtain the coordinate mapping map.

[0125] S404, obtaining a prediction frame to be completed based on the real frame and the coordinate mapping graph.

[0126] Exemplarily, the cloud server can obtain the predicted frame to be completed based on the real frame (including the current frame or the previous frame) and the coordinate mapping map. Specifically, the cloud server can move the pixels in the current frame or the previous frame based on the coordinate mapping map to obtain the predicted frame to be completed. It can be understood that the depth offset map is used to indicate the correct occlusion relationship after the pixel is moved, and the coordinate mapping map is obtained based on the depth offset map, and each coordinate thereof is used to indicate the actual coordinates of the point corresponding to the correct occlusion relationship after the movement in the real frame. Therefore, the cloud server can move the corresponding points in the real frame based on the coordinate mapping map to obtain the predicted frame to be completed.

[0127] Exemplarily, as described above, the information of each position in the coordinate mapping diagram includes real coordinates, written coordinates and source identification. The cloud server can obtain the color from the corresponding position in the corresponding real frame (current frame or previous frame) based on the written coordinates and source identification of each position, and fill it into the predicted frame to be completed.

[0128] For example, Fig.13 For an exemplary diagram of obtaining a prediction frame to be completed, please refer to Fig.13 , the cloud server traverses the position Dp1 in the coordinate map 75, and obtains the write coordinates of Dp1 as (x1, y1), and the coordinate source is the current frame (ie, the source identifier is 1).

[0129] The cloud server may search for the pixel corresponding to the coordinate (x1, y1) in the current frame 130 , such as pixel A, based on the written coordinate (x1, y1) of Dp1, and obtain the color value (eg, RGB) of pixel A.

[0130] The cloud server writes the color value of pixel A into Dp1 of the prediction frame to be completed, that is, in the prediction frame to be completed, the actual coordinates of pixel Dp1 (i.e., the coordinates in the coordinate system of the prediction frame to be completed) are (x1, y1), and the color is the color of pixel A in the current frame. Among them, the coordinates of Dp1 in the prediction frame to be completed are the actual coordinates of the position in the prediction frame to be completed, which can also be understood as the actual coordinates of Dp1 in the coordinate mapping diagram. The coordinates of Dp1 recorded in the coordinate mapping diagram can actually be understood as the actual coordinates of the pixel moved to this position in the original image (such as the current frame).

[0131] For example, Fig.10 As described in, a blank area 74-1 may optionally exist in the depth offset map, and correspondingly, a corresponding blank area 75-1 may also exist in the coordinate mapping map. Similarly, a blank area 131-1 may also exist in the prediction frame to be completed 131, and the source identifier corresponding to each pixel in the blank area 131-1 is 0.5. Optionally, each pixel in the blank area 131-1 may correspond to an actual coordinate, which is the actual coordinate of the pixel in the coordinate system of the prediction frame to be completed 131.

[0132] Fig.14 For an exemplary diagram of a prediction frame to be completed, please refer to Fig.14 , in the prediction frame 140 to be completed, there are pixels 140-2, 140-1 and 140-3. Among them, the pixels 140-2 and 140-3 can obtain the corresponding occlusion relationship based on the current frame or the previous frame and the depth offset map, and then render. The pixel 140-1 is blank, and the specific description can be referred to Figure 1 , I will not go into details here.

[0133] S405 , based on the depth offset map, the coordinate mapping map and the real frame, update the prediction frame to be completed to obtain the prediction frame.

[0134] Exemplarily, the cloud server may complete the blank area in the prediction frame to be completed based on the depth offset map, the coordinate mapping map and the real frame to obtain the prediction frame.

[0135] Optionally, the cloud server can also generate an initial prediction frame, and the cloud server goes through the pixels in the prediction frame to be completed one by one. If the pixel is not gapped (i.e., not blank), the pixel is written into the corresponding position in the initial prediction frame. If the pixel is gapped (i.e., blank), the non-gap points around the pixel are searched in the prediction frame to be completed, and based on the depth value corresponding to the non-gap point, a suitable pixel is selected to write the color of the pixel in the real frame into the initial prediction frame.

[0136] Fig.15For an exemplary diagram of the acquisition process of the prediction frame, please refer to Fig.15 , including but not limited to the following steps:

[0137] S1501, determine whether point P is a gap.

[0138] Exemplarily, the cloud server goes through each pixel point in the prediction frame to be completed one by one, and determines whether point P is a gap (i.e., blank). In one example, if the source corresponding to point P is 0.5 (it can also be determined based on the color of point P, which is not limited in this application), point P can be determined to be a gap point, and S1502 is executed. In another example, if the source corresponding to point P is 1 (for example, the current frame) or 0 (for example, the previous frame), point P can be determined to be a non-gap point, and the color of point P can be written to the corresponding position in the prediction frame. Correspondingly, the coordinates of point P in the prediction frame are the actual coordinates of point P in the prediction frame to be completed, and the color of point P is the color of point P in the prediction frame to be completed.

[0139] S1502, cyclically search for surrounding pixels.

[0140] For example, Fig.16 For an exemplary pixel search diagram, please refer to Fig.16 , the prediction frame to be completed includes Dp1, Dp2, Dp3 and point P. The cloud server traverses to point P and detects that the source of point P is 0.5, that is, it is located in the blank area 131-1, and determines that point P is a gap point.

[0141] The cloud server traverses the pixels around point P with a step size of 1. The step size of 1 is used to indicate that the interval between the surrounding pixels and point P is 1 pixel. Fig.16 As shown, the cloud server traverses the pixel points around point P, including P1 to P8.

[0142] S1503, determine whether P1 is a gap.

[0143] Exemplarily, the cloud server traverses the pixels around point P one by one and determines whether the pixel is a gap point. Fig.16 , the cloud server traverses to point P1, detects that the source of point P1 is 1 (or 0), determines that point P1 is not a gap point, and executes S1504 on point P1.

[0144] Optionally, if points P1 to P8 are all gap points, the cloud server expands the search range and sets the step size to 2 to traverse more pixel points. Optionally, if no non-gap points are still detected, the step size can be increased, for example, the step size can be set to 4 (or 8) to traverse more pixel points.

[0145] In one example, if P1 is a non-gap point, the depth value t1 corresponding to P1 can be obtained from the depth offset map. In another example, if P1 is a gap point, the next pixel point is traversed until a non-gap point pixel point is found.

[0146] S1504, based on the coordinate mapping diagram, obtain the source of P1.

[0147] S1505, determine whether P1 originates from the current frame.

[0148] Exemplarily, as described above, the source identifier corresponding to each point is recorded in the coordinate mapping diagram, for example, the identifier 1 is used to indicate that it comes from the current frame, and the identifier 0 is used to indicate that it comes from the previous frame. The cloud server can determine whether P1 comes from the current frame based on the source identifier of P1. In one example, if P1 comes from the current frame (for example, the source identifier is 1), S1506-1 is executed. In another example, if P1 comes from the previous frame (for example, the source identifier is 0), S1506-2 is executed.

[0149] S1506-1, obtain the depths d1 and t1 of point P1 and point P2 in S1 respectively.

[0150] For example, Fig.17 As shown, the cloud server queries the written coordinates of the point P1 with actual coordinates (x1, y1) in the coordinate mapping map based on the actual coordinates (x1, y1) of the point P1 in the prediction frame to be completed. For example, the written coordinates of the point P1 are (x2, y2). In other words, the point P1 in the prediction frame to be completed is obtained by moving the pixel with actual coordinates (x2, y2) in the current frame, and the point P1 in the coordinate mapping map is obtained by moving the point with actual coordinates (x2, y2) in the depth map of the current frame.

[0151] The cloud server can search for the corresponding point in the current depth map based on the written coordinates of point P1, that is, the point with actual coordinates (x2, y2), such as point X.

[0152] Based on the relative position between point P1 and point P in the prediction frame to be completed (e.g., the vector from point P1 to point P), the cloud server searches for a point in the depth map of the current frame whose relative position to point X is equal to the relative position between point P1 and point P, such as point W. The actual coordinates of point W in the depth map of the current frame are (x3, y3).

[0153] The cloud server obtains the depth d1 of point X in the current depth map, and the depth t1 of point W in the current depth map. The depth d1 of point X is the depth of point P1 in S1, and point W can be understood as Fig.15 For the point P2 described in , the depth t1 of W is the depth of the point P2 in S1.

[0154] S1507-1, compare d1 and t1.

[0155] S1508-1, write the color of the point with a smaller depth value into point P.

[0156] Exemplarily, the cloud server compares d1 and t1. In one example, if d1 is greater than t1, the color of the point corresponding to t1 (e.g., point W) is written to point P. In another example, if d1 is less than t1, the color of the point corresponding to d1 (e.g., point X) is written to point P.

[0157] For an example, please refer to Fig.18 , assuming that d1 is less than t1, the cloud server can determine to write the color of the pixel corresponding to point W into point P in the prediction frame to be completed. Exemplarily, the cloud server can be based on the actual coordinates (x3, y3) of W in the depth map of the current frame, the actual coordinates of pixel W in the current frame are (x3, y3). The cloud server obtains the color value (i.e., color) of pixel W, and the cloud server writes the color value of pixel W into point P in the prediction frame to be completed.

[0158] Still refer to Fig.14 ,Exemplary, cloud server based on Fig.15 According to the process in , the color of the pixel S140-3 can be written into the pixel S140-1, that is, the color of the pixel whose depth is closer to the blank pixel is written into the blank pixel to avoid the problem of image distortion.

[0159] S1506-2, obtain the depths d1 and t1 of point P1 and point P2 in S1 respectively.

[0160] S1507-2, compare d1 and t1.

[0161] S1508-2, write the color of the point with a smaller depth value into point P.

[0162] S1506-2~S1508-2 can refer to S1501-1~S1508-1, and will not be repeated here.

[0163] S1509, obtain the predicted frame.

[0164] Exemplarily, the cloud server executes S1501 to S1508 - 1 (or S1508 - 2 ) on all pixel points in the prediction frame to be completed to fill all gap points in the prediction frame to be completed, thereby obtaining a complete prediction frame.

[0165] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0166] In one example, Fig.19 A schematic block diagram of a device 1900 according to an embodiment of the present application is shown. The device 1900 may include: a processor 1901 and a transceiver / transceiver pin 1902 , and optionally, a memory 1903 .

[0167] The various components of the device 1900 are coupled together via a bus 1904, wherein the bus 1904 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 1904 in the figure.

[0168] Optionally, the memory 1903 may be used for the instructions in the aforementioned method embodiment. The processor 1901 may be used to execute the instructions in the memory 1903, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0169] The apparatus 1900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0170] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0171] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0172] This embodiment also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0173] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0174] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0175] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame; Moving all pixels in the first depth map to obtain a first depth offset map, where the first depth offset map is used to indicate an occlusion relationship between all pixels in the first depth map; Moving all pixels in the second depth map to obtain a second depth offset map, where the second depth offset map is used to indicate an occlusion relationship between all pixels in the second depth map; Obtaining a target depth offset map based on the first depth offset map and the second depth offset map; Based on the target depth offset map, all pixels in the first image frame or the second image frame are moved to obtain a target prediction frame.

2. The method according to claim 1, characterized in that The moving all pixels in the first depth map to obtain a first depth offset map includes: Determine, based on a first motion vector MV map of the first image frame, a first position of a first pixel in the first depth map in the first depth offset map; If a first depth value of the first pixel in the first depth map is greater than a depth value of the first position, writing the first depth value into the first position; Determine, based on the first MV map, a second position of a second pixel in the first depth map; If the first position is the same as the second position, and a second depth value of the second pixel in the first depth map is greater than the first depth value, the second depth value is written into the first position.

3. The method according to claim 2, characterized in that The moving all pixels in the second depth map to obtain a second depth offset map includes: Determine, based on a second MV map of the second image frame, a third position of a third pixel in the second depth map in the second depth offset map; If a third depth value of the third pixel in the second depth map is greater than a depth value of the third position, writing the third depth value into the third position; Determine, based on the second MV map, a fourth position of a fourth pixel in the second depth map in the second depth map; If the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, the fourth depth value is written into the third position.

4. The method according to claim 3, characterized in that The step of obtaining a target depth offset map based on the first depth offset map and the second depth offset map includes: The depth values ​​corresponding to all positions in the first offset map are written into the target depth offset map, wherein, for positions in the target depth offset map where no depth values ​​are written, depth values ​​are obtained from corresponding positions in the second depth offset map and written.

5. The method according to claim 1, characterized in that: The step of moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: Based on the target depth offset map, the first image frame and the second image frame, a coordinate mapping map is obtained; wherein the coordinate mapping map is used to indicate pixel points corresponding to all positions in the target depth offset map in the first image frame or the second image frame.

6. The method according to claim 5, characterized in that The step of moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: Based on the coordinate mapping diagram, the first image frame and the second image frame, a prediction frame to be completed is obtained.

7. The method according to claim 6, characterized in that The step of moving all pixels in the first image frame or the second image frame based on the target depth offset map to obtain a target prediction frame includes: Finding gap points in the prediction frame to be completed; Determine a fifth pixel point from a plurality of pixel points around the gap point, wherein the fifth pixel point is a non-gap point; Based on the coordinate mapping diagram, searching for a sixth pixel corresponding to the fifth pixel in the first image frame or the second image frame; Determine a seventh pixel in the first image frame or the second image frame based on a first relative position between the gap point and the fifth pixel, wherein a second relative position between the seventh pixel and the sixth pixel is the same as the first relative position; Based on the first depth map or the second depth map, obtaining a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel; The color of a pixel corresponding to a smaller depth value between the seventh depth value and the sixth depth value is written into the gap point.

8. An electronic device, characterized in that: include: One or more processors, memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: Acquire a first image frame and a first depth map of the first image frame, and a second image frame and a second depth map of the second image frame; Moving all pixels in the first depth map to obtain a first depth offset map, where the first depth offset map is used to indicate an occlusion relationship between all pixels in the first depth map; Moving all pixels in the second depth map to obtain a second depth offset map, where the second depth offset map is used to indicate an occlusion relationship between all pixels in the second depth map; Obtaining a target depth offset map based on the first depth offset map and the second depth offset map; Based on the target depth offset map, all pixels in the first image frame or the second image frame are moved to obtain a target prediction frame.

9. The electronic device according to claim 8, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Determine, based on a first motion vector MV map of the first image frame, a first position of a first pixel in the first depth map in the first depth offset map; If a first depth value of the first pixel in the first depth map is greater than a depth value of the first position, writing the first depth value into the first position; Determine, based on the first MV map, a second position of a second pixel in the first depth map; If the first position is the same as the second position, and a second depth value of the second pixel in the first depth map is greater than the first depth value, the second depth value is written into the first position.

10. The electronic device according to claim 9, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Determine, based on a second MV map of the second image frame, a third position of a third pixel in the second depth map in the second depth offset map; If a third depth value of the third pixel in the second depth map is greater than a depth value of the third position, writing the third depth value into the third position; Determine, based on the second MV map, a fourth position of a fourth pixel in the second depth map in the second depth map; If the third position is the same as the fourth position, and the fourth depth value of the fourth pixel in the second depth map is greater than the third depth value, the fourth depth value is written into the third position.

11. The electronic device according to claim 10, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: The depth values ​​corresponding to all positions in the first offset map are written into the target depth offset map, wherein, for positions in the target depth offset map where no depth values ​​are written, depth values ​​are obtained from corresponding positions in the second depth offset map and written.

12. The electronic device according to claim 8, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the target depth offset map, the first image frame and the second image frame, a coordinate mapping map is obtained; wherein the coordinate mapping map is used to indicate pixel points corresponding to all positions in the target depth offset map in the first image frame or the second image frame.

13. The electronic device according to claim 12, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Based on the coordinate mapping diagram, the first image frame and the second image frame, a prediction frame to be completed is obtained.

14. The electronic device according to claim 13, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Finding gap points in the prediction frame to be completed; Determine a fifth pixel point from a plurality of pixel points around the gap point, wherein the fifth pixel point is a non-gap point; Based on the coordinate mapping diagram, searching for a sixth pixel corresponding to the fifth pixel in the first image frame or the second image frame; Determine a seventh pixel in the first image frame or the second image frame based on a first relative position between the gap point and the fifth pixel, wherein a second relative position between the seventh pixel and the sixth pixel is the same as the first relative position; Based on the first depth map or the second depth map, obtaining a seventh depth value of the seventh pixel and a sixth depth value of the sixth pixel; The color of a pixel corresponding to a smaller depth value between the seventh depth value and the sixth depth value is written into the gap point.

15. A computer storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

16. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.

17. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.

Images