Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of image processing. The image processing method comprises the steps that a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame and a depth image of the first image frame are acquired, the second image frame is a previous frame image of the first image frame, and pixel points of the motion vector image, the binary image and the depth image are in one-to-one correspondence; processing the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image; and generating a frame insertion image between the first image frame and the second image frame according to the updated motion vector image, the updated depth image, the first image frame and the second image frame.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] Frame interpolation is a technology that generates several frames based on the current frame image and the previous frame image, and inserts them between the current frame image and the previous frame image. Frame interpolation technology can increase the frame rate of the picture, make the picture smoother, and also share the rendering pressure of the graphics processing unit (GPU) of the electronic device and reduce power consumption.

[0003] The relevant technology is mainly based on the interpolation algorithm AMD FidelityFX TM Super Resolution 3 is used to determine the interpolated image, and the quality of the interpolated image is poor. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium, which can solve the technical problem of poor interpolated image quality existing in the related art.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is an image frame previous to the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one;

[0007] Processing the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image;

[0008] An interpolation image between the first image frame and the second image frame is generated according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0010] An acquisition module is used to acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is an image frame previous to the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one;

[0011] A processing module, used for processing the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image;

[0012] A generating module is used to generate an interpolation frame between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0017] The embodiment of the present application obtains a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is an image of the previous frame of the first image frame; processes the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image; generates an interpolated image between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame. That is, the embodiment of the present application updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels in combination with the binary image, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a motion vector image provided for related technology;

[0019] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0020] Figure 3A schematic diagram of a first image frame provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a motion vector image provided by an embodiment of the present application;

[0022] Figure 5 A schematic diagram of a binary image provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a depth image provided in an embodiment of the present application;

[0024] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application;

[0025] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application;

[0026] Fig. 9 A flowchart of another image processing method provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of an original motion vector image provided in an embodiment of the present application;

[0028] Fig.11 A schematic diagram for comparing processed image effects provided in an embodiment of the present application;

[0029] Fig.12 A schematic diagram for comparing another image effect after processing provided in an embodiment of the present application;

[0030] Fig.13 A schematic diagram for comparing another image effect after processing provided in an embodiment of the present application;

[0031] Fig.14 A schematic diagram for comparing another image effect after processing provided in an embodiment of the present application;

[0032] Fig.15 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0033] Fig.16 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Fig.17 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] As mentioned above, interpolation technology can improve the frame rate of the picture, make the picture smoother, and share the rendering pressure of the GPU and reduce power consumption. For example, for 3D games, due to the limitation of the GPU computing power, large-scale open 3D games can usually only run at a refresh rate of 60HZ or even lower, which seriously affects the user's gaming experience. Through game interpolation, the game frame rate can be improved. And because the game interpolation process is carried out in a two-dimensional image space, there is no need to consider various complex calculations in a three-dimensional scene, which can effectively reduce the GPU's computing power and power consumption.

[0038] In the game interpolation algorithm, the quality of the motion vector graph plays a crucial role in the quality of the final interpolated frame. Figure 1 It can be found from the motion vector diagram in the rectangular frame 100 that the motion vector at the edge of the character is not similar to the motion vectors of the foreground (inside the character) and the background, and is inaccurate. The motion vector there should belong to the foreground. Therefore, an algorithm is needed to correct the motion vector at the edge of the character and expand the motion vector of the foreground outward to achieve the purpose of correcting the edge motion vector.

[0039] In order to achieve the above purpose, the most commonly used algorithm is AMD FidelityFX TM The Super Resolution 3 (FSR3) algorithm is based on the principle of minimum depth value. This processing method will cause the following problems:

[0040] During the motion vector expansion process, the motion vector at the position with a smaller depth value may be wrong. Direct expansion will cause the area of ​​the wrong motion vector to become larger, and eventually cause the interpolation image to be broken, affecting the interpolation effect.

[0041] To this end, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which can solve the technical problem of poor quality of interpolated images existing in related technologies and reduce the fragmentation of interpolated images.

[0042] The image processing method, device, electronic device and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0043] The image processing method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablets, and laptop computers. The embodiment of the present application can be applied to scenes such as games and videos.

[0044] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the image processing method may include the following steps:

[0045] S210 , acquiring a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame.

[0046] The second image frame is an image frame previous to the first image frame, and the pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one.

[0047] S220 . Process the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image.

[0048] S230 : Generate an interpolated image between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0049] The embodiment of the present application updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels in combination with the binary image, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0050] The above steps are described in detail below:

[0051] In S210, the first image frame may be an image frame in a video, and the video may be a video obtained based on film and television production, a video shot by a camera, an offline or online game, etc.

[0052] The second image frame and the first image frame may be adjacent image frames. Specifically, the second image frame may be the previous frame of the first image frame or the next frame of the first image frame. The embodiment of the present application takes the second image frame as the previous frame of the first image frame as an example.

[0053] The motion vector image is used to characterize the motion vector of each pixel in the first image frame relative to the second image frame. The depth image is used to characterize the depth value of each pixel in the first image frame. The binary image is used to characterize which pixels belong to the target object and which pixels belong to the non-target object. The target object may be an object of interest to the user, for example, for a game frame, the target object may be a character. Exemplarily, the binary image may use 0 to represent a non-target object and 1 to represent a target object.

[0054] The first image frame, the motion vector image, the binary image, and the depth image have the same size, and the pixels correspond one to one. Figure 3 As an example, the motion vector image, the binary image and the depth image can be respectively referred to in Figure 4-Figure 6 , the pixels corresponding to the same position in different images are the same, which can simplify the subsequent image processing process and there is no need to determine the matching relationship between the pixels in each image.

[0055] In S220, the embodiment of the present application can update the motion vector image and the depth image using a binary image from a pixel level perspective to obtain an updated motion vector image and an updated depth image, thereby providing a more accurate basis for the subsequent generation of an interpolated image between the first image frame and the second image frame.

[0056] The updated motion vector image may have the same number of pixels but the motion vectors of some or all pixels have changed. Alternatively, the number of pixels may be reduced but the motion vectors of some or all pixels have changed. The reduction in the number of pixels will appropriately reduce the resolution of the image, thereby reducing the power consumption of image processing.

[0057] In some embodiments, the electronic device may select an area from the binary image, and update the motion vectors of corresponding pixels in the motion vector image and the depth values ​​of corresponding pixels in the depth image according to the motion vectors and depth values ​​of the pixels in the area.

[0058] The region may be a neighborhood centered on a certain pixel point in the binary image, or may be a certain image block obtained by dividing the binary image into blocks.

[0059] That is, the electronic device may update the motion vector image and the depth image according to the motion vectors and the depth values ​​of the pixels in the neighborhood, and may also update the motion vector image and the depth image according to the motion vectors and the depth values ​​of the pixels in the image block.

[0060] Through the above processing, most of the erroneous motion vectors can be corrected. In order to further improve the image processing effect, illustratively, after S220, the image processing method may further include the following steps:

[0061] The updated motion vector image and the updated depth image are filtered to obtain a filtered motion vector image and a filtered depth image.

[0062] The filtering here can be median filtering or other forms of filtering. By filtering the updated motion vector image and the updated depth image, the motion vector can be made smoother, and the motion vector holes and fragmentation in the target object can be reduced, thereby further improving the quality of the interpolated image.

[0063] In S230, one or more interpolated frames between the first image frame and the second image frame may be generated based on the updated motion vector image and the depth image, as well as the first image frame and the second image frame. The specific generation process is not limited in the embodiment of the present application. The interpolated frame image is inserted between the first image frame and the second image frame to increase the display frame rate of the image and improve the user experience.

[0064] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application, Figure 7 and Figure 2 The difference is that Figure 2 The S220 in can be refined into Figure 7 S710-S730 in.

[0065] S710: Determine a target area from the binary image.

[0066] The target area is the area where some or all of the pixels in the binary image are located. For example, the target area can be an area with any pixel in the binary image as the center point. That is, the electronic device can select a certain pixel from the binary image as the center point, and record the area containing the center point as the target area. For example, the neighborhood with a radius of r and a pixel point A in the binary image as the center can be determined as the target area, and the radius r is less than or equal to the shortest distance from the feature point to the edge of the binary image. For example, the target area can also be a rectangle or other shapes.

[0067] For example, a pixel point may be selected from the pixel points with a pixel value of 1 as pixel point A, and a pixel point may be selected from the pixel points with a pixel value of 0 as pixel point A. Considering that the target object may have multiple edges, in order to improve the image processing effect, for example, each pixel point may be determined as a center point, that is, a target area may be obtained for each pixel point, and a similar process may be performed for each target area later.

[0068] Exemplarily, the target area may also be an image block in a binary image, and the image block is obtained by performing block processing on the binary image. When there are multiple image blocks, the number of pixels contained in different image blocks may be the same or different. In some embodiments, the target area may include N*N pixels, where N is an integer greater than or equal to 1. The N*N pixels may be equivalent to one pixel, which can reduce the number of pixels, reduce the resolution of the image, and thus reduce the processing power consumption of the image.

[0069] The embodiments of the present application can process the motion vector image and the depth image from the perspective of the pixel level or the block level, thereby improving the flexibility of the method while improving the image quality.

[0070] S720: Determine a first target pixel in the target area according to the number of first pixels in the target area.

[0071] The first pixel is a pixel corresponding to the target object in the first image frame. For example, when 1 represents the target object and 0 represents the non-target object, the first pixel is a pixel with a pixel value of 1. The number of first pixels is the number of pixel values ​​1. The target area may include 0, 1 or more first pixels.

[0072] The first target pixel is a pixel in the target area, and the number of the first target pixel can be one or more. The position and number of the first target pixel are related to the number of the first pixel in the target area and the method of selecting the target area, that is, the embodiment of the present application can dynamically determine the first target pixel according to the method of selecting the target area and the number of the first pixel, so as to improve the processing effect of the image. The specific determination process of the first target pixel can be referred to the following embodiment.

[0073] S730 . Update pixel information of a second target pixel in the motion vector image and the depth image according to pixel information of the first target pixel, to obtain an updated motion vector image and an updated depth image.

[0074] Among them, the second target pixel is the pixel corresponding to the first target pixel in the motion vector image and the depth image. The second target pixel and the first target pixel can be in a one-to-one correspondence relationship or a pair-P relationship. For example, when the target area is an area with any pixel in the binary image as the center point, the second target pixel and the first target pixel can be in a one-to-one correspondence relationship. When the target area is an image block in a binary image, the second target pixel and the first target pixel can be in a pair-P relationship, where P represents the number of first target pixels and P is an integer greater than or equal to 1.

[0075] The pixel information may be information representing features of the pixel. For example, in a motion vector image, the pixel information is a motion vector, and in a depth image, the pixel information is a depth value.

[0076] Exemplarily, the electronic device may update the motion vector of the second target pixel corresponding to the first target pixel in the motion vector image according to the motion vector of the first target pixel, and update the depth value of the second target pixel corresponding to the first target pixel in the depth image according to the depth value of the first target pixel.

[0077] The updating process of the motion vector is similar to the updating process of the depth value. Taking the updating of the motion vector of the second target pixel in the motion vector according to the motion vector of the first target pixel as an example, illustratively, the motion vector of the second target pixel in the motion vector image can be directly replaced by the motion vector of the first target pixel, or the motion vector of the first target pixel can be processed to obtain the processed motion vector, and the motion vector of the second target pixel in the motion vector image can be replaced by the processed motion vector.

[0078] The embodiment of the present application flexibly selects a target area from a binary image according to different selection strategies, and further determines a first target pixel from the target area according to the number of pixels corresponding to the target object in the target area, and updates the pixel information of a second target pixel corresponding to the first target pixel in the motion vector image and the depth image according to the pixel information of the first target pixel, thereby reducing the fragmentation of the interpolated image.

[0079] S720: Perform block processing on the binary image to obtain at least one image block, and determine the image block as a target area.

[0080] Each image block includes at least one pixel. When an image block includes one pixel, the image block can be represented as 1*1, and when an image block includes multiple pixels, the image block can be represented as M*N, indicating that the image block includes M*N pixels, and M and N can be the same or different. In some embodiments, M=N=2, or M=N=4, or M=N=8, or M=N=16, etc.

[0081] The embodiments of the present application can process the motion vector image and the depth image from the perspective of the pixel level or the block level, thereby improving the flexibility of the method while improving the image quality.

[0082] The following is an explanation of the process of determining the first target pixel point, which is specifically as follows:

[0083] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application, Figure 8 and Figure 7 The difference is that Figure 7 The S720 in can be refined into Figure 8 S810-S820 in.

[0084] S810: When the target area is an area with any pixel in the binary image as a center point, determine a first target pixel in the target area according to the number of first pixels.

[0085] Exemplarily, when the target area is an area with any pixel in the binary image as the center point, the first target pixel point can be determined according to the number of the first pixel points.

[0086] For example, when the number of first pixel points is greater than a third quantity threshold, the pixel points in the first pixel points whose depth values ​​are less than the first depth threshold can be determined as first target pixel points; when the number of first pixel points is not greater than the third quantity threshold, the pixel points in the target area whose depth values ​​are less than the second depth threshold can be determined as first target pixel points.

[0087] The value of the third quantity threshold can be set according to actual conditions. For example, the third quantity threshold can be set to a value between 0 and 1 (including 0 but excluding 1).

[0088] The first depth threshold can be set according to actual conditions, as long as the selected depth value can be guaranteed to be the minimum.

[0089] Assume that the number of first pixels is N, and N is an integer greater than or equal to 0. Exemplarily, when N>0, a pixel with the smallest depth value can be determined from the first pixels, and the pixel is determined as the first target pixel. Exemplarily, when N=0, a pixel with the smallest depth value can be determined from the target area, and the pixel is determined as the first target pixel.

[0090] The embodiment of the present application can determine the target area with each pixel point as the center point, and flexibly determine the first target pixel point from the first pixel point or the target area according to the number of first pixel points in the target area, thereby improving the flexibility of the method while improving the image processing effect.

[0091] S820: When the target region is an image block of a binary image, determine a first target pixel in the target region according to the number of pixels included in the target region and the number of first pixels in the target region.

[0092] Exemplarily, when the target area is an image block of a binary image, the block type of the target area can be determined according to the number of pixels contained in the target area, and the first target pixel can be adaptively determined according to the block type and the number of first pixels in the target area, thereby improving the accuracy of the first target pixel, and further improving the accuracy of the interpolated image.

[0093] The block type may include a first block type or a second block type, the first block type is used to characterize that the number of pixels contained in the target area is less than a first number threshold, and the second block type is used to characterize that the number of pixels contained in the target area is not less than the first number threshold. That is, according to the number of pixels contained in the target area, the target area can be divided into small blocks or large blocks, the block type of the small blocks is the first block type, and the block type of the large blocks is the second block type.

[0094] The size of the first quantity threshold can be set according to actual conditions. In some embodiments, the first quantity threshold can be set to 17. For example, a target area containing 2*2 pixels or 4*4 pixels can be determined as a small block, and a target area containing 8*8 pixels or 16*16 pixels can be determined as a large block.

[0095] The embodiment of the present application can determine the target area from two perspectives: pixel level and block level, and then adaptively determine the first target pixel point based on different angles and the number of first pixel points contained in the target area, thereby improving the flexibility of the method.

[0096] In some embodiments, when the block type of the target area is the first block type, that is, the target area is a small block, the first target pixel point may be further determined according to the number of first pixel points included in the target area.

[0097] Exemplarily, the above S820 may include the following steps:

[0098] When the number of first pixels in the target area is greater than a fourth number threshold, determining a pixel whose depth value is less than a third depth threshold among the first pixels as a first target pixel;

[0099] When the number of first pixel points in the target area is not greater than the fourth number threshold, a pixel point in the target area having a depth value less than the fourth depth threshold is determined as a first target pixel point.

[0100] Exemplarily, the fourth quantity threshold may be set to a value between 0 and 1 (including 0 but excluding 1), and the third depth threshold is used to minimize the depth value of the first target pixel in the first pixel or the target area.

[0101] Exemplarily, when the target area is a small block, if the number of first pixels contained in the target area N>0, the pixel with the smallest depth value among the first pixels can be determined as the first target pixel; if N=0, the pixel with the smallest depth value in the target area can be determined as the first target pixel.

[0102] When the block type of the target area is the second block type, that is, the target area is a large block, the first target pixel point may be further determined according to the number of first pixel points included in the target area.

[0103] Exemplarily, the above S820 may include the following steps:

[0104] When the number of first pixel points in the target area is not less than a second number threshold, all first pixel points in the target area are determined as first target pixel points;

[0105] When the number of the first pixel points in the target area is less than the second number threshold, a first target pixel point is determined in the target area according to the depth values ​​of the pixels in the target area.

[0106] The target area contains M*M pixels, and the second quantity threshold is For example, when the target area is a large block, if the number of first pixels included in the target area is All first pixel points in the target area may be determined as first target pixel points. The first target pixel point may be further determined according to the depth value of each pixel point in the target area.

[0107] In the embodiment of the present application, when the target area is a large block, the first target pixel point can be adaptively determined according to the number of first pixel points included in the target area, thereby improving the flexibility of the method.

[0108] In some embodiments, the above step of “determining a first target pixel point within the target area according to the depth values ​​of each pixel point within the target area” can be further refined into the following steps:

[0109] Determine the difference between the maximum depth value and the minimum depth value of each pixel in the target area;

[0110] Determine foreground pixels, midground pixels, and background pixels from the target area according to the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area;

[0111] The pixel with the largest number among the foreground pixel points, the mid-ground pixel points and the background pixel points is determined as the first target pixel point.

[0112] For example, d range =d max -d min , d max is the maximum depth value of each pixel in the target area, d min is the minimum depth value of each pixel in the target area, d range is the difference between the maximum depth value and the minimum depth value.

[0113] For the i-th pixel in the target area, the following process can be performed, where 1≤i≤P, and P is the number of pixels contained in the target area:

[0114] if It can be determined that the i-th pixel belongs to the foreground pixel;

[0115] if It can be determined that the i-th pixel belongs to the mid-ground pixel;

[0116] if It can be determined that the i-th pixel belongs to the background pixel.

[0117] The above determination process is only one example, and in actual application, there may be other determination methods. Thus, the number of foreground pixels, mid-ground pixels and background pixels in the target area may be obtained.

[0118] In the embodiment of the present application, for example, the pixel with the largest number can be determined as the first target pixel and participate in the subsequent pixel information update operation. For example, the number of foreground pixels in the target area is the largest, so all foreground pixels can be determined as the first target pixel.

[0119] In the embodiment of the present application, when the target area is a large block and the number of first pixels contained in the target area is small, the number of pixels belonging to the foreground, middle ground and background can be further determined according to the depth value of each pixel in the target area, and the first target pixel can be determined from the pixels of the foreground, middle ground and background based on the number. In this way, the accuracy of the first target pixel can be improved, and the processing effect of subsequent images can be improved.

[0120] The following is an explanation of the process of determining the second target pixel point, which is specifically as follows:

[0121] Different methods for acquiring the target area may lead to different methods for determining the second target pixel. Specifically, in some embodiments, before S730, the image processing method may further include the following steps:

[0122] In the case where the target area is an area with any pixel point in the binary image as the center point, a pixel point corresponding to the center pixel point of the target area in the motion vector image and the depth image is determined as a second target pixel point;

[0123] When the target area is an image block of a binary image, an area corresponding to the target area is determined from the motion vector image and the depth image, all pixels in the area are equivalent to one pixel, and the equivalent pixel is determined as a second target pixel.

[0124] Exemplarily, when the target area is determined from a pixel level perspective, that is, the area with any pixel point in the binary image as the center point is determined as the target area, the pixel point in the motion vector image and the depth image corresponding to the center pixel point of the target area can be determined as the second target pixel point.

[0125] Exemplarily, when the target area is determined from a block level perspective, that is, when the target area is divided into image blocks in a binary image, multiple pixel points in the area corresponding to the target area in the motion vector image and the depth image can be regarded as one pixel point. In other words, the number of pixel points in the area corresponding to the target area is changed from multiple to one, which can reduce the subsequent calculation amount and reduce power consumption.

[0126] The embodiment of the present application can adaptively determine the second target pixel point that needs to be updated in the motion vector image and the depth image according to the method of obtaining the target area, thereby improving the flexibility of the method, especially when determining the target area from a block level perspective, which can reduce the subsequent calculation amount and reduce processing power consumption.

[0127] Fig. 9 A flowchart of another image processing method provided in an embodiment of the present application, Fig. 9 and Figure 7 The difference is that Figure 7The S730 in can be refined into Fig. 9 S910-S930 in the series.

[0128] S910. When the target area is an image block of a binary image and the block type of the target area is a first block type, or when the target area is an area with any pixel point in the binary image as the center point, the pixel information of the second target pixel point in the motion vector image and the depth image is replaced with the pixel information of the first target pixel point to obtain an updated motion vector image and an updated depth image.

[0129] Exemplarily, when the target area is an area with any pixel in the binary image as the center point, the pixel information of the second target pixel in the motion vector image and the depth image can be directly replaced with the pixel information of the first target pixel.

[0130] Taking the example that pixel information includes motion vectors, illustratively, when the target area is an area centered on any pixel in the binary image, the motion vector of the second target pixel in the motion vector image can be directly replaced by the motion vector of the first target pixel.

[0131] For example, when the number of first pixels included in the target area is greater than 0, the motion vector of the pixel with the smallest depth value among the first pixels can be assigned to the second target pixel in the motion vector image, that is, to the pixel corresponding to the central pixel of the binary image in the motion vector image. When the number of first pixels included in the target area is equal to 0, the motion vector of the pixel with the smallest depth value in the target area can be assigned to the second target pixel in the motion vector image, that is, to the pixel corresponding to the central pixel of the binary image in the motion vector image.

[0132] Exemplarily, when the target area is a small block, the motion vector of the second target pixel in the motion vector image may be directly replaced by the motion vector of the first target pixel.

[0133] For example, when the number of first pixels included in the target area is greater than 0, the motion vector of the pixel with the smallest depth value among the first pixels can be assigned to the block corresponding to the target area in the motion vector image, and the block is equivalent to one pixel. When the number of first pixels included in the target area is equal to 0, the motion vector of the pixel with the smallest depth value in the target area can be assigned to the block corresponding to the target area in the motion vector image, and the block is equivalent to one pixel.

[0134] S920. When the target area is an image block of a binary image and the block type of the target area is a second block type, determine the weight of the first target pixel participating in the pixel information update operation according to the number of first pixels in the target area and the depth value of each pixel in the target area.

[0135] Here, the weight of the first target pixel participating in the pixel information update operation is used to characterize the probability of the first target pixel being selected. The larger the weight, the greater the probability of the first target pixel being selected. When the target area is a large block, the weight of each first target pixel participating in the pixel information update operation can be dynamically determined based on the number of first pixels contained in the target area and the depth value of each pixel in the target area.

[0136] The number of first pixel points included in the target area is different, and the corresponding weight determination method is different. The specific weight determination process is described below in conjunction with the number of first pixel points.

[0137] In some embodiments, when the number of first pixels in the target area is not less than a second number threshold, the weight of the first target pixel may be determined according to the depth value of the first pixel and the depth value of the first target pixel.

[0138] Exemplarily, the minimum depth value, maximum depth value, first difference between the depth value of the first target pixel and the minimum depth value, and second difference between the maximum depth value and the minimum depth value of the first pixel can be determined; and the weight of the first target pixel can be determined based on the ratio of the first difference to the second difference.

[0139] The second quantity threshold is For example, when When the minimum depth value and the maximum depth value can be determined from the depth value of the first pixel in the target area, they are denoted as d min1 and d max1 , combined with the depth value of the first target pixel, the weight of each first target pixel can be obtained.

[0140] For example, Among them, p i represents the weight of the i-th first target pixel, that is, the weight of the i-th first pixel in the target area. i is the depth value of the i-th first target pixel, that is, the depth value of the i-th first pixel in the target area.

[0141] In some embodiments, when the number of first pixel points in the target area is less than the second number threshold, the weight of the first target pixel point may be determined according to the depth values ​​of all pixel points in the target area and the depth value of the first target pixel point.

[0142] Exemplarily, the minimum depth value, the maximum depth value, the third difference between the depth value of the first target pixel and the minimum depth value, and the fourth difference between the maximum depth value and the minimum depth value of all pixels in the target area can be determined; and the weight of the first target pixel is determined based on the ratio of the third difference to the fourth difference.

[0143] For example, when When the minimum depth value and the maximum depth value can be determined from the depth values ​​of all pixels in the target area, they are denoted as d min2 and d max2 , thus, the fourth difference d between the minimum depth value and the maximum depth value can be obtained range =d max2 -d min2 Then, combined with the depth value of the first target pixel, the weight of each first target pixel can be obtained.

[0144] For example, Among them, p j represents the weight of the jth first target pixel, d j is the depth value of the jth first target pixel. The first target pixel here is the foreground pixel, mid-ground pixel or background pixel with the largest number in the target area. The sum of the number of foreground pixels, mid-ground pixels and background pixels is equal to the number of pixels contained in the target area. j -d min2 The third difference.

[0145] When the target area is large, the embodiment of the present application can further adaptively determine the weight of each first target pixel participating in subsequent update operations based on the number of first pixels in the target area and the depth value of each pixel in the target area, thereby providing a more accurate basis for subsequent pixel update operations.

[0146] S930: Update pixel information of a second target pixel in the motion vector image and the depth image according to the weight and pixel information of the first target pixel, and obtain an updated motion vector image and an updated depth image.

[0147] After the weights of each first target pixel point are determined, the pixel point information of the second target pixel point in the motion vector image and the depth image can be re-determined based on the weights of each first target pixel point and the pixel point information of each first target pixel point, and the re-determined pixel point information can be used to replace the original pixel point information to achieve the update of the pixel point information.

[0148] The embodiment of the present application can adaptively determine pixel information of corresponding pixels in the motion vector image and the depth image according to the determination method of the target area and the number of first pixels contained in the target area, thereby reducing image fragmentation and improving image quality.

[0149] In some embodiments, the above S930 may include the following steps:

[0150] Determine a weight of the first target pixel point and a weighted sum of corresponding pixel point information, and a weight sum of the first target pixel point;

[0151] The ratio of the weighted sum to the weighted sum is determined as pixel point information of the second target pixel point in the motion vector image and the depth image.

[0152] For example, when hour, Wherein, mv represents the motion vector of the second target pixel in the motion vector image, d represents the depth value of the second target pixel in the depth image, mv and d are block-level pixel information, mv i and d i are the motion vector and depth value of the i-th first target pixel respectively.

[0153] For example, when hour, Among them, N f Indicates the number of pixels in the foreground, middle ground or background with the largest number of pixels in the target area, mv j and d j are the motion vector and depth value of the j-th first target pixel respectively.

[0154] The embodiment of the present application determines the pixel information of the second target pixel in the motion vector image and the depth image based on the weight and pixel information of each first target pixel, taking into account the influence of each first target pixel, so that the pixel information of the corresponding pixel in the motion vector image and the depth image can be determined more accurately, reducing image fragmentation and improving image quality.

[0155] The image processing method provided in the embodiment of the present application is described below with reference to several effect diagrams.

[0156] Fig.10 is the original motion vector image, that is, the unprocessed motion vector image, Fig.11 It is a schematic diagram comparing the effects of the present application and related technologies from a pixel level perspective, wherein 110 is a motion vector image obtained after processing in the embodiment of the present application, and 111 is a motion vector image obtained after processing in the related technology.

[0157] Fig.12120 is a motion vector image obtained after processing and filtering from a pixel level perspective in the embodiment of the present application, and 111 is a motion vector image obtained after processing and filtering by related technologies.

[0158] Fig.13 It is a schematic diagram comparing the effects of the present application and the related technology from the block level perspective, wherein 130 is a motion vector image obtained after processing in the embodiment of the present application, and 131 is a motion vector image obtained after processing in the related technology. Fig.13 Take small blocks as an example.

[0159] Fig.14 140 is a comparison diagram of the effects after median filtering, wherein 140 is a motion vector image obtained after processing and filtering from a block level perspective in the embodiment of the present application, and 141 is a motion vector image obtained after processing and filtering by related technology.

[0160] The yellow ellipse in the above schematic diagram represents an area with obvious contrast. It can be seen from the above renderings that the image processing method provided in the embodiment of the present application can effectively improve the motion vector holes, fragmentation and other phenomena within the character, whether from a pixel level perspective or a block level perspective, and the image quality is better.

[0161] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a processing module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0162] Fig.15 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0163] like Fig.15 As shown, the image processing device 1500 may include:

[0164] An acquisition module 1501 is used to acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is an image frame previous to the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one;

[0165] A processing module 1502 is used to process the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image;

[0166] The generating module 1503 is configured to generate an interpolation frame between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0167] The embodiment of the present application obtains a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is an image of the previous frame of the first image frame; processes the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image; generates an interpolated image between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame. That is, the embodiment of the present application updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels in combination with the binary image, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0168] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0169] Determine a target region from a binary image, where the target region is a region with any pixel point in the binary image as a center point, or the target region is an image block in the binary image, where the image block is obtained by performing block processing on the binary image, and each image block corresponds to a pixel point;

[0170] Determine a first target pixel in the target area according to the number of first pixels in the target area, where the first pixel is a pixel corresponding to the target object in the first image frame;

[0171] According to the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain an updated motion vector image and an updated depth image, wherein the second target pixel is a pixel corresponding to the first target pixel.

[0172] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0173] In the case where the target area is an area with any pixel point in the binary image as a center point, determining a first target pixel point from the target area according to the number of first pixels;

[0174] In the case where the target region is an image block of a binary image, the first target pixel point is determined from the target region according to the number of pixel points included in the target region and the number of first pixel points in the target region.

[0175] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0176] Determine a block type of the target area according to the number of pixels included in the target area, where the block type includes a first block type or a second block type, where the first block type is used to characterize that the number of pixels included in the target area is less than a first number threshold, and the second block type is used to characterize that the number of pixels included in the target area is not less than the first number threshold;

[0177] According to the block type and the number of first pixels in the target area, a first target pixel is determined from the target area.

[0178] In some possible implementations of the embodiments of the present application, the block type is the second block type;

[0179] The processing module 1502 is specifically configured to:

[0180] When the number of first pixel points in the target area is not less than a second number threshold, all first pixel points in the target area are determined as first target pixel points;

[0181] When the number of the first pixel points in the target area is less than the second number threshold, a first target pixel point is determined in the target area according to the depth values ​​of the pixels in the target area.

[0182] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0183] Determine the difference between the maximum depth value and the minimum depth value of each pixel in the target area;

[0184] Determine foreground pixels, midground pixels, and background pixels from the target area according to the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area;

[0185] The pixel with the largest number among the foreground pixel points, the mid-ground pixel points and the background pixel points is determined as the first target pixel point.

[0186] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0187] When the target area is an image block of a binary image and the block type of the target area is a first block type, or when the target area is an area with any pixel in the binary image as a center point, pixel information of the second target pixel in the motion vector image and the depth image is replaced with pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image;

[0188] When the target area is an image block of a binary image and the block type of the target area is a second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target area and the depth value of each pixel in the target area, and the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight and pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image.

[0189] In some possible implementations of the embodiments of the present application, the processing module 1502 is specifically configured to:

[0190] When the number of the first pixel points in the target area is not less than the second number threshold, determining the weight of the first target pixel point according to the depth value of the first pixel point and the depth value of the first target pixel point;

[0191] When the number of the first pixel points in the target area is less than the second number threshold, the weight of the first target pixel point is determined according to the depth values ​​of all the pixels in the target area and the depth value of the first target pixel point.

[0192] The image processing device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a user computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0193] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0194] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 9The various processes in the image processing method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0195] like Fig.16 As shown, an embodiment of the present application further provides an electronic device 1600, including a processor 1601 and a memory 1602, wherein the memory 1602 stores programs or instructions that can be executed on the processor 1601, and when the program or instructions are executed by the processor 1601, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0196] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0197] Fig.17 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0198] The electronic device 1700 includes but is not limited to: a radio frequency unit 1701, a network module 1702, an audio output unit 1703, an input unit 1704, a sensor 1705, a display unit 1706, a user input unit 1707, an interface unit 1708, a memory 1709, and a processor 1710 and other components.

[0199] Those skilled in the art will appreciate that the electronic device 1700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.17 The structure of the electronic device 1700 shown in the figure does not constitute a limitation of the electronic device 1700. The electronic device 1700 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0200] The processor 1710 is used to obtain a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is a previous image frame of the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one;

[0201] Processing the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image;

[0202] An interpolation image between the first image frame and the second image frame is generated according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0203] The embodiment of the present application updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels in combination with the binary image, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0204] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0205] Determine a target region from a binary image, where the target region is a region with any pixel point in the binary image as a center point, or the target region is an image block in the binary image, where the image block is obtained by performing block processing on the binary image, and each image block corresponds to a pixel point;

[0206] Determine a first target pixel in the target area according to the number of first pixels in the target area, where the first pixel is a pixel corresponding to the target object in the first image frame;

[0207] According to the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain an updated motion vector image and an updated depth image, wherein the second target pixel is a pixel corresponding to the first target pixel.

[0208] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0209] In the case where the target area is an area with any pixel point in the binary image as a center point, determining a first target pixel point from the target area according to the number of first pixels;

[0210] In the case where the target region is an image block of a binary image, the first target pixel point is determined from the target region according to the number of pixel points included in the target region and the number of first pixel points in the target region.

[0211] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0212] Determine a block type of the target area according to the number of pixels included in the target area, where the block type includes a first block type or a second block type, where the first block type is used to characterize that the number of pixels included in the target area is less than a first number threshold, and the second block type is used to characterize that the number of pixels included in the target area is not less than the first number threshold;

[0213] According to the block type and the number of first pixels in the target area, a first target pixel is determined from the target area.

[0214] In some possible implementations of the embodiments of the present application, the block type is the second block type;

[0215] The processor 1710 is specifically configured to:

[0216] When the number of first pixel points in the target area is not less than a second number threshold, all first pixel points in the target area are determined as first target pixel points;

[0217] When the number of the first pixel points in the target area is less than the second number threshold, a first target pixel point is determined in the target area according to the depth values ​​of the pixels in the target area.

[0218] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0219] Determine the difference between the maximum depth value and the minimum depth value of each pixel in the target area;

[0220] Determine foreground pixels, midground pixels, and background pixels from the target area according to the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area;

[0221] The pixel with the largest number among the foreground pixel points, the mid-ground pixel points and the background pixel points is determined as the first target pixel point.

[0222] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0223] When the target area is an image block of a binary image and the block type of the target area is a first block type, or when the target area is an area with any pixel in the binary image as a center point, pixel information of the second target pixel in the motion vector image and the depth image is replaced with pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image;

[0224] When the target area is an image block of a binary image and the block type of the target area is a second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target area and the depth value of each pixel in the target area, and the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight and pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image.

[0225] In some possible implementations of the embodiments of the present application, the processor 1710 is specifically configured to:

[0226] When the number of the first pixel points in the target area is not less than the second number threshold, determining the weight of the first target pixel point according to the depth value of the first pixel point and the depth value of the first target pixel point;

[0227] When the number of the first pixel points in the target area is less than the second number threshold, the weight of the first target pixel point is determined according to the depth values ​​of all the pixels in the target area and the depth value of the first target pixel point.

[0228] It should be understood that in the embodiment of the present application, the input unit 1704 may include a graphics processor (Graphics Processing Unit, GPU) 17041 and a microphone 17042, and the graphics processor 17041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 17061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1707 includes a touch panel 17071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 17071 may include two parts: a touch detection device and a touch controller. Other input devices 17072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0229] The memory 1709 can be used to store software programs and various data. The memory 1709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1709 may include a volatile memory or a non-volatile memory, or the memory 1709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0230] The processor 1710 may include one or more processing units; optionally, the processor 1710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1710.

[0231] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0232] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0233] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0234] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0235] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0236] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0237] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0238] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is a previous frame image of the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond one to one; Processing the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image; An interpolated image between the first image frame and the second image frame is generated according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

2. The method according to claim 1, characterized in that The processing of the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image includes: Determine a target area from the binary image, wherein the target area is an area with any pixel point in the binary image as a center point, or the target area is an image block in the binary image, wherein the image block is obtained by performing block processing on the binary image, and each image block corresponds to a pixel point; Determining a first target pixel in the target area according to the number of first pixels in the target area, where the first pixel is a pixel corresponding to the target object in the first image frame; According to the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain an updated motion vector image and an updated depth image, wherein the second target pixel is a pixel corresponding to the first target pixel.

3. The method according to claim 2, characterized in that The step of determining the first target pixel point in the target area according to the number of the first pixel points in the target area comprises: In a case where the target area is an area with any pixel in the binary image as a center point, determining a first target pixel from the target area according to the number of the first pixels; In the case where the target region is an image block of the binary image, a first target pixel is determined from the target region according to the number of pixels included in the target region and the number of first pixels in the target region.

4. The method according to claim 3, characterized in that The determining the first target pixel in the target area according to the number of pixels included in the target area and the number of first pixels in the target area comprises: Determine, according to the number of pixels contained in the target area, a block type of the target area, wherein the block type includes a first block type or a second block type, wherein the first block type is used to characterize that the number of pixels contained in the target area is less than a first number threshold, and the second block type is used to characterize that the number of pixels contained in the target area is not less than the first number threshold; According to the block type and the number of first pixels in the target area, a first target pixel is determined from the target area.

5. The method according to claim 4, characterized in that The block type is a second block type; The determining the first target pixel point from the target area according to the block type and the number of the first pixel points in the target area includes: When the number of first pixel points in the target area is not less than a second number threshold, all first pixel points in the target area are determined as the first target pixel points; When the number of first pixel points in the target area is less than a second number threshold, the first target pixel point is determined in the target area according to the depth value of each pixel point in the target area.

6. The method according to claim 5, characterized in that The step of determining the first target pixel point within the target area according to the depth values ​​of each pixel point within the target area comprises: Determine the difference between the maximum depth value and the minimum depth value of each pixel in the target area; Determine foreground pixels, midground pixels, and background pixels from the target area according to the depth values ​​of each pixel in the target area, the minimum depth value, the maximum depth value, and the difference; The pixel with the largest number among the foreground pixel points, the mid-ground pixel points and the background pixel points is determined as the first target pixel point.

7. The method according to claim 2, characterized in that The updating, according to the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image to obtain an updated motion vector image and an updated depth image includes: When the target area is an image block of the binary image and the block type of the target area is a first block type, or when the target area is an area with any pixel in the binary image as a center point, pixel information of the second target pixel in the motion vector image and the depth image is replaced with pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image; In a case where the target area is an image block of the binary image and the block type of the target area is a second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target area and the depth value of each pixel in the target area, and the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight and pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image.

8. The method according to claim 7, characterized in that The step of determining the weight of the first target pixel participating in the pixel information update operation according to the number of the first pixel in the target area and the depth value of each pixel in the target area includes: When the number of first pixels in the target area is not less than a second number threshold, determining a weight of the first target pixel according to a depth value of the first pixel and a depth value of the first target pixel; When the number of first pixels in the target area is less than a second number threshold, a weight of the first target pixel is determined according to the depth values ​​of all pixels in the target area and the depth value of the first target pixel.

9. An image processing device, characterized in that: include: an acquisition module, configured to acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is a previous image frame of the first image frame, and pixels of the motion vector image, the binary image, and the depth image correspond to each other one by one; A processing module, configured to process the motion vector image and the depth image according to the binary image to obtain an updated motion vector image and an updated depth image; A generating module is used to generate an interpolated frame between the first image frame and the second image frame according to the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Determine a target area from the binary image, wherein the target area is an area with any pixel point in the binary image as a center point, or the target area is an image block in the binary image, wherein the image block is obtained by performing block processing on the binary image, and each image block corresponds to a pixel point; Determining a first target pixel in the target area according to the number of first pixels in the target area, where the first pixel is a pixel corresponding to the target object in the first image frame; According to the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain an updated motion vector image and an updated depth image, wherein the second target pixel is a pixel corresponding to the first target pixel.

11. The device according to claim 10, characterized in that The processing module is specifically used for: In a case where the target area is an area with any pixel in the binary image as a center point, determining a first target pixel from the target area according to the number of the first pixels; In the case where the target region is an image block of the binary image, a first target pixel is determined from the target region according to the number of pixels included in the target region and the number of first pixels in the target region.

12. The device according to claim 11, characterized in that The processing module is specifically used for: Determine, according to the number of pixels contained in the target area, a block type of the target area, wherein the block type includes a first block type or a second block type, wherein the first block type is used to characterize that the number of pixels contained in the target area is less than a first number threshold, and the second block type is used to characterize that the number of pixels contained in the target area is not less than the first number threshold; A first target pixel is determined from the target area according to the block type and the number of first pixels in the target area.

13. The device according to claim 10, characterized in that The processing module is specifically used for: When the target area is an image block of the binary image and the block type of the target area is a first block type, or when the target area is an area with any pixel in the binary image as a center point, pixel information of the second target pixel in the motion vector image and the depth image is replaced with pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image; In a case where the target area is an image block of the binary image and the block type of the target area is a second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target area and the depth value of each pixel in the target area, and the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight and pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image.

14. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

15. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images