Image processing method and apparatus, electronic device, and readable storage medium

Abstract

The present application relates to the technical field of image processing. Disclosed are an image processing method and apparatus, an electronic device, and a readable storage medium. The image processing method comprises: acquiring a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is a frame image preceding the first image frame, and pixel points of the motion vector image, the binary image, and the depth image are in one-to-one correspondence; processing the motion vector image and the depth image on the basis of the binary image to obtain an updated motion vector image and an updated depth image; and generating an interpolated image frame between the first image frame and the second image frame on the basis of the updated motion vector image, the updated depth image, the first image frame, and the second image frame.

Description

Image processing methods, apparatus, electronic devices and readable storage media

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202411434871.4, filed on October 14, 2024, entitled “Image Processing Method, Apparatus, Electronic Device and Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0004] Frame interpolation is a technique that generates several frames based on the current frame and the previous frame and inserts them between the current frame and the previous frame. Frame interpolation can increase the frame rate of the screen, making the screen smoother, and can also reduce the rendering pressure on the graphics processing unit (GPU) of electronic devices, thus reducing power consumption.

[0005] The relevant technologies are mainly based on the frame interpolation algorithm AMD FidelityFX. TM Super Resolution 3 is used to determine the interpolated image, which has poor quality. Summary of the Invention

[0006] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can solve the technical problem of poor image quality in frame interpolation in related technologies.

[0007] In a first aspect, embodiments of this application provide an image processing method, including:

[0008] The binary image of the first image frame, the motion vector image of the first image frame relative to the second image frame, and the depth image of the first image frame are obtained. The second image frame is the previous frame of the first image frame. The pixels of the motion vector image, the binary image, and the depth image correspond one-to-one.

[0009] The motion vector image and depth image are processed based on the binary image to obtain the updated motion vector image and the updated depth image;

[0010] Based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame, an interpolated image is generated between the first image frame and the second image frame.

[0011] Secondly, embodiments of this application provide an image processing apparatus, including:

[0012] The acquisition module is used to acquire the binary image of the first image frame, the motion vector image of the first image frame relative to the second image frame, and the depth image of the first image frame. The second image frame is the previous frame of the first image frame. The pixels of the motion vector image, the binary image, and the depth image correspond one-to-one.

[0013] The processing module is used to process the motion vector image and depth image based on the binary image to obtain the updated motion vector image and the updated depth image;

[0014] The generation module is used to generate interpolated frames between the first image frame and the second image frame based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the method as described in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method as described in the first aspect.

[0019] This embodiment of the application acquires a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is the previous frame of the first image frame. The motion vector image and depth image are processed based on the binary image to obtain updated motion vector images and updated depth images. Based on the updated motion vector images and updated depth images, as well as the first and second image frames, an interpolated image is generated between the first and second image frames. That is, this embodiment of the application updates the motion vectors in the motion vector image and the depth values ​​in the depth image from a pixel perspective, combining the binary image, reducing the fragmentation of the interpolated image and improving the quality of the interpolated image. Attached Figure Description

[0020] Figure 1 is a schematic diagram of a motion vector image provided by related technologies;

[0021] Figure 2 is a flowchart of an image processing method provided in an embodiment of this application;

[0022] Figure 3 is a schematic diagram of a first image frame provided in an embodiment of this application;

[0023] Figure 4 is a schematic diagram of a motion vector image provided in an embodiment of this application;

[0024] Figure 5 is a schematic diagram of a binary image provided in an embodiment of this application;

[0025] Figure 6 is a schematic diagram of a depth image provided in an embodiment of this application;

[0026] Figure 7 is a flowchart of another image processing method provided in an embodiment of this application;

[0027] Figure 8 is a flowchart of another image processing method provided in an embodiment of this application;

[0028] Figure 9 is a flowchart of another image processing method provided in an embodiment of this application;

[0029] Figure 10 is a schematic diagram of an original motion vector image provided in an embodiment of this application;

[0030] Figure 11 is a schematic diagram comparing the processed image effects provided in an embodiment of this application;

[0031] Figure 12 is a schematic diagram comparing another processed image effect provided in an embodiment of this application;

[0032] Figure 13 is a schematic diagram comparing another processed image effect provided in an embodiment of this application;

[0033] Figure 14 is a schematic diagram comparing another processed image effect provided in an embodiment of this application;

[0034] Figure 15 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0035] Figure 16 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0036] Figure 17 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0038] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0039] As mentioned above, frame interpolation technology can improve the frame rate, making the visuals smoother, and can also reduce the rendering load on the GPU, thus lowering power consumption. For example, in 3D games, due to limitations in GPU computing power, large open-world 3D games often run at a refresh rate of 60Hz or even lower, severely impacting the user's gaming experience. Frame interpolation can improve the game's frame rate. Moreover, because the frame interpolation process is performed in a two-dimensional image space, it eliminates the need to consider the complex calculations required in a three-dimensional scene, thereby effectively reducing the GPU's computational load and power consumption.

[0040] In game frame interpolation algorithms, the quality of the motion vector map plays a crucial role in the final image quality of the interpolated frames. However, by observing the motion vector map in the white rectangle 100 in Figure 1, it can be found that the motion vectors at the edge of the character are not similar to the motion vectors of the foreground (within the character) and the background, and are therefore inaccurate. The motion vectors at this point should belong to the foreground. Therefore, the algorithm needs to correct the motion vectors at the edge of the character by expanding the motion vectors of the foreground outward to achieve the purpose of correcting the edge motion vectors.

[0041] To achieve the above objectives, the most commonly used algorithm is currently AMD FidelityFX. TM The Super Resolution 3 (FSR3) algorithm processes data based on the principle of minimizing depth values. This approach can lead to the following problems:

[0042] During motion vector dilation, the motion vector at a location with a smaller depth value may be incorrect. Direct dilation will cause the area of ​​incorrect motion vectors to be larger, eventually leading to fragmented interpolated images and affecting the interpolation effect.

[0043] Therefore, embodiments of this application provide an image processing method, apparatus, electronic device, and readable storage medium, which can solve the technical problem of poor quality of interpolated images in related technologies and reduce the occurrence of broken interpolated images.

[0044] The image processing method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0045] The image processing method provided in this application can be applied to electronic devices such as mobile phones, tablets, and laptops. This application can also be applied to scenarios such as games and videos.

[0046] Figure 2 is a flowchart of an image processing method provided in an embodiment of this application. As shown in Figure 2, the image processing method may include the following steps:

[0047] S210. Obtain the binary image of the first image frame, the motion vector image of the first image frame relative to the second image frame, and the depth image of the first image frame.

[0048] The second image frame is the image preceding the first image frame, and the pixels of the motion vector image, binary image, and depth image correspond one-to-one.

[0049] S220. Process the motion vector image and depth image based on the binary image to obtain the updated motion vector image and the updated depth image.

[0050] S230. Based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame, generate an interpolated image between the first image frame and the second image frame.

[0051] This application embodiment updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels, combining binary images, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0052] The above steps are explained in detail below:

[0053] In S210, the first image frame can be an image frame in a video, which can be a video obtained from film and television production, a video shot by a camera, an offline or online game, etc.

[0054] The second image frame and the first image frame can be adjacent image frames. Specifically, the second image frame can be the frame before the first image frame or the frame after the first image frame. In this embodiment, the second image frame is taken as the image before the first image frame.

[0055] The motion vector image is used to represent the motion vector of each pixel in the first image frame relative to the second image frame. The depth image is used to represent the depth value of each pixel in the first image frame. The binary image is used to represent which pixels belong to the target object and which pixels belong to the non-target object. The target object can be an object of user interest; for example, in a game frame, the target object can be a character. For example, the binary image can use 0 to represent a non-target object and 1 to represent a target object.

[0056] The first image frame, motion vector image, binary image, and depth image are all the same size, and their pixels correspond one-to-one. Taking the first image frame shown in Figure 3 as an example, the motion vector image, binary image, and depth image can be seen in Figures 4-6 respectively. The pixels corresponding to the same position in different images are the same, which simplifies the subsequent image processing process and eliminates the need to determine the matching relationship of pixels in each image.

[0057] In S220, the embodiments of this application can update the motion vector image and depth image from a pixel-level perspective using a binary image to obtain an updated motion vector image and an updated depth image, providing a more accurate basis for subsequently generating interpolated images between the first image frame and the second image frame.

[0058] The updated motion vector image can have the same number of pixels, but the motion vectors of some or all of the pixels have changed. Alternatively, the number of pixels can be reduced, but the motion vectors of some or all of the pixels have changed. Reducing the number of pixels appropriately lowers the image resolution, thereby reducing the power consumption of image processing.

[0059] In some embodiments, the electronic device may select a region from a binary image and update the motion vector of the corresponding pixel in the motion vector image and the depth value of the corresponding pixel in the depth image according to the motion vector and depth value of the pixel in the region, respectively.

[0060] This region can be a neighborhood centered on a pixel in a binary image, or it can be a block of an image obtained by dividing a binary image into blocks.

[0061] In other words, electronic devices can update motion vector images and depth images based on the motion vectors and depth values ​​of pixels in the neighborhood, or based on the motion vectors and depth values ​​of pixels within an image block.

[0062] Through the above processing, most erroneous motion vectors can be corrected. To further improve the image processing effect, for example, after S220, the image processing method may also include the following steps:

[0063] The updated motion vector image and the updated depth image are filtered to obtain the filtered motion vector image and the filtered depth image.

[0064] The filtering here can be median filtering or other forms of filtering. By filtering the updated motion vector image and the updated depth image, the motion vector can be made smoother, and there will be fewer holes and fragments in the motion vector within the target object, thereby further improving the quality of the interpolated image.

[0065] In S230, based on the updated motion vector image and depth image, as well as the first image frame and the second image frame, one or more interpolated frames can be generated between the first image frame and the second image frame. The specific generation process is not limited in this embodiment. Inserting the interpolated frames between the first image frame and the second image frame can increase the display frame rate of the image and improve the user experience.

[0066] Figure 7 is a flowchart of another image processing method provided in the embodiment of this application. The difference between Figure 7 and Figure 2 is that S220 in Figure 2 can be refined into S710-S730 in Figure 7.

[0067] S710. Determine the target region from the binary image.

[0068] The target region is the area containing some or all of the pixels in a binary image. For example, the target region can be a region centered on any pixel in the binary image. That is, the electronic device can select a pixel in the binary image as the center point and define the region containing that center point as the target region. For instance, the target region can be defined as the neighborhood of pixel A in the binary image with a radius of r, where the radius r is less than or equal to the shortest distance from that feature point to the edge of the binary image. For example, the target region can also be a rectangle or other shapes.

[0069] For example, a pixel can be selected as pixel A from pixels with a value of 1, or a pixel can be selected as pixel A from pixels with a value of 0. Considering that the target object may have multiple edges, in order to improve the image processing effect, for example, each pixel can also be determined as a center point. That is, for each pixel, a target region can be obtained, and a similar process can be performed for each target region.

[0070] For example, the target region can also be an image block in a binary image, which is obtained by dividing the binary image into blocks. When there are multiple image blocks, the number of pixels contained in different image blocks can be the same or different. In some embodiments, each image block includes at least one pixel. When an image block contains one pixel, the image block can be represented as 1*1; when an image block contains multiple pixels, the image block can be represented as M*N, indicating that the image block contains M*N pixels, where M and N can be the same or different. In some embodiments, M = N = 2, or M = N = 4, or M = N = 8, or M = N = 16, etc. In some embodiments, when M = N, the M*N pixels can be equivalent to one pixel, thus reducing the number of pixels, reducing the image resolution, and consequently reducing the image processing power consumption.

[0071] The embodiments of this application can process motion vector images and depth images from a pixel-level or block-level perspective, thereby improving the flexibility of the method while enhancing image quality.

[0072] S720. Determine the first target pixel from the target region based on the number of first pixels in the target region.

[0073] In this context, the first pixel is the pixel corresponding to the target object in the first image frame. For example, when 1 represents the target object and 0 represents a non-target object, the first pixel is the pixel with a value of 1. The number of first pixels is also the number of pixels with a value of 1. The target region may include 0, 1, or more first pixels.

[0074] The first target pixel is a pixel in the target region, and the number of first target pixels can be one or more. The position and number of the first target pixels are related to the number of first pixels in the target region and the selection method of the target region. That is, the embodiments of this application can dynamically determine the first target pixels according to the selection method of the target region and the number of first pixels, which can improve the image processing effect. The specific determination process of the first target pixel can be found in the following embodiments.

[0075] S730. Based on the pixel information of the first target pixel, update the pixel information of the second target pixel in the motion vector image and the depth image to obtain the updated motion vector image and the updated depth image.

[0076] Here, the second target pixel is the pixel in the motion vector image and depth image that corresponds to the first target pixel. The second target pixel and the first target pixel can have a one-to-one correspondence, or they can be a pair of P. For example, when the target region is a region centered at any pixel in the binary image, the second target pixel and the first target pixel can have a one-to-one correspondence. When the target region is an image block in the binary image, the second target pixel and the first target pixel can have a pair of P, where P represents the number of first target pixels, and P is an integer greater than or equal to 1.

[0077] Pixel information can be information that characterizes the features of a pixel. For example, in a motion vector image, pixel information is the motion vector, and in a depth image, pixel information is the depth value.

[0078] For example, the electronic device can update the motion vector of the second target pixel corresponding to the first target pixel in the motion vector image according to the motion vector of the first target pixel, and update the depth value of the second target pixel corresponding to the first target pixel in the depth image according to the depth value of the first target pixel.

[0079] The process of updating motion vectors is similar to that of updating depth values. Taking the update of motion vector of second target pixel in motion vector based on motion vector of first target pixel as an example, for instance, the motion vector of second target pixel in motion vector image can be directly replaced with the motion vector of first target pixel, or the motion vector of first target pixel can be processed to obtain processed motion vector, and the motion vector of second target pixel in motion vector image can be replaced with processed motion vector.

[0080] In this application embodiment, target regions are flexibly selected from binary images according to different selection strategies. Based on the number of pixels corresponding to the target object in the target region, a first target pixel is further determined from the target region. The pixel information of the second target pixel corresponding to the first target pixel in the motion vector image and depth image is updated based on the pixel information of the first target pixel, thereby reducing the occurrence of frame interpolation image fragmentation.

[0081] The process of determining the first target pixel is explained below:

[0082] Figure 8 is a flowchart of another image processing method provided by an embodiment of this application. The difference between Figure 8 and Figure 7 is that S720 in Figure 7 can be refined into S810-S820 in Figure 8.

[0083] S810. When the target region is a region centered on any pixel in the binary image, determine the first target pixel from the target region based on the number of first pixels.

[0084] For example, when the target region is a region centered on any pixel in a binary image, the first target pixel can be determined based on the number of first pixels.

[0085] For example, if the number of first pixels is greater than the third threshold, pixels with depth values ​​less than the first depth threshold can be identified as first target pixels; if the number of first pixels is not greater than the third threshold, pixels with depth values ​​less than the second depth threshold in the target region can be identified as first target pixels.

[0086] The value of the third quantity threshold can be set according to the actual situation. For example, the third quantity threshold can be set to a value between 0 and 1 (including 0 but not including 1).

[0087] The first depth threshold can be set according to the actual situation, as long as the selected depth value is minimized.

[0088] Assume the number of first pixels is N, where N is an integer greater than or equal to 0. For example, when N > 0, the pixel with the smallest depth value can be determined from the first pixels and designated as the first target pixel. For example, when N = 0, the pixel with the smallest depth value can be determined from the target region and designated as the first target pixel.

[0089] The embodiments of this application can determine the target region with each pixel as the center point, and flexibly determine the first target pixel from the first pixel or the target region according to the number of first pixels in the target region, thereby improving the flexibility of the method while improving the image processing effect.

[0090] S820. In the case where the target region is an image block of a binary image, the first target pixel is determined from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region.

[0091] For example, when dividing an image into blocks for a target region that is a binary image, the block type of the target region can be determined based on the number of pixels contained in the target region. Based on the block type and the number of first pixels in the target region, the first target pixel can be adaptively determined, which improves the accuracy of the first target pixel and thus improves the accuracy of the interpolated image.

[0092] The segmentation type can include a first segmentation type or a second segmentation type. The first segmentation type indicates that the number of pixels contained in the target region is less than a first threshold, while the second segmentation type indicates that the number of pixels contained in the target region is not less than the first threshold. That is, based on the number of pixels contained in the target region, the target region can be divided into small blocks or large blocks. The segmentation type for small blocks is the first segmentation type, and the segmentation type for large blocks is the second segmentation type.

[0093] The size of the first quantity threshold can be set according to the actual situation. In some embodiments, the first quantity threshold can be set to 17. For example, a target area containing 2*2 pixels or 4*4 pixels can be determined as a small block, and a target area containing 8*8 pixels or 16*16 pixels can be determined as a large block.

[0094] The embodiments of this application can determine the target region from both pixel-level and block-level perspectives. Then, for different perspectives, the first target pixel is adaptively determined by combining the number of first pixels contained in the target region, thereby improving the flexibility of the method.

[0095] In some embodiments, when the target region is a first block type, that is, when the target region is a small block, the first target pixel can be further determined based on the number of first pixels contained in the target region.

[0096] For example, the above S820 may include the following steps:

[0097] If the number of first pixels in the target area is greater than the fourth number threshold, the first pixel whose depth value is less than the third depth threshold is determined as the first target pixel.

[0098] If the number of first pixels in the target area is not greater than the fourth number threshold, pixels in the target area with a depth value less than the fourth depth threshold are identified as the first target pixels.

[0099] For example, the fourth quantity threshold can be set to a value between 0 and 1 (inclusive of 0 but exclusive of 1), and the third depth threshold is used to minimize the depth value of the first target pixel in the first pixel or target region.

[0100] For example, when the target area is a small block, if the number of first pixels N in the target area is greater than 0, the pixel with the smallest depth value among the first pixels can be determined as the first target pixel. If N = 0, the pixel with the smallest depth value in the target area can be determined as the first target pixel.

[0101] When the target region is divided into two blocks, i.e., the target region is a large block, the first target pixel can be further determined based on the number of first pixels contained in the target region.

[0102] For example, the above S820 may include the following steps:

[0103] If the number of first pixels in the target area is not less than the second number threshold, all first pixels in the target area are determined as first target pixels.

[0104] If the number of first pixels in the target area is less than the second threshold, the first target pixel is determined from the target area based on the depth value of each pixel in the target area.

[0105] Given that the target region contains M*M pixels, the second quantity threshold is... For example, when the target region is a large block, if the target region contains a certain number of first pixels... All first pixels within the target area can be designated as the first target pixel. If The first target pixel can be further determined based on the depth values ​​of each pixel within the target area.

[0106] In embodiments of this application, when the target area is a large block, the first target pixel can be adaptively determined based on the number of first pixels contained in the target area, thereby improving the flexibility of the method.

[0107] In some embodiments, the above step of "determining the first target pixel from the target area based on the depth value of each pixel in the target area" can be further refined into the following steps:

[0108] Determine the difference between the maximum and minimum depth values ​​of each pixel within the target area;

[0109] Based on the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area, determine the foreground pixels, midground pixels, and background pixels in the target area;

[0110] The pixel with the largest number among the foreground, midground, and background pixels is determined as the first target pixel.

[0111] For example, d range =d max -d min d max d represents the maximum depth value of each pixel within the target area. min d represents the minimum depth value of each pixel within the target region. range It is the difference between the maximum depth value and the minimum depth value.

[0112] For the i-th pixel within the target region, the following process can be performed, where 1 ≤ i ≤ P, and P is the number of pixels contained in the target region:

[0113] if It can be determined that the i-th pixel belongs to the foreground pixel;

[0114] if It can be determined that the i-th pixel belongs to the mid-field pixel;

[0115] if It can be determined that the i-th pixel belongs to the background pixel.

[0116] The above determination process is just one example; in practical applications, other methods can be used. This allows us to obtain the number of foreground, midground, and background pixels within the target area.

[0117] In this embodiment of the application, for example, the pixel with the largest number can be determined as the first target pixel and participate in the subsequent pixel information update calculation. For example, if the number of foreground pixels in the target area is the largest, then all foreground pixels can be determined as the first target pixel.

[0118] This application embodiment addresses the case where the target area is a large block. When the number of first pixels contained in the target area is small, the number of pixels belonging to the foreground, midground, and background can be further determined based on the depth values ​​of each pixel within the target area. The first target pixel is then determined from the foreground, midground, and background pixels based on this number. This improves the accuracy of the first target pixel, thereby enhancing the processing effect of subsequent images.

[0119] The process of determining the second target pixel is explained below:

[0120] The method for obtaining the target region differs, and the method for determining the second target pixel also differs. Specifically, in some embodiments, before S730, the image processing method may further include the following steps:

[0121] When the target region is a region centered on any pixel in the binary image, the pixel in the motion vector image and the depth image that corresponds to the center pixel of the target region is determined as the second target pixel.

[0122] When the target region is a binary image segmented into blocks, the region corresponding to the target region is determined from the motion vector image and the depth image. All pixels in the region are equivalent to one pixel, and the equivalent pixel is determined as the second target pixel.

[0123] For example, when the target region is determined from a pixel-level perspective, that is, when the region centered on any pixel in the binary image is determined as the target region, the pixel in the motion vector image and the depth image corresponding to the center pixel of the target region can be determined as the second target pixel.

[0124] For example, when the target region is determined from a block-level perspective, that is, when the target region is divided into blocks in a binary image, multiple pixels in the region corresponding to the target region in the motion vector image and depth image can be regarded as one pixel. In other words, the number of pixels in the region corresponding to the target region is reduced from multiple to one, which can reduce the amount of subsequent calculations and reduce power consumption.

[0125] The embodiments of this application can adaptively determine the second target pixel points that need to be updated in the motion vector image and depth image according to the method of obtaining the target region, which improves the flexibility of the method. In particular, when determining the target region from the block level, it can reduce the subsequent amount of calculation and reduce the processing power consumption.

[0126] Figure 9 is a flowchart of another image processing method provided by an embodiment of this application. The difference between Figure 9 and Figure 7 is that S730 in Figure 7 can be refined into S910-S930 in Figure 9.

[0127] S910. If the target region is an image block of a binary image and the block type of the target region is the first block type, or if the target region is a region centered on any pixel in the binary image, replace the pixel information of the second target pixel in the motion vector image and the depth image with the pixel information of the first target pixel to obtain the updated motion vector image and the updated depth image.

[0128] For example, when the target region is a region centered on any pixel in the binary image, the pixel information of the second target pixel in the motion vector image and the depth image can be directly replaced with the pixel information of the first target pixel.

[0129] Taking the pixel information including motion vectors as an example, for instance, when the target region is a region centered on any pixel in the binary image, the motion vector of the second target pixel in the motion vector image can be directly replaced with the motion vector of the first target pixel.

[0130] For example, when the number of first pixels contained in the target region is greater than 0, the motion vector of the pixel with the smallest depth value in the first region can be assigned to the second target pixel in the motion vector image, that is, assigned to the pixel in the motion vector image corresponding to the center pixel of the binary image. When the number of first pixels contained in the target region is equal to 0, the motion vector of the pixel with the smallest depth value in the target region can be assigned to the second target pixel in the motion vector image, that is, assigned to the pixel in the motion vector image corresponding to the center pixel of the binary image.

[0131] For example, when the target region is a small block, the motion vector of the second target pixel in the motion vector image can be directly replaced with the motion vector of the first target pixel.

[0132] For example, when the number of first pixels contained in the target region is greater than 0, the motion vector of the pixel with the smallest depth value among the first pixels can be assigned to the block in the motion vector image corresponding to the target region, and this block is equivalent to one pixel. When the number of first pixels contained in the target region is equal to 0, the motion vector of the pixel with the smallest depth value in the target region can be assigned to the block in the motion vector image corresponding to the target region, and this block is equivalent to one pixel.

[0133] S920. When the target region is an image block of a binary image and the block type of the target region is the second block type, determine the weight of the first target pixel in the pixel information update operation based on the number of first pixels in the target region and the depth value of each pixel in the target region.

[0134] Here, the weight of the first target pixel in the pixel information update operation is used to characterize the probability that the first target pixel is selected. The larger the weight, the greater the probability that the first target pixel is selected. When the target area is a large block, the weight of each first target pixel in the pixel information update operation can be dynamically determined according to the number of first pixels contained in the target area and the depth value of each pixel in the target area.

[0135] The number of first pixels contained in the target area varies, and the corresponding weights are determined in different ways. The specific process of determining the weights is explained below in conjunction with the number of first pixels.

[0136] In some embodiments, if the number of first pixels in the target region is not less than the second quantity threshold, the weight of the first target pixel can be determined based on the depth value of the first pixel and the depth value of the first target pixel.

[0137] For example, the minimum depth value, the maximum depth value, the first difference between the depth value and the minimum depth value of the first target pixel, and the second difference between the maximum depth value and the minimum depth value of the first pixel can be determined; the weight of the first target pixel is determined based on the ratio of the first difference and the second difference.

[0138] Using the second quantity threshold as For example, when At this time, the minimum and maximum depth values ​​can be determined from the depth values ​​of the first pixel within the target area, denoted as d respectively. min1 and d max1 By combining the depth value of the first target pixel, the weight of each first target pixel can be obtained.

[0139] For example, Where, p i d represents the weight of the i-th first target pixel, that is, the weight of the i-th first pixel within the target region. i Let be the depth value of the i-th first target pixel, that is, the depth value of the i-th first pixel within the target area.

[0140] In some embodiments, if the number of first pixels in the target area is less than a second quantity threshold, the weight of the first target pixel can be determined based on the depth values ​​of all pixels in the target area and the depth value of the first target pixel.

[0141] For example, the minimum depth value, maximum depth value, third difference between the depth value of the first target pixel and the minimum depth value, and fourth difference between the maximum depth value and the minimum depth value of all pixels in the target area can be determined; the weight of the first target pixel can be determined based on the ratio of the third difference and the fourth difference.

[0142] For example, when At this point, the minimum and maximum depth values ​​can be determined from the depth values ​​of all pixels within the target area, denoted as d. min2 and d max2 Therefore, the fourth difference d between the minimum depth value and the maximum depth value can be obtained. range =d max2 -d min2 Then, by combining the depth value of the first target pixel, the weight of each first target pixel can be obtained.

[0143] For example, Where, p j d represents the weight of the j-th first target pixel. jd represents the depth value of the j-th first target pixel. Here, the first target pixel is the most numerous foreground, midground, or background pixel within the target region. The sum of the number of foreground, midground, and background pixels equals the total number of pixels contained within the target region. j -d min2 This is the third difference.

[0144] When the target area is a large block, this application embodiment can further adaptively determine the weight of each first target pixel participating in the subsequent update operation based on the number of first pixels in the target area and the depth value of each pixel in the target area, providing a more accurate basis for the update operation of subsequent pixels.

[0145] S930. Based on the weight and pixel information of the first target pixel, update the pixel information of the second target pixel in the motion vector image and the depth image to obtain the updated motion vector image and the updated depth image.

[0146] Once the weights of each first target pixel are determined, the pixel information of the second target pixel in the motion vector image and depth image can be re-determined based on the weights and pixel information of each first target pixel. The re-determined pixel information is then used to replace the original pixel information, thus updating the pixel information.

[0147] The embodiments of this application can adaptively determine the pixel information of corresponding pixels in the motion vector image and depth image according to the method of determining the target area and the number of first pixels contained in the target area, thereby reducing image fragmentation and improving image quality.

[0148] In some embodiments, S930 may include the following steps:

[0149] Determine the weight of the first target pixel and the weighted sum of the corresponding pixel information, as well as the weighted sum of the first target pixels;

[0150] The ratio of the weighted sum to the weighted sum is used to determine the pixel information of the second target pixel in the motion vector image and the depth image.

[0151] For example, when hour, Where mv represents the motion vector of the second target pixel in the motion vector image, d represents the depth value of the second target pixel in the depth image, and mv and d are block-level pixel information. i and d i These are the motion vector and depth value of the i-th first target pixel, respectively.

[0152] For example, when hour, Where, N f This represents the number of pixels contained in the foreground, midground, or background with the most pixels within the target area, MV. j and d j These are the motion vector and depth value of the j-th first target pixel, respectively.

[0153] This application embodiment determines the pixel information of the second target pixel in the motion vector image and depth image based on the weight and pixel information of each first target pixel, taking into account the influence of each first target pixel, thereby more accurately determining the pixel information of the corresponding pixel in the motion vector image and depth image, reducing image fragmentation and improving image quality.

[0154] The image processing method provided in the embodiments of this application will be described below with reference to several effect diagrams.

[0155] Figure 10 is the original motion vector image, that is, the unprocessed motion vector image. Figure 11 is a schematic diagram comparing the effects of pixel-level processing in this application and related technologies. In this application, 110 is the motion vector image obtained after processing in the embodiment of this application, and 111 is the motion vector image obtained after processing in related technologies.

[0156] Figure 12 shows a comparison of the effects after median filtering, where 120 is the motion vector image obtained after pixel-level processing and filtering in the embodiment of this application, and 121 is the motion vector image obtained after processing and filtering using related techniques.

[0157] Figure 13 is a schematic diagram comparing the effects of block-level processing in this application with related technologies, where 130 is the motion vector image obtained after processing in the embodiment of this application, and 131 is the motion vector image obtained after processing in related technologies. Figure 13 uses a small block as an example.

[0158] Figure 14 is a comparison of the effects after median filtering, where 140 is the motion vector image obtained after processing and filtering from the block level in the embodiment of this application, and 141 is the motion vector image obtained after processing and filtering with related techniques.

[0159] In the above schematic diagram, the yellow ellipse represents the area with obvious contrast. As can be seen from the above effect diagram, the image processing method provided by the embodiment of this application can effectively improve the phenomena such as hollowness and fragmentation of motion vectors in the figure from both pixel-level and block-level perspectives, resulting in better image quality.

[0160] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device or a processing module within that image processing device for executing the image processing method. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0161] Figure 15 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application.

[0162] As shown in Figure 15, the image processing device 1500 may include:

[0163] The acquisition module 1501 is used to acquire the binary image of the first image frame, the motion vector image of the first image frame relative to the second image frame, and the depth image of the first image frame. The second image frame is the previous frame image of the first image frame. The pixels of the motion vector image, the binary image, and the depth image correspond one-to-one.

[0164] The processing module 1502 is used to process the motion vector image and the depth image based on the binary image to obtain the updated motion vector image and the updated depth image.

[0165] The generation module 1503 is used to generate an interpolated frame between the first image frame and the second image frame based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

[0166] This embodiment of the application acquires a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is the previous frame of the first image frame. The motion vector image and depth image are processed based on the binary image to obtain updated motion vector images and updated depth images. Based on the updated motion vector images and updated depth images, as well as the first and second image frames, an interpolated image is generated between the first and second image frames. That is, this embodiment of the application updates the motion vectors in the motion vector image and the depth values ​​in the depth image from a pixel perspective, combining the binary image, reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0167] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0168] The target region is determined from the binary image. The target region is the region centered on any pixel in the binary image, or the target region is an image block in the binary image. The image block is obtained by dividing the binary image into blocks, and each image block corresponds to a pixel.

[0169] Based on the number of first pixels in the target region, the first target pixel is determined from the target region. The first pixel is the pixel corresponding to the target object in the first image frame.

[0170] Based on the pixel information of the first target pixel, update the pixel information of the second target pixel in the motion vector image and the depth image to obtain the updated motion vector image and the updated depth image. The second target pixel is the pixel corresponding to the first target pixel.

[0171] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0172] When the target region is a region centered on any pixel in the binary image, the first target pixel is determined from the target region based on the number of first pixels.

[0173] In the case where the target region is a binary image segment, the first target pixel is determined from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region.

[0174] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0175] Based on the number of pixels contained in the target region, the block type of the target region is determined. The block type includes a first block type or a second block type. The first block type is used to indicate that the number of pixels contained in the target region is less than a first number threshold, and the second block type is used to indicate that the number of pixels contained in the target region is not less than the first number threshold.

[0176] The first target pixel is determined from the target region based on the block type and the number of first pixels in the target region.

[0177] In some possible implementations of the embodiments of this application, the block type is a second block type;

[0178] Processing module 1502 is specifically used for:

[0179] If the number of first pixels in the target area is not less than the second number threshold, all first pixels in the target area are determined as first target pixels.

[0180] If the number of first pixels in the target area is less than the second threshold, the first target pixel is determined from the target area based on the depth value of each pixel in the target area.

[0181] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0182] Determine the difference between the maximum and minimum depth values ​​of each pixel within the target area;

[0183] Based on the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area, determine the foreground pixels, midground pixels, and background pixels in the target area;

[0184] The pixel with the largest number among the foreground, midground, and background pixels is determined as the first target pixel.

[0185] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0186] If the target region is a binary image block and the block type of the target region is the first block type, or if the target region is a region centered on any pixel in the binary image, the pixel information of the second target pixel in the motion vector image and the depth image is replaced with the pixel information of the first target pixel to obtain the updated motion vector image and the updated depth image.

[0187] When the target region is a binary image block and the block type of the target region is the second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target region and the depth value of each pixel in the target region. Then, according to the weight of the first target pixel and the pixel information, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain the updated motion vector image and the updated depth image.

[0188] In some possible implementations of the embodiments of this application, the processing module 1502 is specifically used for:

[0189] If the number of first pixels in the target area is not less than the second number threshold, the weight of the first target pixel is determined based on the depth value of the first pixel and the depth value of the first target pixel.

[0190] If the number of first pixels in the target area is less than the second threshold, the weight of the first target pixel is determined based on the depth values ​​of all pixels in the target area and the depth value of the first target pixel.

[0191] The image processing device in this application embodiment can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0192] The electronic device in this application embodiment can be a terminal with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0193] The image processing apparatus provided in this application embodiment can implement the various processes in the image processing method embodiments of Figures 1 to 9 and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0194] As shown in Figure 16, this application embodiment also provides an electronic device 1600, including a processor 1601 and a memory 1602. The memory 1602 stores programs or instructions that can run on the processor 1601. When the program or instructions are executed by the processor 1601, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0195] It should be noted that the electronic devices in the embodiments of this application include the mobile terminals and non-mobile terminals mentioned above.

[0196] Figure 17 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0197] The electronic device 1700 includes, but is not limited to, components such as: radio frequency unit 1701, network module 1702, audio output unit 1703, input unit 1704, sensor 1705, display unit 1706, user input unit 1707, interface unit 1708, memory 1709, and processor 1710.

[0198] Those skilled in the art will understand that electronic device 1700 may also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to processor 1710 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The structure of electronic device 1700 shown in Figure 17 does not constitute a limitation on electronic device 1700. Electronic device 1700 may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0199] The processor 1710 is used to acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame. The second image frame is the previous frame of the first image frame. The pixels of the motion vector image, the binary image, and the depth image correspond one-to-one.

[0200] The motion vector image and depth image are processed based on the binary image to obtain the updated motion vector image and the updated depth image;

[0201] Based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame, an interpolated image is generated between the first image frame and the second image frame.

[0202] This application embodiment updates the motion vector in the motion vector image and the depth value in the depth image from the perspective of pixels, combining binary images, thereby reducing the fragmentation of the interpolated image and improving the quality of the interpolated image.

[0203] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0204] The target region is determined from the binary image. The target region is the region centered on any pixel in the binary image, or the target region is an image block in the binary image. The image block is obtained by dividing the binary image into blocks, and each image block corresponds to a pixel.

[0205] Based on the number of first pixels in the target region, the first target pixel is determined from the target region. The first pixel is the pixel corresponding to the target object in the first image frame.

[0206] Based on the pixel information of the first target pixel, update the pixel information of the second target pixel in the motion vector image and the depth image to obtain the updated motion vector image and the updated depth image. The second target pixel is the pixel corresponding to the first target pixel.

[0207] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0208] When the target region is a region centered on any pixel in the binary image, the first target pixel is determined from the target region based on the number of first pixels.

[0209] In the case where the target region is a binary image segment, the first target pixel is determined from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region.

[0210] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0211] Based on the number of pixels contained in the target region, the block type of the target region is determined. The block type includes a first block type or a second block type. The first block type is used to indicate that the number of pixels contained in the target region is less than a first number threshold, and the second block type is used to indicate that the number of pixels contained in the target region is not less than the first number threshold.

[0212] The first target pixel is determined from the target region based on the block type and the number of first pixels in the target region.

[0213] In some possible implementations of the embodiments of this application, the block type is a second block type;

[0214] Processor 1710, specifically used for:

[0215] If the number of first pixels in the target area is not less than the second number threshold, all first pixels in the target area are determined as first target pixels.

[0216] If the number of first pixels in the target area is less than the second threshold, the first target pixel is determined from the target area based on the depth value of each pixel in the target area.

[0217] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0218] Determine the difference between the maximum and minimum depth values ​​of each pixel within the target area;

[0219] Based on the depth value, minimum depth value, maximum depth value, and difference of each pixel in the target area, determine the foreground pixels, midground pixels, and background pixels in the target area;

[0220] The pixel with the largest number among the foreground, midground, and background pixels is determined as the first target pixel.

[0221] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0222] If the target region is a binary image block and the block type of the target region is the first block type, or if the target region is a region centered on any pixel in the binary image, the pixel information of the second target pixel in the motion vector image and the depth image is replaced with the pixel information of the first target pixel to obtain the updated motion vector image and the updated depth image.

[0223] When the target region is a binary image block and the block type of the target region is the second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target region and the depth value of each pixel in the target region. Then, according to the weight of the first target pixel and the pixel information, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain the updated motion vector image and the updated depth image.

[0224] In some possible implementations of embodiments of this application, the processor 1710 is specifically used for:

[0225] If the number of first pixels in the target area is not less than the second number threshold, the weight of the first target pixel is determined based on the depth value of the first pixel and the depth value of the first target pixel.

[0226] If the number of first pixels in the target area is less than the second threshold, the weight of the first target pixel is determined based on the depth values ​​of all pixels in the target area and the depth value of the first target pixel.

[0227] It should be understood that, in this embodiment, the input unit 1704 may include a graphics processing unit (GPU) 17041 and a microphone 17042. The GPU 17041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 17061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1707 includes a touch panel 17071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 17071 may include a touch detection device and a touch controller. Other input devices 17072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0228] The memory 1709 can be used to store software programs and various data. The memory 1709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1709 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0229] Processor 1710 may include one or more processing units; optionally, processor 1710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1710.

[0230] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0231] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0232] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0233] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0234] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the image processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0235] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0236] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0237] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, comprising: Obtain the binary image of the first image frame, the motion vector image of the first image frame relative to the second image frame, and the depth image of the first image frame, wherein the second image frame is the previous frame image of the first image frame, and the pixels of the motion vector image, the binary image, and the depth image correspond one-to-one. The motion vector image and the depth image are processed based on the binary image to obtain an updated motion vector image and an updated depth image; Based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame, an interpolated image is generated between the first image frame and the second image frame.

2. The method of claim 1, wherein, The step of processing the motion vector image and the depth image based on the binary image to obtain updated motion vector images and updated depth images includes: Determine a target region from the binary image. The target region is a region centered on any pixel in the binary image, or the target region is an image block in the binary image. The image block is obtained by dividing the binary image into blocks, and each image block corresponds to one pixel. Based on the number of first pixels in the target region, a first target pixel is determined from the target region, wherein the first pixel is the pixel corresponding to the target object in the first image frame; Based on the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain the updated motion vector image and the updated depth image, wherein the second target pixel is the pixel corresponding to the first target pixel.

3. The method of claim 2, wherein, Determining the first target pixel from the target region based on the number of first pixels in the target region includes: When the target region is a region centered on any pixel in the binary image, a first target pixel is determined from the target region based on the number of the first pixels. When the target region is an image block of the binary image, a first target pixel is determined from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region.

4. The method of claim 3, wherein, Determining the first target pixel from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region includes: Based on the number of pixels contained in the target region, the block type of the target region is determined. The block type includes a first block type or a second block type. The first block type is used to indicate that the number of pixels contained in the target region is less than a first number threshold, and the second block type is used to indicate that the number of pixels contained in the target region is not less than the first number threshold. Based on the block type and the number of first pixels in the target region, a first target pixel is determined from the target region.

5. The method of claim 4, wherein, The block type is the second block type; The step of determining the first target pixel from the target region based on the block type and the number of first pixels in the target region includes: If the number of first pixels in the target area is not less than the second number threshold, all first pixels in the target area are determined as the first target pixels. If the number of first pixels in the target area is less than the second number threshold, the first target pixel is determined from the target area based on the depth value of each pixel in the target area.

6. The method according to claim 5, wherein, Determining the first target pixel from the target region based on the depth values ​​of each pixel within the target region includes: Determine the difference between the maximum and minimum depth values ​​of each pixel within the target area; Based on the depth values ​​of each pixel within the target area, the minimum depth value, the maximum depth value, and the difference, foreground pixels, mid-ground pixels, and background pixels are determined from the target area. The pixel with the largest number among the foreground pixels, the midground pixels, and the background pixels is determined as the first target pixel.

7. The method according to claim 2, wherein, The step of updating the pixel information of the second target pixel in the motion vector image and the depth image based on the pixel information of the first target pixel to obtain the updated motion vector image and the updated depth image includes: If the target region is an image block of the binary image and the block type of the target region is the first block type, or if the target region is a region centered on any pixel in the binary image, the pixel information of the second target pixel in the motion vector image and the depth image is replaced with the pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image. When the target region is an image block of the binary image and the block type of the target region is the second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target region and the depth value of each pixel in the target region. Then, the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight of the first target pixel and the pixel information, so as to obtain the updated motion vector image and the updated depth image.

8. The method according to claim 7, wherein, The step of determining the weight of the first target pixel in the pixel information update operation based on the number of first pixels in the target region and the depth value of each pixel in the target region includes: If the number of first pixels in the target region is not less than the second number threshold, the weight of the first target pixel is determined based on the depth value of the first pixel and the depth value of the first target pixel. If the number of first pixels in the target area is less than the second number threshold, the weight of the first target pixel is determined based on the depth values ​​of all pixels in the target area and the depth value of the first target pixel.

9. An image processing apparatus, comprising: The acquisition module is used to acquire a binary image of a first image frame, a motion vector image of the first image frame relative to a second image frame, and a depth image of the first image frame, wherein the second image frame is the previous frame of the first image frame, and the pixels of the motion vector image, the binary image, and the depth image correspond one-to-one. The processing module is used to process the motion vector image and the depth image based on the binary image to obtain an updated motion vector image and an updated depth image; The generation module is configured to generate an interpolated frame between the first image frame and the second image frame based on the updated motion vector image and the updated depth image, as well as the first image frame and the second image frame.

10. The apparatus according to claim 9, wherein, The processing module is specifically used for: Determine a target region from the binary image. The target region is a region centered on any pixel in the binary image, or the target region is an image block in the binary image. The image block is obtained by dividing the binary image into blocks, and each image block corresponds to one pixel. Based on the number of first pixels in the target region, a first target pixel is determined from the target region, wherein the first pixel is the pixel corresponding to the target object in the first image frame; Based on the pixel information of the first target pixel, the pixel information of the second target pixel in the motion vector image and the depth image is updated to obtain the updated motion vector image and the updated depth image, wherein the second target pixel is the pixel corresponding to the first target pixel.

11. The apparatus according to claim 10, wherein, The processing module is specifically used for: When the target region is a region centered on any pixel in the binary image, a first target pixel is determined from the target region based on the number of the first pixels. When the target region is an image block of the binary image, a first target pixel is determined from the target region based on the number of pixels contained in the target region and the number of first pixels in the target region.

12. The apparatus according to claim 11, wherein, The processing module is specifically used for: Based on the number of pixels contained in the target region, the block type of the target region is determined. The block type includes a first block type or a second block type. The first block type is used to indicate that the number of pixels contained in the target region is less than a first number threshold, and the second block type is used to indicate that the number of pixels contained in the target region is not less than the first number threshold. Based on the block type and the number of first pixels in the target region, a first target pixel is determined from the target region.

13. The apparatus according to claim 10, wherein, The processing module is specifically used for: If the target region is an image block of the binary image and the block type of the target region is the first block type, or if the target region is a region centered on any pixel in the binary image, the pixel information of the second target pixel in the motion vector image and the depth image is replaced with the pixel information of the first target pixel to obtain an updated motion vector image and an updated depth image. When the target region is an image block of the binary image and the block type of the target region is the second block type, the weight of the first target pixel participating in the pixel information update operation is determined according to the number of first pixels in the target region and the depth value of each pixel in the target region. Then, the pixel information of the second target pixel in the motion vector image and the depth image is updated according to the weight of the first target pixel and the pixel information, so as to obtain the updated motion vector image and the updated depth image.

14. An electronic device comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as claimed in any one of claims 1 to 8.

15. A readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 8.

16. A computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method as claimed in any one of claims 1 to 8.

17. A chip comprising a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method as claimed in any one of claims 1 to 8.

18. An electronic device configured to perform the steps of the method as claimed in any one of claims 1 to 8.

Images