Image inpainting method and device, electronic equipment and computer readable storage medium

Abstract

This application discloses an image restoration method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring a target image, and acquiring a foreground mask corresponding to the target image and a corresponding depth image to be restored, wherein the foreground mask describes the image position of a foreground region in the target image, and the depth image to be restored describes the depth information of the target image; determining target pixels to be restored in the depth image to be restored based on the foreground mask, and restoring the depth values ​​of the target pixels to obtain a target depth image corresponding to the target image. The above-described image restoration method, apparatus, electronic device, and computer-readable storage medium can efficiently and accurately restore depth errors in depth images, improving the accuracy of depth images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, in particular to an image repairing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] In recent years, 3D (Three Dimension) video and stereo imaging technology have developed rapidly. Depth images containing depth information play a key role in robot navigation and target tracking systems, three-dimensional scene reconstruction and virtual viewpoint rendering. In image processing, depth images also play an important role. For example, in image blurring processing, depth images can be used to better estimate the blurring intensity of the image. In face beautification processing, depth images can be used to more accurately perform processing such as three-dimensional enhancement of facial features and adjustment of facial features.

[0003] However, the depth information in the depth images obtained by hardware or software in the industry is still inaccurate. Therefore, how to improve the accuracy of the depth image has become a technical problem to be solved. SUMMARY

[0004] The embodiments of the present application disclose an image repairing method and device, electronic equipment and computer readable storage medium, which can efficiently and accurately repair depth errors in a depth image and improve the accuracy of the depth image.

[0005] The embodiments of the present application disclose an image repairing method, comprising:

[0006] obtaining a target image, and obtaining a foreground mask corresponding to the target image and a to-be-repaired depth image corresponding to the target image, the foreground mask being used to describe image positions of a foreground region in the target image, and the to-be-repaired depth image being used to describe depth information of the target image;

[0007] determining a target pixel point to be repaired in the to-be-repaired depth image according to the foreground mask, and repairing a depth value of the target pixel point to obtain a target depth image corresponding to the target image.

[0008] The embodiments of the present application disclose an image repairing device, comprising:

[0009] an obtaining module configured to obtain a target image, and obtain a foreground mask corresponding to the target image and a to-be-repaired depth image corresponding to the target image, the foreground mask being used to describe image positions of a foreground region in the target image, and the to-be-repaired depth image being used to describe depth information of the target image;

[0010] The repair module is used to determine the target pixel in the depth image to be repaired based on the foreground mask, and repair the depth value of the target pixel to obtain the target depth image corresponding to the target image.

[0011] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method described above.

[0012] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described above.

[0013] The image restoration method, apparatus, electronic device, and computer-readable storage medium provided in this application embodiment acquire a target image, and acquire a foreground mask and a corresponding depth image to be restored based on the target image. According to the foreground mask, target pixels to be restored in the depth image to be restored are determined, and the depth values ​​of the target pixels are restored to obtain a target depth image corresponding to the target image. Since the foreground mask can accurately describe the image position of the foreground region in the target image, for depth images to be restored with depth errors, the erroneous target pixels can be accurately identified according to the foreground mask, which can efficiently and accurately restore depth errors in the depth image and improve the accuracy of the depth image. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 Here is a flowchart of an image restoration method in one embodiment;

[0016] Figure 2 This is a flowchart illustrating the fully automated repair method used in one embodiment to repair the depth image to be repaired.

[0017] Figure 3 This is a schematic diagram of the depth image to be repaired and the foreground mask in one embodiment;

[0018] Figure 4 This is a flowchart illustrating a semi-automatic repair method used in one embodiment to repair a depth image to be repaired.

[0019] Figure 5This is a schematic diagram of selecting the region to be repaired in the depth image to be repaired in one embodiment;

[0020] Figure 6 This is a block diagram of an image restoration apparatus in one embodiment;

[0021] Figure 7 This is a structural block diagram of an electronic device in one embodiment. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0024] It is understood that the terms "first," "second," etc., used in this application may be used to describe various elements herein, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first pixel may be referred to as a second pixel, and similarly, a second pixel may be referred to as a first pixel. Both the first pixel and the second pixel are pixels, but they are not the same pixel. Furthermore, it should be noted that the terms "multiple," etc., used in the embodiments of this application refer to two or more.

[0025] In related technologies, to improve the accuracy of depth images, the following methods are commonly used to repair depth images with depth information errors: 1. Repairing depth images using traditional algorithms, such as bilateral filtering and guided filtering algorithms, to optimize uneven edges in the depth image; 2. Repairing depth images using edge detection algorithms, such as using the Sobel operator to extract the edges corresponding to the original image in the depth image, and deleting and filling erroneous areas in the depth image based on the extracted edges; 3. Using convolutional neural networks for monocular depth estimation, and using the estimation results to fill large holes in the depth image.

[0026] The aforementioned restoration methods still suffer from inaccurate depth information. For example, when using edge detection algorithms to restore depth images, the edge extraction effect is weak when the foreground and background are close together, resulting in errors in the restored depth information. Another example is that traditional algorithms can only fill small holes in depth images and cannot accurately restore depth images with large areas of incorrect depth information. Yet another example is that the depth estimation results output by convolutional neural networks for monocular depth estimation have a probability of error.

[0027] In this application embodiment, an image restoration method, apparatus, electronic device, and computer-readable storage medium are provided, which can efficiently and accurately repair depth errors in depth images, improve the accuracy of depth images, and are applicable to depth images with different types of depth information errors, thus having greater applicability.

[0028] like Figure 1 As shown, in one embodiment, an image restoration method is provided, which can be applied to electronic devices, including but not limited to mobile phones, smart wearable devices, tablets, PCs (Personal Computers), vehicle terminals, digital cameras, etc., and this application embodiment does not limit the scope of the application. The method may include the following steps:

[0029] Step 110: Obtain the target image, and obtain the foreground mask and the corresponding depth image to be repaired.

[0030] The target image can be in RGB (Red-Green-Blue), YUV (Luma-Chroma), YCbCr, or other formats, and is not limited thereto. Optionally, the target image can be an image captured in real time by an electronic device through a camera, or an image stored in the memory of an electronic device. The target image can include, but is not limited to, images of various scenes such as portraits, landscapes, and plants.

[0031] In some embodiments, the target image may also be a sample image from the training dataset, such as images from public datasets such as COCO, PASCAL VOC series, CityScapes, etc. The training dataset can be used to train the image processing model, which may include, but is not limited to, depth estimation models and foreground recognition models (such as portrait recognition models, face recognition models, etc.).

[0032] Electronic devices can acquire a foreground mask corresponding to a target image. This foreground mask can be used to describe the image location of the foreground region in the target image and to determine whether each pixel in the original sample image belongs to the foreground region or the background region. The foreground mask can also label pixels in the target image that belong to the foreground region.

[0033] The foreground mask can use different pixel values ​​to represent whether a pixel belongs to the foreground region or the background region. The foreground region can refer to the image area containing the foreground object of interest in the target image; for example, if the target image is a portrait image, the foreground region can be the portrait region. The background region refers to the image area in the target image other than the foreground region. For example, in the foreground mask, the pixel value of a pixel belonging to the foreground region is 1, and the pixel value of a pixel belonging to the background region is 0; or the pixel value of a pixel belonging to the foreground region is 255, and the pixel value of a pixel belonging to the background region is 0, etc. Optionally, the foreground mask can also use different pixel values ​​to represent the probability that a pixel belongs to the foreground region; the larger the pixel value, the greater the probability that the pixel belongs to the foreground region.

[0034] In some embodiments, the foreground mask corresponding to the target image can be obtained by performing foreground recognition on the target image using a trained foreground recognition model. The foreground recognition model can extract image features from the target image and determine the foreground region in the target image based on these image features to obtain the foreground mask. The foreground recognition model can be trained based on foreground sample images with labeled foreground regions. Optionally, the foreground recognition model can be any of the following, including but not limited to foreground segmentation models and foreground matting models, and its model architecture can be including but not limited to FCN (Fully Convolutional Networks for Semantic Segmentation), U-net, PSPNet (Pyramid Scene Parsing Network), and Deeplab series model architectures.

[0035] In other embodiments, the foreground mask corresponding to the target image can also be obtained by manual annotation. Pixels belonging to the foreground region in the target image can be annotated manually to obtain the foreground mask corresponding to the target image.

[0036] An electronic device can acquire a depth image to be repaired corresponding to a target image. This depth image is used to describe the depth information of the target image. The depth image to be repaired may include the depth value corresponding to each pixel in the target image. This depth value can be used to characterize the distance between the object and the camera. Optionally, the larger the depth value, the farther the distance. In the embodiments of this application, the depth image to be repaired may refer to a depth image that may have incorrect depth information. This depth image to be repaired may be an unrepaired depth image obtained by depth estimation of the target image using a depth estimation method.

[0037] In some embodiments, the depth image to be repaired can be obtained by depth estimation using a hardware device. For example, depth estimation can be performed using multiple cameras (e.g., dual cameras), structured light, or TOF (Time of Flight). Optionally, while the electronic device is acquiring a target image via a camera, it can also acquire the corresponding depth image to be repaired via a hardware device.

[0038] In other embodiments, the depth image to be repaired may also be obtained by using software depth estimation. Software depth estimation may include, but is not limited to, using a trained depth estimation model or other neural network for depth estimation. The depth estimation model can be trained using a depth training set, which may include multiple training images and the depth image corresponding to each training image.

[0039] Since the depth values ​​corresponding to foreground regions are usually smaller, while the depth values ​​corresponding to background regions are usually larger, the depth image to be repaired can also be used to distinguish the foreground and background regions of the target image. For example, pixels in the depth image to be repaired with a depth value greater than a first depth threshold belong to the background region, while pixels with a depth value less than a second depth threshold belong to the foreground region. The first depth threshold can be greater than the second depth threshold. Because the depth image to be repaired may contain depth information errors, leading to misidentification of the foreground and background, it is necessary to repair areas with depth information errors in the depth image to be repaired.

[0040] Step 120: Based on the foreground mask, determine the target pixel in the depth image to be repaired, and repair the depth value of the target pixel to obtain the target depth image corresponding to the target image.

[0041] In some embodiments, since the foreground mask can accurately mark pixels belonging to the foreground region in the target image, it can be used as a reference to determine whether the depth values ​​of each pixel in the depth image to be repaired are accurate, and pixels with inaccurate depth values ​​in the depth image to be repaired are identified as target pixels to be repaired. Optionally, the target pixels to be repaired may include: pixels in the depth image to be repaired that are mistakenly identified as background regions, and / or pixels in the depth image to be repaired that are mistakenly identified as foreground regions.

[0042] For example, if the depth value of a pixel in the target image is greater than the first depth threshold in the depth image to be repaired (i.e., it is identified as a background area in the depth image to be repaired), but the pixel is marked as belonging to the foreground area in the foreground mask, it can be said that the depth value of the pixel in the depth image to be repaired is incorrect, and the pixel corresponding to the depth value in the depth image to be repaired can be identified as the target pixel.

[0043] For example, if the depth value of a pixel in the target image is less than the second depth threshold in the depth image to be repaired (i.e., it is identified as a foreground region in the depth image to be repaired), but the pixel is marked as belonging to the background region in the foreground mask, it can be said that the depth value of the pixel in the depth image to be repaired is incorrect, and the pixel corresponding to the depth value in the depth image to be repaired can be identified as the target pixel.

[0044] In some embodiments, multiple different foreground masks corresponding to the target image can be obtained. These multiple different foreground masks can be obtained separately through different foreground recognition models. The multiple different foreground masks can be combined to jointly determine whether the depth values ​​of each pixel in the depth image to be repaired are accurate. For example, if a pixel in the target image is most frequently labeled as belonging to the background region in the multiple different foreground masks, and if the depth value corresponding to that pixel in the depth image to be repaired is less than a second depth threshold (i.e., it is identified as a foreground region in the depth image to be repaired), it can be said that the depth value corresponding to that pixel in the depth image to be repaired is incorrect.

[0045] Electronic devices can repair the depth value of the target pixel in the depth image to be repaired, so that the repaired depth value of the target pixel is consistent with the region marked in the foreground mask, thereby obtaining an accurate target depth image.

[0046] In some embodiments, for depth images to be repaired with different error types, the electronic device may employ different repair strategies, determine the target pixels to be repaired in the depth image to be repaired based on the foreground mask, and repair the depth values ​​of the target pixels. The error type may include, but is not limited to, at least one of edge region depth information errors, large area image region depth information errors, and complex texture region depth information errors.

[0047] Among them, depth information errors in edge regions may include situations where edge regions in the depth image to be repaired have protrusions or missing parts, small areas of burrs or holes, etc.; depth information errors in larger areas of image regions may include situations where larger areas of image regions in the depth image to be repaired have holes or redundancy, etc.; depth information errors in complex texture regions may include situations where complex background textures cause depth information errors, etc.

[0048] For each type of error, a corresponding repair strategy can be set. For example, for errors involving depth information in edge regions, a fully automatic repair strategy can be used; for errors involving depth information in larger image regions, a semi-automatic repair strategy can be used, but this is not the only option. A fully automatic repair strategy refers to a method where the repair of the depth image is performed entirely by electronic equipment without human intervention; a semi-automatic repair strategy refers to a method where human intervention is involved (such as manually selecting image regions with depth information errors in the depth image to be repaired) in conjunction with electronic equipment to perform the repair.

[0049] Optionally, the error type of the depth image to be repaired can be determined manually. The depth image to be repaired can be overlaid with a foreground mask, and then the overlaid depth image to be repaired and the foreground mask can be manually examined to determine the image area where the depth information of the depth image to be repaired is incorrect, thereby determining the error type.

[0050] In some embodiments, multiple target images and the target depth image corresponding to each target image can form a training set for training a depth estimation model, which can improve the accuracy of the trained depth estimation model.

[0051] In this embodiment, a target image is acquired, along with a foreground mask and a corresponding depth image to be repaired. Based on the foreground mask, target pixels to be repaired in the depth image are determined, and the depth values ​​of the target pixels are repaired to obtain the target depth image corresponding to the target image. Since the foreground mask can accurately describe the image position of the foreground region in the target image, for depth images with depth errors, the erroneous target pixels can be accurately identified based on the foreground mask, which can efficiently and accurately repair depth errors in the depth image and improve the accuracy of the depth image.

[0052] In some embodiments, the electronic device may employ a fully automated repair method to repair the depth image to be repaired. For example... Figure 2 As shown, a fully automatic restoration method is used to restore the depth image to be restored, which may include the following steps:

[0053] Step 202: Using the foreground mask as a reference image, traverse each first pixel point contained in the depth image to be repaired.

[0054] Optionally, in the fully automatic repair method, the entire depth image to be repaired can be traversed at the pixel level, and each first pixel in the depth image to be repaired can be checked one by one to determine whether there is a depth value error, so as to determine the target pixel to be repaired in the depth image to be repaired.

[0055] In some embodiments, if only the depth information of the edge regions in the depth image to be repaired is incorrect, the foreground mask can be used as a reference image to traverse each first pixel in the depth image to be repaired. That is, if the error type in the depth image to be repaired only includes edge region depth information errors, a fully automatic repair method can be used to repair the depth image. If only the edge regions in the depth image to be repaired are incorrect, it indicates that the area with incorrect depth information in the depth image to be repaired is small. Using a fully automatic repair method can ensure the accuracy of the repair result without affecting the repair efficiency.

[0056] In the fully automatic repair method, the foreground mask can be used as a reference to determine whether each first pixel in the depth image to be repaired has an incorrect depth value. It can be determined whether the depth value of the current first pixel in the depth image to be repaired matches the pixel value of the corresponding second pixel in the foreground mask. If they do not match, it means that the depth value of the current first pixel is incorrect.

[0057] Specifically, the pixel coordinates of the current first pixel in the depth image to be repaired are the same as the pixel coordinates of the corresponding second pixel in the foreground mask. Matching the depth value of the current first pixel with the pixel value of the corresponding second pixel in the foreground mask can mean that the region to which the current first pixel belongs, defined based on its depth value, is consistent with the region to which the corresponding second pixel belongs, as marked in the foreground mask.

[0058] Step 204: If the depth value of the current first pixel in the depth image to be repaired does not match the pixel value of the corresponding second pixel in the foreground mask, then the current first pixel is determined to be the target pixel to be repaired.

[0059] In some embodiments, the depth value of the current first pixel does not match the pixel value of the corresponding second pixel in the foreground mask, which may include, but is not limited to, at least one of the following:

[0060] Case 1: The depth value of the current first pixel is within the first depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the first pixel value.

[0061] Here, the first depth range is the depth range corresponding to the foreground region in the depth image to be repaired, and the first pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the background region. When the depth value of the current first pixel is within the first depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is also the first pixel value, it indicates that the region to which the current first pixel belongs in the depth image to be repaired is the foreground region, while the region to which the corresponding second pixel belongs in the foreground mask is the background region, and the two are inconsistent.

[0062] Case 2: The depth value of the current first pixel is within the second depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the second pixel value.

[0063] Here, the second depth range is the depth range corresponding to the background region in the depth image to be repaired, and the second pixel value is the pixel value in the foreground mask used to represent that a pixel belongs to the foreground region. When the depth value of the current first pixel is within the second depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the second pixel value, it indicates that the region to which the current first pixel belongs in the depth image to be repaired is the background region, while the region to which the corresponding second pixel belongs in the foreground mask is the foreground region, and the two are inconsistent.

[0064] For example, Figure 3This is a schematic diagram of the depth image to be repaired and the foreground mask in one embodiment. For example... Figure 3 As shown, in the depth image to be repaired, a pixel with a depth value of 'a' indicates that its depth value belongs to the second depth range; a pixel with a depth value of 'b' indicates that its depth value belongs to the first depth range. In the foreground mask, a pixel with a pixel value of 0 indicates that it belongs to the background region; a pixel with a pixel value of 255 indicates that it belongs to the foreground region. Specifically, pixel 302 in the depth image to be repaired has a depth value of 'b' (indicating it belongs to the foreground region), but the corresponding pixel in the foreground mask has a pixel value of '0' (indicating it belongs to the background region). Therefore, pixel 302 is a pixel in the depth image to be repaired that was mistakenly identified as belonging to the foreground region. Similarly, pixel 304 in the depth image to be repaired has a depth value of 'a' (indicating it belongs to the background region), but the corresponding pixel in the foreground mask has a pixel value of '255' (indicating it belongs to the foreground region). Therefore, pixel 304 is a pixel in the depth image to be repaired that was mistakenly identified as belonging to the background region. Both pixels 302 and 304 are target pixels to be repaired.

[0065] It should be noted that the first depth range and the second depth range mentioned above can be determined based on the specific depth information contained in the depth image to be repaired. The first pixel value and the second pixel value can also be determined based on the specific pixel value in the foreground mask. For example, the first pixel value is 0 and the second pixel value is 255, but it is not limited to these.

[0066] Step 206: Repair the depth value of the target pixel so that the repaired depth value of the target pixel matches the pixel value of the corresponding second pixel in the foreground mask.

[0067] In some embodiments, if the pixel value of the second pixel corresponding to the target pixel in the foreground mask is the first pixel value, it indicates that the target pixel belongs to the background region. Therefore, the depth value of the target pixel can be assigned a depth value belonging to a second depth range. Optionally, the depth value belonging to the second depth range can be the maximum depth value corresponding to the target window containing the target pixel in the depth image to be repaired. The size of the target window can be preset; for example, the size of the target window can be 1 / 10 of the image size of the depth image to be repaired, or the size of the target window can be 12*12, 15*15, etc., but is not limited thereto. The target pixel can be located in the middle of the target window or in other positions within the target window; this application does not limit this. By repairing the target pixel using the pixel values ​​within the target window containing the target pixel, the depth value of the repaired target pixel can be made more natural and conforms to the image characteristics of the depth image to be repaired.

[0068] Optionally, the depth value belonging to the second depth range can also be a depth value randomly determined from the second depth range, or the average background depth corresponding to the background area in the image to be repaired.

[0069] In some embodiments, if the pixel value of the second pixel corresponding to the target pixel in the foreground mask is the second pixel value, indicating that the target pixel belongs to the foreground region, then the depth value of the target pixel can be assigned a depth value belonging to the first depth range. Optionally, the depth value belonging to the first depth range can be the minimum depth value corresponding to the target window containing the target pixel in the depth image to be repaired. Optionally, the depth value belonging to the second depth range can also be a depth value randomly determined from the first depth range, or the average foreground depth corresponding to the foreground region in the image to be repaired, etc.

[0070] As one implementation method, when only the depth information of the edge regions in the depth image to be repaired is incorrect, the edge features of the depth image to be repaired can be extracted, and the edge regions of the depth image to be repaired can be determined based on these edge features. A foreground mask can be used as a reference image, and each first pixel point contained in the edge region of the depth image to be repaired can be traversed to determine the target pixel point to be repaired. This can reduce the computational load and improve the repair efficiency.

[0071] It should be noted that a fully automatic repair method can also be used to repair depth images with other error types, or the error type of the depth image to be repaired can be ignored and the fully automatic repair method can be used for all repairs.

[0072] In this embodiment, a fully automatic repair method is used to repair the depth information of the depth image to be repaired. The depth image to be repaired is traversed at the pixel level, which can more accurately repair the depth errors in the depth image and improve the accuracy of the depth image.

[0073] In some embodiments, the electronic device may employ a semi-automatic repair method to repair the depth image to be repaired. For example... Figure 4 As shown, a semi-automatic repair method is used to repair the depth image to be repaired, which may include the following steps:

[0074] Step 402: Determine the selected region to be repaired in the depth image to be repaired.

[0075] In some embodiments, if the image region with depth information errors in the depth image to be repaired includes image regions other than edge regions, a selected region to be repaired in the depth image to be repaired can be determined. That is, when the error type of the depth image to be repaired includes error types other than edge region depth information errors, such as depth information errors in large-area image regions or depth information errors in complex texture regions, a semi-automatic repair method can be used to repair the depth image to be repaired, which can improve the repair efficiency of the depth image while ensuring the accuracy of the repair.

[0076] In semi-automatic restoration methods, the areas to be restored can be manually selected within the depth image. The restoration personnel can compare the depth image to be restored with a foreground mask to define one or more areas to be restored. These areas are the regions with depth information errors identified by the restoration personnel through comparison with the foreground mask. The shape of the areas to be restored can be arbitrary, including regular shapes such as rectangles, squares, and circles, as well as irregular shapes; no specific limitations are imposed.

[0077] Optionally, the selection operation of the area to be repaired in the depth image selected by the repair personnel may include, but is not limited to, drawing a closed curve in the depth image through touch operation, gesture operation, eye interaction operation, etc., and the image area enclosed by the closed curve is the area to be repaired. Alternatively, a closed curve may be drawn in the depth image through an input device (such as a mouse, stylus, etc.).

[0078] For example, Figure 5 This is a schematic diagram illustrating the selection of the region to be repaired in the depth image to be repaired, as shown in one embodiment. Figure 5 As shown, the repair personnel can draw a closed curve in the depth image 512 to be repaired through various operation methods. The image area 512 enclosed by the closed curve is the area to be repaired.

[0079] Step 404: Based on the current repair mode, perform pixel-level traversal of the area to be repaired and the mask area in the foreground mask corresponding to the area to be repaired, according to pixel coordinates; wherein, the repair mode includes missing filling mode or redundant deletion mode.

[0080] In semi-automatic restoration methods, the image restoration personnel can set the current restoration mode, which may include a missing fill mode or a redundancy removal mode. The missing fill mode fills in holes in the area to be restored. In this mode, pixels misidentified as background areas are identified as target pixels, and depth values ​​are restored for these target pixels. The redundancy removal mode removes redundant parts (such as burrs or large facial redundancies) in the area to be restored. In this mode, pixels misidentified as foreground areas are identified as target pixels, and depth values ​​are restored for these target pixels.

[0081] Optionally, the image restoration personnel can compare the depth image to be restored with the foreground mask to determine whether there are holes or redundant parts in the defined area to be restored, and thus select the appropriate restoration mode. When the electronic device detects the mode selection operation, it can determine the selected restoration mode as the current restoration mode.

[0082] In some embodiments, after determining the various regions to be repaired contained in the image to be repaired, a mask region in the foreground mask corresponding to each region to be repaired can be determined. The mask region corresponding to the region to be repaired refers to the image region in the foreground mask that has the same image position as the region to be repaired. For each region to be repaired, the region to be repaired and the mask region in the foreground mask corresponding to the region to be repaired can be traversed at the pixel level according to pixel coordinates. When the current repair mode is the missing filling mode, it can be determined whether each pixel in the region to be repaired is a pixel that was mistakenly identified as a background region. When the current repair mode is the redundancy removal mode, it can be determined whether each pixel in the region to be repaired is a pixel that was mistakenly identified as a foreground region.

[0083] Step 406: In the case that the current repair mode is missing fill mode, determine the pixels in the area to be repaired that were mistakenly identified as background areas as target pixels.

[0084] In some embodiments, when the current repair mode is a missing fill mode, if the pixel value of the current pixel coordinates in the mask region is the second pixel value, and the depth value corresponding to the current pixel coordinates in the region to be repaired is greater than the average foreground depth of the depth image to be repaired, then the pixel point corresponding to the current pixel coordinates in the region to be repaired can be identified as a pixel point that was mistakenly identified as a background region, and thus the pixel point corresponding to the current pixel coordinates in the region to be repaired can be determined as the target pixel point. Here, the second pixel value is the pixel value in the foreground mask used to characterize that the pixel point belongs to the foreground region.

[0085] Before determining the target pixel in the area to be repaired, the electronic device can traverse the area to be repaired and the foreground mask, determine the pixel coordinates corresponding to the pixel marked as the foreground area in the foreground mask (i.e., the pixel with the second pixel value), determine the depth value corresponding to the same pixel coordinate in the area to be repaired, and then calculate the average value of each determined depth value to obtain the average depth of the foreground.

[0086] It should be noted that other methods can also be used to determine the pixels in the area to be repaired that have been mistakenly identified as background areas. For example, in the fully automated repair method described above, the method can be used to determine whether the depth value corresponding to the current pixel coordinates in the area to be repaired is within the second depth range, and whether the pixel value of the current pixel coordinates in the mask area is the second pixel value. However, this method is not limited to these methods.

[0087] Step 408: In the depth image to be repaired, determine the minimum depth value corresponding to the target window containing the target pixel, and assign the depth value of the target pixel to the minimum depth value.

[0088] When the current repair mode is missing fill mode, the depth value of the target pixel in the area to be repaired can be assigned the depth value corresponding to the foreground area. Optionally, the depth value of the target pixel can be assigned the minimum depth value corresponding to the target window containing the target pixel. The size of the target window can be preset, for example, the size of the target window can be 1 / 10 of the image size of the depth image to be repaired, or the size of the target window can be 12*12, 15*15, etc., but is not limited to these.

[0089] Alternatively, the depth value of the target pixel can be directly assigned to other depth values ​​belonging to the foreground region, such as the average depth of the foreground, without limitation here.

[0090] Step 410: In the case that the current repair mode is the redundancy deletion mode, determine the pixels in the area to be repaired that were mistakenly identified as the foreground area as the target pixels.

[0091] In some embodiments, when the current repair mode is a redundancy removal mode, if the pixel value of the current pixel coordinates in the mask region is the first pixel value, and the depth value corresponding to the current pixel coordinates in the region to be repaired is less than the average background depth of the depth image to be repaired, then the pixel point corresponding to the current pixel coordinates in the region to be repaired can be identified as a pixel point that was mistakenly identified as a foreground region, and the pixel point corresponding to the current pixel coordinates in the region to be repaired is determined as the target pixel point. Here, the first pixel value is the pixel value in the foreground mask used to characterize that the pixel point belongs to the background region.

[0092] Before determining the target pixel in the area to be repaired, the electronic device can traverse the area to be repaired and the foreground mask, determine the pixel coordinates corresponding to the pixel in the foreground mask that is marked as the background area (i.e., the pixel with the first pixel value), determine the depth value corresponding to the same pixel coordinate in the area to be repaired, and then calculate the average value of each determined depth value to obtain the average background depth.

[0093] It should be noted that other methods can also be used to determine the pixels in the area to be repaired that were mistakenly identified as the foreground area. For example, in the fully automated repair method described above, the method can be used to determine whether the depth value corresponding to the current pixel coordinates in the area to be repaired is within the first depth range, and whether the pixel value of the current pixel coordinates in the mask area is the first pixel value. However, this method is not limited to these methods.

[0094] Step 412: In the depth image to be repaired, determine the maximum depth value corresponding to the target window containing the target pixel, and assign the depth value of the target pixel to the maximum depth value.

[0095] When the current repair mode is redundancy deletion mode, the depth value of the target pixel in the area to be repaired can be assigned the depth value corresponding to the background area. Optionally, the depth value of the target pixel can be assigned the maximum depth value corresponding to the target window containing the target pixel.

[0096] In this embodiment, by repairing the target pixel using the depth values ​​contained within the target window, the depth values ​​of the repaired target pixel can be made to transition naturally, further improving the accuracy of the target depth image corresponding to the repaired target image.

[0097] Alternatively, the repair personnel may not set a repair mode. For each area to be repaired, they can directly traverse the area to be repaired and the corresponding mask area to identify each target pixel in the area to be repaired that has a depth value error.

[0098] In this embodiment, a semi-automatic repair method is used to repair the depth information of the depth image. This method involves traversing the manually selected repair area and its corresponding mask area to identify and repair target pixels with depth errors within the repair area. This approach more accurately repairs depth errors in the depth image and improves repair efficiency. Furthermore, repair modes can be set, employing different strategies to identify and repair target pixels within each repair area, further enhancing repair efficiency.

[0099] like Figure 6As shown, in one embodiment, an image restoration device 600 is provided, which can be applied to the above-mentioned electronic device. The image restoration device 600 may include an acquisition module 610 and a restoration module 620.

[0100] The acquisition module 610 is used to acquire a target image, as well as a foreground mask and a corresponding depth image to be repaired. The foreground mask is used to describe the image position of the foreground region in the target image, and the depth image to be repaired is used to describe the depth information of the target image.

[0101] The repair module 620 is used to determine the target pixel in the depth image to be repaired based on the foreground mask, and repair the depth value of the target pixel to obtain the target depth image corresponding to the target image.

[0102] In this embodiment, since the foreground mask can accurately describe the image position of the foreground region in the target image, for depth images with depth errors, the erroneous target pixels can be accurately identified based on the foreground mask, which can efficiently and accurately repair depth errors in the depth image and improve the accuracy of the depth image.

[0103] In one embodiment, the repair module 620 includes a traversal unit.

[0104] The traversal unit is used to traverse each first pixel point contained in the depth image to be repaired, with the foreground mask as a reference image, and if the depth value of the current first pixel point in the depth image to be repaired does not match the pixel value of the corresponding second pixel point in the foreground mask, then the current first pixel point is determined to be the target pixel point to be repaired.

[0105] In one embodiment, the repair module 620 is further configured to, when the depth information of only the edge region in the depth image to be repaired is incorrect, traverse each first pixel point contained in the depth image to be repaired, using the foreground mask as a reference image.

[0106] In one embodiment, the fact that the depth value of the current first pixel does not match the pixel value of the corresponding second pixel in the foreground mask includes: the depth value of the current first pixel is within a first depth range, and the pixel value of the corresponding second pixel in the foreground mask is the first pixel value; or, the depth value of the current first pixel is within a second depth range, and the pixel value of the corresponding second pixel in the foreground mask is the second pixel value.

[0107] Wherein, the first depth range is the depth range corresponding to the foreground region in the depth image to be repaired, the second depth range is the depth range corresponding to the background region in the depth image to be repaired, the first pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the background region, and the second pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the foreground region.

[0108] In this embodiment, a fully automatic repair method is used to repair the depth information of the depth image to be repaired. The depth image to be repaired is traversed at the pixel level, which can more accurately repair the depth errors in the depth image and improve the accuracy of the depth image.

[0109] In one embodiment, the traversal unit is further configured to determine a selected region to be repaired in the depth image to be repaired; and to perform pixel-level traversal of the region to be repaired and the mask region in the foreground mask corresponding to the region to be repaired according to the current repair mode, based on pixel coordinates; wherein the repair mode includes a missing padding mode or a redundancy removal mode; and when the current repair mode is a missing padding mode, to determine the pixels in the region to be repaired that are mistakenly identified as background regions as target pixels; and when the current repair mode is a redundancy removal mode, to determine the pixels in the region to be repaired that are mistakenly identified as foreground regions as target pixels.

[0110] In one embodiment, the traversal unit is further configured to determine the region to be repaired in the depth image to be repaired if the image region with depth information error in the depth image to be repaired includes other image regions besides edge regions.

[0111] In one embodiment, the traversal unit is further configured to, when the current repair mode is a missing filling mode, determine the pixel point corresponding to the current pixel coordinate in the area to be repaired as the target pixel point if the pixel value of the current pixel coordinate in the mask region is the second pixel value and the depth value corresponding to the current pixel coordinate in the area to be repaired is greater than the average depth of the foreground of the depth image to be repaired; wherein, the second pixel value is the pixel value in the foreground mask used to characterize that the pixel point belongs to the foreground region.

[0112] In one embodiment, the traversal unit is further configured to, when the current repair mode is the redundancy deletion mode, determine the pixel point corresponding to the current pixel coordinate in the area to be repaired as the target pixel point if the pixel value of the current pixel coordinate in the mask region is the first pixel value and the depth value corresponding to the current pixel coordinate in the area to be repaired is less than the average background depth of the depth image to be repaired; wherein, the first pixel value is the pixel value in the foreground mask used to characterize the pixel point as belonging to the background region.

[0113] In one embodiment, the repair module 620 includes a repair unit in addition to a traversal unit.

[0114] The repair unit is configured to, when the current repair mode is missing filling mode, determine the minimum depth value corresponding to the target window containing the target pixel in the depth image to be repaired, and assign the depth value of the target pixel as the minimum depth value; and when the current repair mode is redundancy deletion mode, determine the maximum depth value corresponding to the target window containing the target pixel in the depth image to be repaired, and assign the depth value of the target pixel as the maximum depth value.

[0115] In this embodiment, a semi-automatic repair method is used to repair the depth information of the depth image. This method involves traversing the manually selected repair area and its corresponding mask area to identify and repair target pixels with depth errors within the repair area. This approach more accurately repairs depth errors in the depth image and improves repair efficiency. Furthermore, repair modes can be set, employing different strategies to identify and repair target pixels within each repair area, further enhancing repair efficiency.

[0116] Figure 7 This is a structural block diagram of an electronic device in one embodiment. For example... Figure 7 As shown, the electronic device 700 may include one or more of the following components: a processor 710 and a memory 720 coupled to the processor 710, wherein the memory 720 may store one or more computer programs, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 710.

[0117] The processor 710 may include one or more processing cores. The processor 710 connects to various parts within the electronic device 700 using various interfaces and lines, and performs various functions and processes data of the electronic device 700 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 720, and by calling data stored in the memory 720. Optionally, the processor 710 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 710 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 710 and may be implemented separately using a communication chip.

[0118] The memory 720 may include random access memory (RAM) or read-only memory (ROM). The memory 720 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 720 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the electronic device 700 during use.

[0119] Understandably, the electronic device 700 may include more or fewer structural elements than those shown in the above block diagram, such as a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, sensors, etc., and may not be limited herein.

[0120] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the above embodiments.

[0121] This application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

[0122] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, etc.

[0123] Any references to memory, storage, databases, or other media used herein may include non-volatile and / or volatile memory. Suitable non-volatile memory may include ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache. By way of illustration and not limitation, RAM may take many forms, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), and Direct Rambus DRAM (DRDRAM).

[0124] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0125] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0126] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] The foregoing has provided a detailed description of an image restoration method, apparatus, electronic device, and computer-readable storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image restoration method, characterized in that, include: Acquire a target image, and acquire a foreground mask and a corresponding depth image to be repaired corresponding to the target image. The foreground mask is used to describe the image position of the foreground region in the target image, and the depth image to be repaired is used to describe the depth information of the target image. Based on the foreground mask, the target pixel in the depth image to be repaired is determined, and the depth value of the target pixel is repaired to obtain the target depth image corresponding to the target image. The step of determining the target pixel to be repaired in the depth image to be repaired based on the foreground mask includes: Using the foreground mask as a reference image, the first pixel points contained in the depth image to be repaired are traversed. If the depth value of the current first pixel in the depth image to be repaired does not match the pixel value of the corresponding second pixel in the foreground mask, then the current first pixel is determined to be the target pixel to be repaired. The current depth value of the first pixel does not match the pixel value of the corresponding second pixel in the foreground mask, including: The current depth value of the first pixel is within a first depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the first pixel value; or, The depth value of the current first pixel is within the second depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the second pixel value; Wherein, the first depth range is the depth range corresponding to the foreground region in the depth image to be repaired, the second depth range is the depth range corresponding to the background region in the depth image to be repaired, the first pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the background region, and the second pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the foreground region.

2. The method according to claim 1, characterized in that, The step of traversing each first pixel point contained in the depth image to be repaired, using the foreground mask as a reference image, includes: If the depth information of only the edge region in the depth image to be repaired is incorrect, the foreground mask is used as a reference image to traverse each first pixel point contained in the depth image to be repaired.

3. The method according to claim 1, characterized in that, The step of determining the target pixel to be repaired in the depth image to be repaired based on the foreground mask includes: Determine the selected area to be repaired in the depth image to be repaired; According to the current repair mode, the area to be repaired and the mask area in the foreground mask corresponding to the area to be repaired are traversed at the pixel level according to the pixel coordinates; wherein, the repair mode includes missing filling mode or redundant deletion mode; When the current repair mode is the missing filling mode, the pixels in the area to be repaired that were mistakenly identified as background areas are identified as target pixels. When the current repair mode is the redundancy removal mode, the pixels in the area to be repaired that were mistakenly identified as foreground areas are identified as target pixels.

4. The method according to claim 3, characterized in that, When the current repair mode is the missing fill mode, determining the pixels in the area to be repaired that were mistakenly identified as background areas as target pixels includes: When the current repair mode is the missing filling mode, if the pixel value of the current pixel coordinate in the mask area is the second pixel value, and the depth value corresponding to the current pixel coordinate in the area to be repaired is greater than the average depth of the foreground of the depth image to be repaired, then the pixel point corresponding to the current pixel coordinate in the area to be repaired is determined as the target pixel point. The second pixel value is the pixel value in the foreground mask used to characterize that the pixel belongs to the foreground region.

5. The method according to claim 3, characterized in that, When the current repair mode is the redundancy removal mode, determining the pixels in the area to be repaired that were mistakenly identified as foreground regions as target pixels includes: When the current repair mode is the redundancy deletion mode, if the pixel value of the current pixel coordinate in the mask area is the first pixel value, and the depth value corresponding to the current pixel coordinate in the area to be repaired is less than the average background depth of the depth image to be repaired, then the pixel point corresponding to the current pixel coordinate in the area to be repaired is determined as the target pixel point. Wherein, the first pixel value is the pixel value in the foreground mask used to characterize that the pixel belongs to the background region.

6. The method according to claim 3, characterized in that, The process of repairing the depth value of the target pixel includes: When the current repair mode is the missing filling mode, in the depth image to be repaired, determine the minimum depth value corresponding to the target window containing the target pixel, and assign the depth value of the target pixel to the minimum depth value; When the current repair mode is the redundancy deletion mode, in the depth image to be repaired, the maximum depth value corresponding to the target window containing the target pixel is determined, and the depth value of the target pixel is assigned to the maximum depth value.

7. The method according to any one of claims 3 to 6, characterized in that, The process of determining the region to be repaired in the depth image to be repaired includes: If the image region with depth information error in the depth image to be repaired includes image regions other than edge regions, the repair region is determined in the depth image to be repaired.

8. An image restoration device, characterized in that, include: The acquisition module is used to acquire a target image, and to acquire a foreground mask and a corresponding depth image to be repaired corresponding to the target image. The foreground mask is used to describe the image position of the foreground region in the target image, and the depth image to be repaired is used to describe the depth information of the target image. The repair module is used to determine the target pixel to be repaired in the depth image to be repaired based on the foreground mask, and repair the depth value of the target pixel to obtain the target depth image corresponding to the target image; The repair module includes a traversal unit; The traversal unit is used to traverse each first pixel point contained in the depth image to be repaired, using the foreground mask as a reference image. If the depth value of the current first pixel in the depth image to be repaired does not match the pixel value of the corresponding second pixel in the foreground mask, then the current first pixel is determined to be the target pixel to be repaired. The current depth value of the first pixel does not match the pixel value of the corresponding second pixel in the foreground mask, including: The current depth value of the first pixel is within a first depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the first pixel value; or, The depth value of the current first pixel is within the second depth range, and the pixel value of the second pixel corresponding to the current first pixel in the foreground mask is the second pixel value; Wherein, the first depth range is the depth range corresponding to the foreground region in the depth image to be repaired, the second depth range is the depth range corresponding to the background region in the depth image to be repaired, the first pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the background region, and the second pixel value is the pixel value in the foreground mask used to represent that the pixel belongs to the foreground region.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images