Occluded image restoration method, device, computer equipment and readable storage medium
By performing depth information clustering and progressive inpainting on occluded images, the problem of inaccurate removal of occluders in traditional methods is solved, thus improving the quality of image restoration.
Patent Information
- Application Number
- CN202411046384.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Traditional methods for restoring occluded images often fail to accurately and effectively remove occluders, resulting in blurred, artifact-ridden, or color-distorted images with low quality.
By acquiring the image to be processed, the occlusion object mask is determined, and clustering is performed according to the depth information of each occlusion object mask to obtain a merged mask of at least two image depth categories. The image occlusion is then repaired sequentially according to the depth order from the image foreground to the image background.
It reduces confusion between objects occluded at different depths during the image occlusion restoration process, improves the image quality after removing occluders, and achieves accurate occluded image restoration.
Smart Images

Figure CN118587128B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for restoring an occluded image. Background Art
[0002] With the development of computer technology, digital image processing technology has been widely used in various fields. Especially in image restoration technology, removing occlusions in images has important application value in the fields of cultural relics restoration, image editing and scene analysis. In daily life and work, we often encounter situations where images are partially blocked, which not only affects the visual effect of the image, but also brings difficulties to subsequent image analysis and processing.
[0003] However, traditional occluded image restoration methods often find it difficult to accurately and effectively remove occluders when dealing with occluders, resulting in problems such as blur, artifacts or color distortion in the restored image, resulting in low image quality after removing the occluders. Summary of the invention
[0004] Based on this, it is necessary to provide an occluded image restoration method, device, computer equipment, computer-readable storage medium and computer program product that can improve the image quality of the image obtained after removing the occluder in order to solve the above technical problems.
[0005] In a first aspect, the present application provides a method for restoring an occluded image, comprising:
[0006] Acquire an image to be processed, wherein the image to be processed includes at least two blocking objects that block the main object;
[0007] Determine each occluded object mask of the image to be processed, where the occluded object mask is used to characterize the distribution area of the occluded object in the image to be processed;
[0008] Clustering the occluded object masks according to their respective depth information to obtain merged masks of at least two image depth categories;
[0009] According to the respective merged masks of at least two image depth categories, image occlusion restoration is performed on the images to be processed in sequence according to the image depth sequence from the image foreground to the image background, so as to obtain a target image after removing the occluded object.
[0010] In a second aspect, the present application also provides a device for restoring an occluded image, comprising:
[0011] A module for acquiring an image to be processed, used for acquiring an image to be processed, wherein the image to be processed includes at least two blocking objects that block the main object;
[0012] An occluded object mask determination module is used to determine each occluded object mask of the image to be processed, and the occluded object mask is used to characterize the distribution area of the occluded object in the image to be processed;
[0013] A merged mask obtaining module, used for clustering the masks of the occluded objects according to their respective depth information, to obtain merged masks of at least two image depth categories;
[0014] The image occlusion restoration module is used to perform image occlusion restoration on the images to be processed in sequence according to the merged masks of at least two image depth categories, and in the order of image depth from image foreground to image background, to obtain a target image after removing the occluded objects.
[0015] In one of the embodiments, the merged mask acquisition module is further used to cluster each occlusion object mask into at least two image depth categories according to the depth information of each occlusion object mask, each image depth category corresponds to a different depth information range; and merge the occlusion object masks included in each of the at least two image depth categories to obtain merged masks for each of the at least two image depth categories.
[0016] In one of the embodiments, the merged mask acquisition module is also used to split the merged mask of the target image depth category based on the depth information to obtain at least two split masks when the target image depth category in at least two image depth categories meets the mask splitting condition; and use the at least two split masks as the merged masks of the target image depth category respectively.
[0017] In one of the embodiments, the image occlusion repair module is further used to determine the repair processing order of the merged masks of at least two image depth categories according to the image depth sequence from the image foreground to the image background; determine the first merged mask from the merged masks of at least two image depth categories according to the repair processing order; perform image occlusion repair on the image to be processed based on the first merged mask through the image repair model to obtain a repaired image of the first repair processing; for each repair processing starting from the second repair processing in the repair processing sequence, perform this repair processing on the repaired image obtained by the previous repair processing based on the merged mask corresponding to this repair processing through the image repair model to obtain the repaired image of this repair processing, until the repaired image obtained by the last repair processing is obtained; according to the repaired image obtained by the last repair processing, obtain the target image after removing the occluded object.
[0018] In one embodiment, the occluded image restoration device also includes a depth information determination module, which is used to determine the pixel depth information of each pixel in the image to be processed; based on the pixel depth information of each pixel, the pixel depth information of each pixel covered by each occluded object mask is determined respectively; according to the pixel depth information of each pixel covered by each occluded object mask, the depth information of each occluded object mask is obtained.
[0019] In one embodiment, the depth information determination module is also used to filter the pixel points covered by each occluding object mask in each occluding object mask to obtain the filtered target pixel points; based on the pixel depth information of the target pixel points of each occluding object mask, the depth information of each occluding object mask is obtained respectively.
[0020] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above occluded image restoration method are implemented.
[0021] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above occluded image restoration method are implemented.
[0022] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above occluded image restoration method are implemented.
[0023] The above-mentioned occluded image restoration method, apparatus, computer device, computer-readable storage medium and computer program product, for an image to be processed including at least two occluded objects, clusters the occluded object masks corresponding to the depth information of the occluded object masks used to characterize the occluded objects to obtain merged masks of at least two image depth categories, and sequentially performs image occlusion restoration on the image to be processed in the order of image depth from the image foreground to the image background to obtain a target image after removing the occluded objects. By clustering the occluded object masks characterizing the occluded objects according to the depth information corresponding to the occluded object masks to obtain merged masks of at least two image depth categories, and sequentially performing image occlusion restoration in the order of image depth from the image foreground to the image background, the confusion between occluded objects of different depths during the image occlusion restoration process can be reduced, and accurate occluded image restoration can be performed, thereby improving the image quality of the image obtained after removing the occluders. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 A diagram showing an application environment of an occluded image restoration method in an embodiment;
[0026] Figure 2 is a schematic flow chart of a method for restoring an occluded image in one embodiment;
[0027] Figure 3 is a schematic flow chart of a method for restoring an occluded image in yet another embodiment;
[0028] Figure 4 A schematic diagram of a process flow of image occlusion repair processing in one embodiment;
[0029] Figure 5 is a schematic flow chart of a method for restoring an occluded image in yet another embodiment;
[0030] Figure 6 A schematic diagram of a process of mask cluster splitting processing in one embodiment;
[0031] Figure 7 A schematic diagram of comparison of occlusion removal of a city scene image in one embodiment;
[0032] Figure 8 is a structural block diagram of an occluded image restoration device in one embodiment;
[0033] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0035] The occluded image restoration method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 may include a collection device, such as a camera, so that images can be collected by the terminal 102, such as shooting in an urban scene to obtain a scene image of the urban scene. When the collected image includes at least two occluding objects that occlude the main object, it is necessary to perform image restoration processing on the collected image, specifically, it is necessary to remove the occluding objects in the image, then the terminal 102 can use the collected image as the image to be processed, and send the image to be processed to the server 104. After receiving the image to be processed sent by the terminal 102, the server 104 clusters each occlusion object mask according to the depth information corresponding to the occlusion object mask used to characterize the occlusion object, and obtains the merged mask of at least two image depth categories. The server 104 then performs image occlusion repair on the image to be processed in sequence according to the image depth order from the image foreground to the image background according to the merged mask of the at least two image depth categories, and obtains the target image after removing the occlusion object. The server 104 can feed back the target image after removing the occlusion object to the terminal 102.
[0036] In some embodiments, the occluded image restoration method can also be implemented independently by the terminal 102 or the server 104. For example, after the terminal 102 acquires the image to be processed, the terminal 102 can directly perform image processing on the image to be processed to obtain the target image after removing the occluded object; for example, the server 104 can directly obtain the image to be processed from the data storage system, and perform image processing based on the obtained image to be processed to obtain the target image after removing the occluded object.
[0037] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.
[0038] In an exemplary embodiment, Figure 2 As shown, a method for restoring an occluded image is provided, and the method is applied to Figure 1The server in the example is used to illustrate, including the following steps 202 to 208. Among them:
[0039] Step 202: Acquire an image to be processed, wherein the image to be processed includes at least two occluding objects that occlude the main object.
[0040] Among them, the image to be processed is an image that needs to be processed for occlusion restoration, that is, the image to be processed is an occluded image, which includes an occluded object that is occluded by the main object, and the occluded object may include at least two. The main object is the object that needs to be displayed as the core in the image, and the specific object type of the main object is determined according to the actual scene. For example, in an urban scene image, the main object may be a building in the city; in another example, in a sports scene image, the main object is a sports venue. The occluded object is an object that forms an occlusion for the main object, and may specifically include various types of occluders, such as birds, pedestrians, numbers, vehicles and other occluders.
[0041] Exemplarily, the server may obtain an image to be processed that needs to be repaired by image occlusion, and the image to be processed may be sent by the terminal to the server. The image to be processed needs to remove occlusion objects, that is, the image to be processed includes occlusion objects that occlude the main object, and the occlusion objects include multiple ones.
[0042] Step 204 : determining masks of each occluding object of the image to be processed, where the occluding object mask is used to characterize the distribution area of the occluding object in the image to be processed.
[0043] Among them, the occluded object mask characterizes the distribution area of the occluded object in the image to be processed, that is, the occluded object mask is determined according to the distribution position of the occluded object in the image to be processed, such as the occluded object mask body may include a mask map (Maks). Exemplarily, the server may determine the occluded object mask corresponding to the image to be processed, and the occluded object mask is determined according to the distribution position of the occluded object in the image to be processed, and specifically, the corresponding occluded object mask may be determined according to the contour of the occluded object. In a specific application, each occluded object may have a corresponding occluded object mask, and the occluded object mask may cover the corresponding occluded object.
[0044] Step 206 : clustering the occluding object masks according to their respective depth information to obtain respective merged masks of at least two image depth categories.
[0045] Among them, the depth information is the vertical distance from the actual object or scene point corresponding to each pixel in the image to the camera lens plane. The depth information is usually represented in the form of a depth map, in which the grayscale value or color value of each pixel represents the depth value corresponding to the point. Depth information is the key to understanding the three-dimensional structure of the image. It enables the computer to perceive and understand the shape, size and position relationship of the object in the image. Depth information can be applied to computer vision tasks such as object detection, scene understanding, and three-dimensional reconstruction. Depth information can be determined based on various methods such as stereo vision, structured light, time of flight, and monocular depth estimation. For the depth information of the occluded object mask, it can be obtained according to the depth information of the pixels covered by the occluded object mask in the image to be processed. Specifically, the depth information of the occluded object mask can be averaged according to the depth information of each pixel covered by the occluded object mask, so as to obtain the depth information of the occluded object mask. The merged mask corresponds to the image depth category. Different image depth categories can correspond to different merged masks. The merged mask can be obtained by merging the occluded object masks belonging to the same image depth category.
[0046] Exemplarily, the server may determine the depth information of each occluding object mask, and the server may perform depth estimation for each occluding object mask to obtain the depth information of each occluding object mask. The server may cluster each occluding object mask based on the depth information of each occluding object mask to obtain at least two image depth categories by clustering, each image depth category may have a corresponding depth information range, and occluding object masks belonging to the same depth information range may be clustered into the same image depth category. Specifically, the server may cluster each occluding object mask based on at least one of various clustering algorithms such as K-Means Clustering, Hierarchical Clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Spectral Clustering, or Fuzzy C-Means (FCM) to obtain at least two image depth categories. For each image depth category, the occlusion object masks clustered to the image depth category can be merged to obtain a merged mask corresponding to the image depth category. In a specific application, the merged mask for each image depth category is obtained by merging the occlusion object masks belonging to the same image depth category. Specifically, at least part of the occlusion object masks belonging to the same image depth category can be merged to obtain a corresponding merged mask, that is, the merged mask of the same image depth category can include one or more. The image depth category includes at least two types, that is, merged masks including at least two depth information ranges. The merged masks corresponding to different types of image depth categories can be processed for image restoration in different orders.
[0047] For example, the occlusion object masks include occlusion object mask 1, occlusion object mask 2, occlusion object mask 3, occlusion object mask 4, occlusion object mask 5 and occlusion object mask 6, a total of six. The server can cluster each occlusion object mask according to the depth information of each occlusion object mask to obtain multiple image depth categories. For example, three image depth categories can be clustered. For image depth category 1, it includes {occlusion object mask 1, occlusion object mask 4, occlusion object mask 6}, image depth category 2 includes {occlusion object mask 2, occlusion object mask 3}, and image depth category 3 includes {occlusion object mask 5}. The server can merge the occlusion object masks included in each image depth category to obtain merged masks corresponding to various image depth categories. That is, merged mask 1 of image depth category 1 is obtained by merging occlusion object mask 1, occlusion object mask 4 and occlusion object mask 6; merged mask 2 of image depth category 2 is obtained by merging occlusion object mask 2 and occlusion object mask 3; merged mask 2 of image depth category 3 can be obtained based on occlusion object mask 5.
[0048] Step 208 , according to the respective merged masks of at least two image depth categories, and in the order of image depth from image foreground to image background, image occlusion restoration is performed on the images to be processed in sequence to obtain a target image after removing the occluded objects.
[0049] Among them, the foreground of an image refers to the most front-most and most eye-catching part of an image, which is usually directly opposite to the viewer's line of sight and is the first area to be noticed in the image. The foreground can be an object, a group of objects, a person, or a specific part of an image. The background of an image refers to the part of an image behind the foreground. It provides the environment and context of the image and helps the viewer understand the scene or situation in which the foreground elements are located. The background can be a natural environment (such as mountains and oceans), a man-made environment (such as cities and buildings), or an abstract pattern or color. In the same image, the vertical distance from the foreground of the image to the plane of the camera lens is shorter than the vertical distance from the background of the image to the plane of the camera lens, that is, the pixels in the background of the image have a larger depth value than the pixels in the foreground of the image. Image occlusion repair refers to a repair processing method for removing occluded objects in an image. The target image is the image obtained after removing the occluded objects in the image to be processed, that is, the target image is the image obtained after the occluded image is restored.
[0050] Optionally, for each merged mask of each image depth category, the server can perform image occlusion repair processing on the image to be processed in different orders. Specifically, the server can perform image occlusion repair on the image to be processed according to the merged masks of various image depth categories in the order of image depth from image foreground to image background in the image to be processed, so as to obtain a target image after removing the occluded object. In a specific application, the server can perform progressive image occlusion repair on the image to be processed based on the merged mask in the order of image foreground first and image background second, to obtain a target image.
[0051] For example, the image depth categories include image depth category 1, image depth category 2 and image depth category 3. For the three image depth categories, they are sorted in order of image depth from image foreground to image background as image depth category 3-image depth category 1-image depth category 2, that is, compared with image depth category 1 and image depth category 3, the image depth of image depth category 2 is the deepest and belongs to the image background; compared with image depth category 1 and image depth category 2, the image depth of image depth category 3 is the shallowest and belongs to the image foreground, and the image depth of image depth category 1 is between image depth category 3 and image depth category 2. At this time, the server can perform image occlusion repair processing on the processed image according to the order of image depth category 3-image depth category 1-image depth category 2, and according to the merged masks of various image depth categories. Specifically, the server can first perform image occlusion repair processing on the processed image according to merged mask 3 of image depth category 3, so as to remove the occluded objects covered in merged mask 3, thereby obtaining the image after the first repair; the server can further perform image occlusion repair processing on the image after the first repair according to merged mask 1 of image depth category 1, so as to remove the occluded objects covered in merged mask 1, thereby obtaining the image after the second repair; finally, the server can perform image occlusion repair processing on the image after the second repair according to merged mask 2 of image depth category 2, so as to remove the occluded objects covered in merged mask 2, thereby obtaining the target image after removing each occluded object.
[0052] In a specific application, such as Figure 3 As shown, for the acquired image to be processed, the server can determine each occluded object mask of the image to be processed, and the server clusters each occluded object mask according to the depth information of each occluded object mask to obtain a merged mask of at least two image depth categories. The server performs image occlusion repair on the image to be processed in sequence according to the image depth sequence from the image foreground to the image background and the merged masks of at least two image depth categories to obtain a target image after removing the occluded object.
[0053] In the above-mentioned occluded image restoration method, for an image to be processed including at least two occluded objects, clustering is performed according to the depth information corresponding to the occluded object mask used to characterize the occluded objects to obtain merged masks of at least two image depth categories, and image occlusion restoration is performed on the image to be processed in sequence according to the image depth sequence from the image foreground to the image background to obtain a target image after removing the occluded objects. By clustering the occluded object masks characterizing the occluded objects according to the depth information corresponding to the occluded object masks to obtain merged masks of at least two image depth categories, and performing image occlusion restoration in sequence according to the image depth sequence from the image foreground to the image background, confusion between occluded objects of different depths during the image occlusion restoration process can be reduced, and accurate occluded image restoration can be performed, thereby improving the image quality of the image obtained after removing the occluders.
[0054] In an exemplary embodiment, each occlusion object mask is clustered according to its respective depth information to obtain a merged mask for each of at least two image depth categories, including: clustering each occlusion object mask into at least two image depth categories according to its respective depth information, each image depth category corresponding to a different depth information range; and merging the occlusion object masks included in each of the at least two image depth categories to obtain a merged mask for each of the at least two image depth categories.
[0055] Among them, the depth information may include a depth value, the depth information range may include a depth value range, and the occluding object masks whose depth values belong to the same depth value range can be clustered into the same image depth category. For example, the depth information range of an image depth category A may be [m, n], that is, the occluding object masks whose depth values are greater than or equal to m and less than or equal to n can be clustered into the image depth category A.
[0056] Exemplarily, for each occluding object mask, the server can determine the depth information of each occluding object mask respectively, and the depth information of the occluding object mask can be obtained by fusion according to the depth information of the pixel points covered by the occluding object mask. The server can cluster the occluding object masks according to the depth information of each occluding object mask, that is, the server clusters each occluding object mask based on the dimension of depth information through a clustering algorithm to cluster each occluding object mask into at least two image depth categories. Each image depth category corresponds to a different depth information range, that is, the image depth category is divided based on the depth information range. For the image depth category obtained by clustering, it includes at least one occluding object mask, and the depth information of each occluding object mask clustered to the same image depth category belongs to the same depth information range, that is, the depth information range corresponding to the image depth category. For each image depth category, the server can merge the occluding object masks included in the image depth category to obtain the merged masks of various image depth categories.
[0057] In this embodiment, the server clusters the occluded object masks into at least two image depth categories based on the depth information of the occluded object masks, and merges the occluded object masks belonging to the same image depth category to obtain merged masks of various image depth categories, so that image occlusion repair is performed in sequence according to the merged masks of various image depth categories in order from image foreground to image background. This can reduce confusion between occluded objects of different depths during the image occlusion repair process, and can accurately restore the occluded image, thereby improving the image quality of the image obtained after removing the occluder.
[0058] In an exemplary embodiment, after clustering each occluding object mask according to their respective depth information to obtain respective merged masks of at least two image depth categories, the method further includes: when a target image depth category among the at least two image depth categories meets a mask splitting condition, splitting the merged mask of the target image depth category based on the depth information to obtain at least two split masks; and using the at least two split masks as the merged masks of the target image depth category, respectively.
[0059] The mask splitting condition is used to determine whether to split the merged mask, and can be pre-configured according to the density of the occluding object masks included in the same image depth category, where the density can include various parameters such as pixel coverage and the number of occluding objects. The target image depth category is the image depth category that meets the mask splitting condition among various image depth categories, that is, the target image depth category is the image depth category for which the corresponding merged mask needs to be split.
[0060] Exemplarily, the server may determine a pre-set mask splitting condition, and make a determination for each image depth category based on the mask splitting condition to determine whether it is necessary to split the merged mask corresponding to the image depth category. For example, the mask splitting condition includes a pixel coverage rate threshold. When the pixel coverage rate of merged mask A corresponding to image depth category A exceeds the pixel coverage rate threshold, it can be considered that image depth category A meets the mask splitting condition and belongs to the target image depth category that needs to be split for merged mask A; for another example, the mask splitting condition includes an occlusion object number threshold. When the number of occlusion objects corresponding to merged mask B of image depth category B exceeds the occlusion object number threshold, it can be considered that image depth category B meets the mask splitting condition and belongs to the target image depth category that needs to be split for merged mask B. The server may split the merged mask corresponding to the target image depth category based on the depth information of the target image depth category, such as splitting based on depth information, splitting based on semantic information, and splitting based on region size, etc., by analyzing occlusion objects of different depths, semantic categories, and region sizes in the merged mask, so as to split the merged mask corresponding to the target image depth category into at least two masks. The server may use both split masks as the merged mask of the target image depth category, that is, may split the large merged mask corresponding to the target image depth category into multiple relatively small merged masks.
[0061] In this embodiment, for the target image depth category that meets the mask splitting conditions, the server splits the merged mask of the target image depth category, and uses the split masks as the merged masks of the target image depth category, so that the large merged mask can be split into multiple relatively small merged masks, which is conducive to more refined division of the merged mask, can reduce confusion between occluded objects of different depths during the image occlusion repair process, and can accurately restore the occluded image, thereby improving the image quality of the image obtained after removing the occluder.
[0062] In an exemplary embodiment, Figure 4 As shown, the process of progressive image occlusion restoration, i.e., performing image occlusion restoration on the images to be processed in sequence according to the merged masks of at least two image depth categories and in the image depth order from the image foreground to the image background, to obtain the target image after removing the occluded object includes steps 402 to 410. Among them:
[0063] Step 402 : determining an inpainting processing order of respective merged masks of at least two image depth categories according to an image depth order from an image foreground to an image background.
[0064] The repair processing order is the processing order of performing image occlusion repair on the image to be processed based on the merged masks of various image depth categories. Optionally, for the merged masks of various image depth categories, the server can determine the repair processing order of each merged mask based on the depth information range of the image depth category and in the image depth order from the image foreground to the image background. In a specific application, the image depth category whose depth information range belongs to the image foreground can be repaired first.
[0065] For example, the image depth categories include image depth category 1, image depth category 2 and image depth category 3, among which the image depth category 1 corresponds to a depth information range of [d1, d2), the image depth category 2 corresponds to a depth information range of [d2, d3), and the image depth category 3 corresponds to a depth information range of [d3, d4], wherein d1<d2<d3<d4, then according to the image depth order from the image foreground to the image background, the repair processing order that can be determined is image depth category 1-image depth category 2-image depth category 3, that is, in the order of image depth category 1, image depth category 2, and image depth category 3, image occlusion repair is performed on the image to be processed according to the merged masks of each image depth category.
[0066] In some embodiments, when the image depth category includes multiple merge masks, they can be further sorted in the image depth order from the image foreground to the image background based on the depth information of each merge mask to obtain the restoration processing order between the merge masks.
[0067] Step 404 : determining a first merged mask from respective merged masks of at least two image depth categories according to the inpainting processing order.
[0068] The first merged mask is the merged mask corresponding to the first image occlusion repair for the image to be processed, that is, the first merged mask may be the first merged mask in the repair processing sequence. Exemplarily, the server may determine the first merged mask from each merged mask according to the repair processing sequence of each merged mask of each image depth category, that is, determine the first merged mask used when performing image occlusion repair for each image to be processed.
[0069] Step 406: Perform image occlusion restoration on the image to be processed based on the first merged mask using the image restoration model to obtain a restored image of the first restoration process.
[0070] The image restoration model may include a network model trained based on an artificial neural network algorithm or a deep learning algorithm. Specifically, the image restoration model may be trained based on at least one of a fully connected (FC) network, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep residual network (ResNet) or an attention mechanism (Attention Mechanism), Unet (U-shaped network), total variation (Total Variation), a generative adversarial network (GAN), a deep belief network (DBN), and a convolutional auto-encoder (CAE). The image restoration model may be pre-trained based on historical images, and image occlusion restoration processing is performed on the input image to remove the occluded object of the input image, and the image after the occluded object is removed is output.
[0071] Optionally, the server may obtain a pre-trained image restoration model, and perform image occlusion restoration on the image to be processed based on the first merged mask through the image restoration model to obtain a restored image of the first restoration process. In a specific application, the server may input the first merged mask and the image to be processed into the image restoration model, so that the image restoration model performs image occlusion restoration on the image to be processed based on the first merged mask, thereby obtaining a restored image of the first restoration process.
[0072] Step 408, starting from the second repair process in the repair process sequence, each repair process is performed on the repaired image obtained by the previous repair process through the image repair model based on the merged mask corresponding to this repair process, to obtain the repaired image of this repair process, until the repaired image obtained by the last repair process is obtained.
[0073] Exemplarily, starting from the second restoration, the server can use the restored image obtained by the previous restoration as the image for this restoration. Specifically, for the Nth (N≥2) restoration process, the server can determine the merged mask corresponding to the Nth restoration process based on the restoration process order, and obtain the restored image obtained by the N-1th restoration process. The server can use the restored image obtained by the N-1th restoration process as the image to be restored this time. The server can perform the Nth restoration process on the restored image obtained by the N-1th restoration process based on the merged mask corresponding to the Nth restoration process through the image restoration model, thereby obtaining the restored image of this restoration process, and perform the next restoration process based on the restored image of this restoration process, that is, perform the N+1th restoration process based on the restored image of the Nth restoration process. After the iteration completes all the restoration processes, that is, after the Nth restoration process is completed, the restored image of the Nth restoration process is obtained. The image restoration model can output the restored image obtained by the last restoration process as the restoration result of the image restoration model for the image to be processed.
[0074] Step 410, obtaining a target image after removing the occluded object based on the repaired image obtained by the last repair process.
[0075] Optionally, the server may obtain a target image after removing the occluded object based on the repaired image of the last repair process output by the image repair model. Specifically, the server may directly use the repaired image output by the image repair model as the target image after the repair is completed.
[0076] In this embodiment, the server iteratively performs multiple image occlusion repair processes on the image to be processed based on each merged mask through the image repair model in accordance with a determined repair processing order, thereby performing progressive image occlusion repair processing in the order of image depth from image foreground to image background, reducing confusion between occluded objects of different depths during the image occlusion repair process, and accurately restoring the occluded image, thereby improving the image quality of the image obtained after removing the occluder.
[0077] In an exemplary embodiment, the occluded image restoration method also includes: determining the pixel depth information of each pixel in the image to be processed; based on the pixel depth information of each pixel, determining the pixel depth information of each pixel covered by each occluded object mask; and obtaining the depth information of each occluded object mask according to the pixel depth information of the pixel covered by each occluded object mask.
[0078] Among them, the pixel depth information is the depth information of the pixel points in the image to be processed, and specifically may include the depth value of the pixel points. Optionally, for each pixel point in the image to be processed, the server may determine the pixel depth information of each pixel point respectively, and specifically may adopt a depth estimation algorithm to obtain the pixel depth information of each pixel point respectively. The server may determine the pixel depth information of each occluding object mask associated pixel point according to the pixel depth information of each pixel point respectively, wherein the associated pixel point is the pixel point covered by the occluding object mask in the image to be processed. In a specific implementation, for each occluding object mask, the server may determine the pixel points covered by each occluding object mask in the image to be processed respectively, and determine the pixel depth information of the pixel points covered by the occluding object mask. The server may determine the depth information of each occluding object mask according to the pixel depth information of the pixel points covered by each occluding object mask. In some embodiments, for each occluding object mask, the server may fuse the pixel depth information of the pixel points covered by the occluding object mask to obtain the depth information of the occluding object mask. For example, the server may average the pixel depth information of the pixels covered by the occluding object mask, and use the average value as the depth information of the occluding object mask. By traversing each occluding object mask, the server may determine the depth information of each occluding object mask.
[0079] In this embodiment, the server can obtain the corresponding depth information based on the pixel depth information of the pixel points covered by the occluding object mask, so as to accurately determine the depth information of the occluding object mask, so as to cluster the occluding object masks representing the occluding object according to the depth information corresponding to the occluding object mask to obtain merged masks of at least two image depth categories, and perform image occlusion repair in sequence according to the image depth order from the image foreground to the image background. This can reduce confusion between occluding objects of different depths during the image occlusion repair process, and can accurately restore the occluded image, thereby improving the image quality of the image obtained after removing the occluder.
[0080] In an exemplary embodiment, the depth information of each occluding object mask is obtained based on the pixel depth information of the pixel points covered by each occluding object mask, including: for each occluding object mask in each occluding object mask, the pixel points covered by the occluding object mask are filtered to obtain the filtered target pixel points; based on the pixel depth information of the target pixel points of each occluding object mask, the depth information of each occluding object mask is obtained respectively.
[0081] Among them, the target pixel point is the pixel point after screening the pixel points covered by the occluding object mask, which is used to determine the depth information of the occluding object mask. Optionally, for each occluding object mask, the server can determine the pixel points covered by the occluding object mask, and screen the pixel points covered by the occluding object mask. Specifically, the server can use various screening methods such as random sampling and Gaussian distribution screening to screen the target pixel points from the pixels covered by the occluding object mask. The server can determine the pixel depth information of the target pixel point, and obtain the depth information corresponding to the occluding object mask based on the pixel depth information of the target pixel point. By traversing each occluding object mask, the server can determine the depth information of each occluding object mask according to the pixel depth information of each target pixel point of each occluding object mask.
[0082] In this embodiment, the server screens the pixels covered by each occluding object mask, and then determines the depth information of each occluding object mask based on the pixel depth information of the screened target pixel points, which is helpful to eliminate pixel points that introduce depth errors, thereby improving the accuracy of the depth information.
[0083] The present application also provides an application scenario, in which the above-mentioned occluded image restoration method is applied. Specifically, the application of the occluded image restoration method in the application scenario is as follows:
[0084] Image restoration technology uses known parts in the image to infer and fill in damaged or occluded parts to achieve reasonable structure and texture. Although this technology has important value in the fields of cultural relics restoration, image editing and scene analysis, its application in urban scenes still faces challenges. Two-dimensional images are often affected by dynamic targets and environmental occlusion, which occludes semantic subjects and affects urban analysis and three-dimensional reconstruction. At present, image restoration is mostly based on datasets such as ImageNet, Celeb, and Place2, and training data is obtained by adding masks. However, there is a lack of datasets for outdoor scenes, and the image occlusion shapes are irregular, resulting in poor restoration effects.
[0085] Specifically, urban image restoration has important application value in modern society, especially in the fields of intelligent transportation systems, urban planning and disaster management. However, existing urban image restoration technologies have significant limitations in dealing with occlusion problems. Traditional methods usually rely on pixel information of two-dimensional images and cannot effectively deal with occlusion phenomena that frequently occur in complex urban environments. In recent years, with the development of deep learning and computer vision technology, image restoration methods based on depth information have gradually become a research hotspot. These methods use depth information to more accurately reconstruct occluded image areas. However, most of the current methods still have problems such as high computational complexity, slow processing speed, and poor adaptability to different occlusion scenarios.
[0086] Early image restoration methods include diffusion-based methods and block or example-based methods. Diffusion-based methods use neighboring original information to repair missing areas, but they are prone to introduce blur, especially when the missing area is large, the semantics of the restored image is relatively confusing. Block-based methods repair missing areas by block-level matching of low-level features, but they are not ideal when dealing with large areas of missing images in complex scenes. With the development of deep learning, although image restoration technology has made progress, its application in urban scenes still faces challenges. Existing methods are mostly based on mask processing of regular shapes, and there is a lack of image restoration research specifically for urban scenes. The complexity, diversity and large scale of urban scenes, as well as the irregularity of image occlusion, make it difficult for existing methods to achieve ideal results in urban parsing and 3D reconstruction tasks.
[0087] In general, it is difficult for traditional image occlusion restoration methods to effectively handle complex and irregular occlusions in urban scene images, resulting in unsatisfactory restoration effects. Secondly, traditional image occlusion restoration methods lack a unified solution to deal with different types of occlusions. Moreover, most existing deep learning methods are based on general data sets and are not optimized for the special needs of urban scenes. Based on this, the occluded image restoration method provided in this embodiment involves a progressive mask restoration technology for urban scene images. First, the semantic elements and occlusion effects in the image are analyzed, and the semantic masks are sorted; then, based on the depth information, the masks in the same depth range are merged and gradually removed to minimize confusion and maintain semantic consistency. Specifically, in the occluded image restoration method provided in this embodiment, the mask merging and progressive removal technology based on depth information is used to restore the urban occluded image. By obtaining the depth information, the masks in the same depth range are merged, and then the occluders of each depth layer are gradually removed, thereby reducing confusion and improving the accuracy and semantic consistency of image restoration. Through this progressive processing, the details and structure of the image can be better preserved, ensuring that the restored image has high-quality visual effects.
[0088] Specifically, the occluded image restoration method provided in this embodiment analyzes the semantic information in the image, determines the semantics of the core semantic entity and the mask, and uses a deep network for progressive restoration, thereby realizing the restoration of urban images under complex occlusions. Verification on the Cityscapes and ADE20K datasets shows that the occluded image restoration method provided in this embodiment can significantly improve the visual effect and signal-to-noise ratio of the image, and has a wide range of applicability and can be seamlessly combined with any image restoration model for application.
[0089] The occluded image restoration method provided in this embodiment involves urban occluded image restoration processing based on progressive processing. The key idea of the occluded image restoration method provided in this embodiment is to determine the semantics of the core semantic entities and masks by analyzing the semantic information in the image, and to use a deep network to sort the importance of different masks for progressive restoration. This method can not only effectively handle complex and irregular occlusions in urban scene images, but also significantly improve the visual effect of the image while ensuring the image signal-to-noise ratio. Specifically, the occluded image restoration method provided in this embodiment analyzes the semantic information in the image, identifies and classifies different semantic entities in the image, such as buildings, vehicles, pedestrians, etc., through semantic segmentation technology, and determines the depth range of each entity in combination with depth information, thereby accurately generating semantic masks. These masks can not only represent the categories of each semantic entity, but also reflect its position and depth information in the image. The occluded image restoration method provided in this embodiment can effectively identify and retain core semantic entities during the image restoration process, ensuring that the restored image has high-quality semantic consistency and structural integrity.
[0090] Specifically, the occluded image restoration method provided in this embodiment is aimed at image restoration processing of urban scenes. It adopts a progressive mask processing strategy to solve the common multi-object occlusion problem in urban scenes. This strategy extracts multiple object masks to be restored and combines the existing image restoration model to effectively remove the occluded objects in the scene to restore the integrity of the image. Figure 5As shown, the complete path from the original image input to the final restoration result in the occluded image restoration method provided by this embodiment is shown, including depth estimation, mask preprocessing, mask clustering and the final step-by-step image restoration process. Specifically, for the input image to be processed, the server can determine the corresponding depth image through the depth estimation model, and the depth estimation model can specifically include a monocular depth estimation model, such as a Metric 3D model. In addition, the depth estimation model can also include at least one of a DPT (Dense Prediction Transformer) model, a MiDaS (Mixed Depth and Scale) model, and an AdaBins (Adaptive Box) model. The depth image can include the depth information of each pixel in the input image, and specifically can include the depth value of each pixel. On the other hand, the server can determine the corresponding masks of each occluded object for the input image, and use the depth image to cluster the masks of each occluded object, such as using a fuzzy C-Means (FCM) clustering algorithm for clustering to obtain a merged mask of at least two image depth categories. The server can input the merged mask into the restoration model to perform image occlusion restoration, and the restoration model outputs the image after the occluded object is removed. The restoration model is used to perform image occlusion restoration processing on the image to be processed using the input merged mask, and output the image after the occluded object is removed. Specifically, the restoration model is an image restoration model, which can include at least one of the EdgeConnect model, the Image Transformer model, and the Partial Convolutional Neural Networks (PConv).
[0091] Specifically, in the data preprocessing stage, the original image needs to be collected first, and after the collection, the mask processing is performed and the occluded area in the image is marked. Then, the feature information of the image is extracted through the deep network, and the core semantic entities and their positions in the image are determined in combination with the semantic segmentation model. Specifically, the monocular depth estimation model can be applied to calculate the depth value of each pixel of the input image. For the mask of the object to be repaired, the depth information in the area is obtained through the corresponding pixel points, and the random sampling method is used to reduce the data error to improve the effectiveness of statistical inference. Furthermore, it is assumed that the depth value after random sampling conforms to the Gaussian distribution, and the mean and standard deviation are used as descriptions. By identifying and excluding points with significant depth differences, the depth estimation corresponding to the mask is ensured to be more accurate. The data error is reduced by random sampling, and the Gaussian distribution assumption is used to exclude outliers to improve the accuracy of depth estimation. The core goal of the data preprocessing stage is to provide accurate depth and semantic information for image restoration. The data preprocessing stage ensures the accuracy and consistency of the data through depth estimation and semantic segmentation, thus laying a solid foundation for subsequent image restoration. In this way, the image restoration task can more accurately restore the occluded image parts, ensuring that the restored image has high quality and semantic consistency.
[0092] In the image restoration task, the semantic mask of the input image is first extracted, and then the masks of the objects to be restored are clustered. In order to reduce the processing complexity and the time-consuming method, the Fuzzy C-Means (FCM) clustering algorithm is used to merge the masks. Then, the DIANA (Divisive Analysis Clustering) method is used to split the merged masks. It is assumed that the masks are located in the image plane after being clustered according to the depth range, and the masks in the same plane are used as the input of the image restoration model. The clustering results are split according to the standard of mask density, and finally a mask set is obtained. Among them, the standard of mask density can include but is not limited to pixel coverage, the number of target objects and the distribution of target objects in the image. The specific splitting methods include splitting based on depth information, splitting based on semantic information and splitting based on region size. By analyzing the target objects of different depths, semantic categories and region sizes in the mask, the mask is effectively separated and processed to improve the accuracy and effect of image restoration. Finally, the processing order of the image inpainting model is determined according to the clustering results, starting from the nearest mask and gradually processing to the farther masks, so as to improve the inpainting quality and efficiency and effectively avoid image errors caused by mask processing errors.
[0093] Specifically, in the mask cluster splitting process, as Figure 6As shown, the server can obtain an input mask (i.e., an occluded object mask), and the input mask body includes a triangular mask, a rectangular mask, and a circular mask, wherein the circular mask is located before the triangular mask and the rectangular mask, and the rectangular mask is located before the triangular mask, that is, the image depth of the triangular mask is higher than the image depth of the rectangular mask, and the image depth of the rectangular mask is higher than the image depth of the circular mask. The server can sort each input mask according to image depth, and in the obtained mask depth sorting result, from shallow to deep image depth, they are circular mask, rectangular mask, and triangular mask, that is, each input mask is sorted from near to far. The server can cluster each input mask based on the mask depth sorting result to obtain a mask depth clustering result, and specifically the circular mask, rectangular mask, and triangular mask are clustered into the same image depth category. At this time, it is determined that the image depth category meets the mask splitting condition, and the server can split it. The obtained mask splitting result includes two split masks, specifically the circular mask is one split mask, and the mask combined by the rectangular mask and the triangle mask is another split mask. Both of the two split masks can be used as the merged mask of the image depth category. The server can perform image occlusion repair based on the respective merged masks of various image depth categories to obtain an image after removing the occluded object from the input image.
[0094] In view of the complex and irregular occlusion problem in urban scene images, the occluded image restoration method provided in this embodiment adopts a multi-stage processing method to gradually reduce the impact of the occluded area, making the image restoration more natural and having a high signal-to-noise ratio. Figure 7 As shown in the urban scene occlusion removal comparison diagram, the occluded image restoration method provided by this embodiment performs image occlusion repair based on four groups of images to be processed, A, B, C, and D, to obtain respective target images, in which various occluded objects included in the corresponding images to be processed are removed, thereby solving the occlusion occurring in the urban scene images.
[0095] The occluded image restoration method provided in this embodiment involves progressive image repair processing. Through a multi-stage progressive processing strategy, the covered area is gradually repaired to ensure process consistency and efficiency, and finally achieve high-quality image restoration. The occluded image restoration method provided in this embodiment uses the depth information of the image to guide the repair order, and repairs from the foreground to the background layer by layer, so that the repair result of each stage can provide more accurate semantic and structural information for the next stage. The occluded image restoration method provided in this embodiment significantly improves the image repair quality by relying on the progressive image repair method of the depth order, and solves the problem that the traditional method is prone to blur or semantic confusion in large-area occluded areas. The method repairs the missing area layer by layer through multi-stage depth order processing, ensuring that the repair result of each stage is more accurate, thereby improving the quality of the final image. At the same time, by guiding the repair order, avoiding global search, improving processing efficiency, and reducing the consumption of computing resources. This method is not only applicable to occluded areas of regular shapes, but can also effectively handle occluded areas of irregular shapes, especially occlusion problems in complex urban scenes, and achieves a wider range of applications. The occluded image restoration method provided in this embodiment can significantly improve the restoration quality and efficiency, reduce the computational cost, and thus enhance the applicability and scalability of the method.
[0096] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0097] Based on the same inventive concept, the embodiment of the present application also provides an occluded image restoration device for implementing the occluded image restoration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more occluded image restoration device embodiments provided below can refer to the limitations of the occluded image restoration method above, and will not be repeated here.
[0098] In an exemplary embodiment, Figure 8As shown, an occluded image restoration device 800 is provided, comprising: a to-be-processed image acquisition module 802, an occluded object mask determination module 804, a merged mask acquisition module 806 and an image occlusion repair module 808, wherein:
[0099] The to-be-processed image acquisition module 802 is used to acquire the to-be-processed image, wherein the to-be-processed image includes at least two blocking objects that block the main object;
[0100] The occluded object mask determination module 804 is used to determine each occluded object mask of the image to be processed, and the occluded object mask is used to characterize the distribution area of the occluded object in the image to be processed;
[0101] A merged mask obtaining module 806 is used to cluster the occluding object masks according to the depth information of each occluding object mask to obtain merged masks of at least two image depth categories;
[0102] The image occlusion restoration module 808 is used to perform image occlusion restoration on the images to be processed in sequence according to the merged masks of at least two image depth categories and in the order of image depth from image foreground to image background, so as to obtain a target image after removing the occluded objects.
[0103] In one embodiment, the merged mask acquisition module 806 is further used to cluster each occlusion object mask into at least two image depth categories according to the depth information of each occlusion object mask, each image depth category corresponds to a different depth information range; and merge the occlusion object masks included in each of the at least two image depth categories to obtain merged masks for each of the at least two image depth categories.
[0104] In one embodiment, the merged mask acquisition module 806 is also used to split the merged mask of the target image depth category based on the depth information to obtain at least two split masks when the target image depth category in at least two image depth categories meets the mask splitting condition; and use the at least two split masks as the merged masks of the target image depth category respectively.
[0105] In one embodiment, the image occlusion repair module 808 is also used to determine the repair processing order of the merged masks of at least two image depth categories according to the image depth sequence from the image foreground to the image background; determine the first merged mask from the merged masks of at least two image depth categories according to the repair processing order; perform image occlusion repair on the image to be processed based on the first merged mask through the image repair model to obtain a repaired image of the first repair processing; from the second repair processing in the repair processing sequence onwards, perform this repair processing on the repaired image obtained by the previous repair processing based on the merged mask corresponding to this repair processing through the image repair model to obtain the repaired image of this repair processing, until the repaired image obtained by the last repair processing is obtained; according to the repaired image obtained by the last repair processing, obtain the target image after removing the occluded object.
[0106] In one embodiment, it also includes a depth information determination module, which is used to determine the pixel depth information of each pixel in the image to be processed; based on the pixel depth information of each pixel, the pixel depth information of each pixel covered by each occluding object mask is determined respectively; according to the pixel depth information of each pixel covered by each occluding object mask, the depth information of each occluding object mask is obtained.
[0107] In one embodiment, the depth information determination module is also used to filter the pixel points covered by each occluding object mask in each occluding object mask to obtain the filtered target pixel points; based on the pixel depth information of the target pixel points of each occluding object mask, the depth information of each occluding object mask is obtained respectively.
[0108] Each module in the above-mentioned occluded image restoration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0109] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the occluded image restoration method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for restoring an occluded image is implemented.
[0110] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0111] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0112] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0113] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0115] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0117] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for restoring an occluded image. It is characterized in that The method comprises: Acquire an image to be processed, wherein the image to be processed includes at least two blocking objects that block the main object; Determine each occluding object mask of the image to be processed, wherein the occluding object mask is used to characterize a distribution area of the occluding object in the image to be processed; Clustering the occluding object masks according to their respective depth information to obtain merged masks of at least two image depth categories; According to the respective merged masks of the at least two image depth categories, image occlusion restoration is performed on the images to be processed in sequence according to the image depth order from the image foreground to the image background, so as to obtain a target image after removing the occluded object.
2. The method according to claim 1, It is characterized in that The step of clustering the occluding object masks according to the depth information of the occluding object masks to obtain respective merged masks of at least two image depth categories includes: Clustering the occluding object masks into at least two image depth categories according to the depth information of each occluding object mask, each image depth category corresponding to a different depth information range; The occluded object masks respectively included in the at least two image depth categories are merged to obtain merged masks respectively included in the at least two image depth categories.
3. The method according to claim 1, It is characterized in that After clustering the occluding object masks according to the depth information of the occluding object masks to obtain respective merged masks of at least two image depth categories, the method further includes: When a target image depth category among the at least two image depth categories meets a mask splitting condition, splitting a merged mask of the target image depth category based on depth information to obtain at least two split masks; The at least two split masks are respectively used as merged masks of the depth category of the target image.
4. The method according to claim 1, It is characterized in that The method of performing image occlusion repair on the images to be processed in sequence according to the merged masks of the at least two image depth categories and in the order of image depth from the image foreground to the image background to obtain a target image after removing the occluded object comprises: Determining, according to the image depth order from the image foreground to the image background, an inpainting processing order of the merged masks of the at least two image depth categories; Determining a first merged mask from respective merged masks of the at least two image depth categories according to the inpainting processing order; Using the image restoration model, performing image occlusion restoration on the image to be processed based on the first merged mask to obtain a restored image of the first restoration process; In each restoration process starting from the second restoration process in the restoration process sequence, the restoration process is performed on the restoration image obtained by the previous restoration process based on the merge mask corresponding to the restoration process, through the image restoration model, to obtain the restoration image of the restoration process, until the restoration image obtained by the last restoration process is obtained; According to the repaired image obtained by the last repair process, a target image after removing the occluded object is obtained.
5. The method according to any one of claims 1 to 4, It is characterized in that The method further comprises: Determine pixel depth information of each pixel in the image to be processed; Based on the pixel depth information of each pixel point, respectively determine the pixel depth information of each pixel point covered by each mask of the occluded object; Depth information of each of the occluding object masks is obtained according to pixel depth information of pixel points covered by each of the occluding object masks.
6. The method according to claim 5, It is characterized in that The obtaining the depth information of each of the occluding object masks according to the pixel depth information of the pixel points covered by each of the occluding object masks comprises: For each of the occluding object masks, filter the pixel points covered by the occluding object mask to obtain the filtered target pixel points; Based on the pixel depth information of the target pixel points of each of the occluding object masks, the depth information of each of the occluding object masks is obtained respectively.
7. An occluded image restoration device, It is characterized in that The device comprises: A module for acquiring an image to be processed, used for acquiring an image to be processed, wherein the image to be processed includes at least two blocking objects that block the main object; An occluded object mask determination module, used to determine each occluded object mask of the image to be processed, wherein the occluded object mask is used to characterize a distribution area of the occluded object in the image to be processed; A merged mask obtaining module, configured to cluster the occluding object masks according to the depth information of the occluding object masks, and obtain merged masks of at least two image depth categories; The image occlusion restoration module is used to perform image occlusion restoration on the images to be processed in sequence according to the merged masks of the at least two image depth categories and in the order of image depth from image foreground to image background, so as to obtain a target image after removing the occluded object.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
