Image processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide an image processing method, device, electronic device and storage medium. Among them, the method may include performing a cutout process on the original image to be processed to obtain a cutout result, and the cutout result includes a reserved area map containing the reserved area in the original image, and a transparency map corresponding to the original image. According to the difference between the pixels in the reserved area map and the pixels in the material image containing the target area, the color difference between the reserved area and the target area is determined. According to the color difference, the pixels in the reserved area map are adjusted in hue to obtain an adjusted reserved area map that matches the hue of the target area. Based on the transparency map, the adjusted reserved area map is fused with the material image to obtain a target image in which the non-reserved area is replaced with the target area.

Description

Technical Field

[0001] The present application relates to computer vision technology, and specifically to image processing methods, devices, electronic devices and storage media. Background Art

[0002] As a part of basic image editing technology, region replacement is widely used in various image editing software, camera backend algorithms and other scenarios. At present, the industry generally uses segmentation models to perform semantic segmentation on the original image to be processed to obtain a rough mask result of the replaced area. Then, the mask result is fused with the image containing the target area to replace the replaced area with the target area.

[0003] However, in the related art, there is an obvious difference in color tone between the target area and the original image, and direct replacement often results in obvious inconsistent picture tones, resulting in poor area replacement effect. Summary of the invention

[0004] In view of this, the embodiments of the present application at least provide an image processing method, device, electronic device and storage medium.

[0005] The present application provides an image processing method, the method comprising: performing a matting process on an original image to be processed to obtain a matting result, the matting result comprising a reserved area map including a reserved area in the original image, and a transparency map corresponding to the original image; the reserved area is a foreground or background in the original image; the numerical value of a pixel in the transparency map indicates the transparency of the pixel; according to the difference between the pixels in the reserved area map and the pixels in a material image including the target area, determining the color difference between the reserved area and the target area; the target area is used to replace the non-reserved area in the original image; adjusting the hue of the pixels in the reserved area map according to the color difference to obtain an adjusted reserved area map matching the hue of the target area; based on the transparency map, performing image fusion on the adjusted reserved area map and the material image to obtain a target image in which the non-reserved area is replaced by the target area.

[0006] In some embodiments, determining the color difference between the reserved area and the target area based on the difference between pixels in the reserved area map and pixels in the material image including the target area includes: sampling the pixels in the reserved area map and the pixels in the material image respectively to obtain reserved area sampling points and material sampling points; and determining the color difference between the reserved area and the target area based on the difference between the pixel values ​​of the reserved area sampling points and the pixel values ​​of the material sampling points.

[0007] In some embodiments, the method further includes: sampling pixels in the transparency map to obtain transparency sampling points; determining the color difference between the reserved area and the target area based on the difference between the pixel values ​​of the reserved area sampling points and the pixel values ​​of the material sampling points, including: determining a first pixel mean of the reserved area sampling points based on the pixel values ​​of the reserved area sampling points and the transparency values ​​of the transparency sampling points; determining a second pixel mean of the material sampling points based on the pixel values ​​of the material sampling points; and determining the color difference between the reserved area and the target area according to the difference between the first pixel mean and the second pixel mean.

[0008] In some embodiments, the hue adjustment of pixels in the reserved area map according to the color difference to obtain an adjusted reserved area map that matches the hue of the target area includes: adjusting the pixel values ​​of the pixels in the reserved area map based on the color difference to obtain a preliminary adjusted reserved area map; the pixels in the preliminary adjusted reserved area map incorporate the color difference; and adjusting the pixel values ​​of the pixels in the preliminary adjusted reserved area map based on the difference between the pixel mean value of the pixels in the preliminary adjusted reserved area map and the pixel mean value of the pixels in the reserved area map before the preliminary adjustment to obtain an adjusted reserved area map that matches the hue of the target area.

[0009] In some embodiments, based on the transparency map, the adjusted reserved area map is fused with the material image to obtain a target image in which the non-reserved area is replaced with the target area, including: obtaining a first result by fusing the transparency map with the reserved area map; obtaining a second result by fusing the material image with a reverse transparency map corresponding to the transparency map; and obtaining the target image by fusing the first result with the second result.

[0010] In some embodiments, the matting process is performed on the original image to be processed to obtain the matting result, including: obtaining a tripartite map corresponding to the original image to be processed; the numerical value corresponding to each pixel in the tripartite map represents the probability that the pixel belongs to any of the retained area, non-retained area or to-be-determined area in the original image; and matting process is performed based on the tripartite map and the original image to obtain the matting result.

[0011] In some embodiments, obtaining a tripartite map corresponding to the original image to be processed includes: performing semantic segmentation processing on the original image to be processed to obtain a semantic probability map of the original image, wherein the numerical value of a pixel in the semantic probability map indicates a first probability that the pixel belongs to the reserved area in the original image; and performing probability conversion processing based on the semantic probability map to obtain a tripartite map corresponding to the original image.

[0012] In some embodiments, the performing semantic segmentation processing on the original image to be processed to obtain a semantic probability map corresponding to the original image includes: performing semantic segmentation processing on the original image through a semantic segmentation network to obtain the semantic probability map output by the semantic segmentation network; the performing matting processing based on the tripartite map and the original image includes: performing matting processing based on the tripartite map and the original image through a matting network.

[0013] In some embodiments, the probability conversion processing is performed based on the semantic probability map to obtain a tripartite map corresponding to the original image, including: converting a first probability of each pixel in the semantic probability map through probability conversion to obtain a second probability that the pixel belongs to the area to be determined in the tripartite map; and generating the tripartite map according to the first probability and the second probability of the pixel.

[0014] In some embodiments, the higher the probability that the first probability represents that the pixel belongs to the foreground or the background, the lower the probability that the second probability obtained by probability conversion represents that the pixel belongs to the to-be-determined area in the tri-map; generating the tri-map according to the first probability and the second probability of the pixel includes: for each pixel in the original image, performing probability fusion according to the first probability and the second probability corresponding to the pixel, and determining the value corresponding to the pixel in the tri-map.

[0015] In some embodiments, performing matting processing according to the trisection map and the original image to obtain the matting result includes: performing matting processing according to the trisection map and the original image to obtain a retained area residual and an initial transparency map of the original image; the values ​​of pixels in the initial transparency map indicate the transparency of the pixels; obtaining the retained area map based on the original image and the retained area residual; and adjusting the values ​​of pixels in the initial transparency map according to the trisection map to obtain a transparency map corresponding to the original image.

[0016] In some embodiments, before semantic segmentation is performed on the original image to be processed, the method further includes: scaling the original image; obtaining the retained area map based on the original image and the retained area residual, including: enlarging the retained area residual obtained by matting the tri-image and the original image to a scale before the scaling; and obtaining the retained area map based on the enlarged retained area residual and the original image.

[0017] In some embodiments, the non-retained area includes a sky area in the original image; and the target area includes a sky area in the material image.

[0018] The present application proposes an image processing device, which includes: a cutout module, which is used to perform cutout processing on an original image to be processed to obtain a cutout result, wherein the cutout result includes a reserved area map containing a reserved area in the original image, and a transparency map corresponding to the original image; the reserved area is the foreground or background in the original image; the numerical value of the pixel in the transparency map indicates the transparency of the pixel; a determination module, which is used to determine the color difference between the reserved area and the target area according to the difference between the pixels in the reserved area map and the pixels in the material image containing the target area; the target area is used to replace the non-reserved area in the original image; an adjustment module, which is used to adjust the hue of the pixels in the reserved area map according to the color difference, to obtain an adjusted reserved area map that matches the hue of the target area; and a fusion module, which is used to fuse the adjusted reserved area map with the material image based on the transparency map, to obtain a target image in which the non-reserved area is replaced by the target area.

[0019] The present application proposes an electronic device, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor implements the image processing method shown in any of the aforementioned embodiments by running the executable instructions.

[0020] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to enable a processor to execute the image processing method shown in any of the aforementioned embodiments.

[0021] In the technical solutions disclosed in the image processing method, device, electronic device and storage medium provided in the embodiments of the present application, on the one hand, the color difference between the reserved area and the target area can be determined based on the pixel values ​​of the pixels in the reserved area obtained by cutout processing of the original image and the pixel values ​​of the pixels in the target area, and then the hue of the pixels in the reserved area can be adjusted according to the color difference to unify the hue of the pixels in the reserved area and the pixels in the target area, so that in the process of area replacement, it can be ensured that the hue of the target area matches the non-reserved area in the original image, thereby improving the area replacement effect.

[0022] On the other hand, the three-part image can be used for cutout, so that the detailed information of the junction between the area and the non-retained area can be well preserved. When replacing the area, it is helpful to improve the connection effect between the retained area and the non-retained area.

[0023] On the other hand, the matting network can also be processed by channel compression and the original image can be scaled so that the running time and memory consumption of the matting process are within the burden range of the mobile terminal, thereby eliminating the need for regional replacement through the server and ensuring data security and privacy.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in one or more embodiments of the present application or related technologies, the drawings required for use in the embodiments or related technology descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 A flowchart of an image processing method shown in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of a method for determining color difference shown in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a method for determining color difference shown in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of a flow chart of a color tone adjustment method shown in an embodiment of the present application;

[0030] Figure 5A schematic diagram of a process of a cutout method shown in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of a process for obtaining a trisection image according to an embodiment of the present application;

[0032] Figure 7a A schematic diagram of a character image shown in an embodiment of the present application;

[0033] Figure 7b A schematic diagram of a semantic probability graph shown in an embodiment of the present application;

[0034] Figure 7c A schematic diagram of a three-part diagram shown in an embodiment of the present application;

[0035] Figure 7d A schematic diagram of a transparency diagram shown in an embodiment of the present application;

[0036] Figure 7e A schematic diagram of a foreground image shown in an embodiment of the present application;

[0037] Figure 7f A schematic diagram of a target image shown in an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a region replacement process shown in an embodiment of the present application;

[0039] Fig. 9 Based on Figure 8 Schematic diagram of the regional replacement process;

[0040] Fig.10 A flowchart of a network training method shown in an embodiment of the present application;

[0041] Fig.11 A structural schematic diagram of an image processing method shown in an embodiment of the present application;

[0042] Fig.12 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The exemplary embodiments will be described in detail below, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0044] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. It should also be understood that the word "if" used in this article, depending on the context, can be interpreted as "at the time of" or "when" or "in response to determining".

[0045] The present application relates to the field of augmented reality, by acquiring the image information of the target object in the real environment, and then using various vision-related algorithms to detect or identify the relevant features, states and attributes of the target object, so as to obtain an AR effect that combines virtuality and reality that matches the specific application. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and posture or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstruction, virtual effect superposition and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, limb beautification, special effects display, virtual model display and other interactive scenes. The relevant features, states and attributes of the target object can be detected or identified through a convolutional neural network. The aforementioned convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0046] In view of this, the present application proposes an image processing method. When replacing a region, the method can ensure that the color tone of the target region matches the reserved region in the original image, thereby improving the region replacement effect.

[0047] See also Figure 1 , Figure 1 A flowchart of an image processing method shown in an embodiment of the present application.

[0048] Figure 1 The processing method shown can be applied to an electronic device. The electronic device can execute the method by carrying software logic corresponding to the processing method. The type of the electronic device can be a laptop, a computer, a mobile phone, a PDA (Personal Digital Assistant, PDA), etc. The type of the electronic device is not particularly limited in this application. The electronic device can also be a client device and / or a server device, which is not particularly limited here.

[0049] like Figure 1 As shown, the image processing method may include S102-S108. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0050] S102, performing a cutout process on the original image to be processed to obtain a cutout result, wherein the cutout result includes a retained area map of the retained area in the original image, and a transparency map corresponding to the original image; the retained area is the foreground or background in the original image; and the value of a pixel in the transparency map indicates the transparency of the pixel.

[0051] The original image includes an image to be replaced with a region. The original image may include a reserved region and a non-reserved region. In a region replacement scenario, the non-reserved region is generally used as a replaced region and is replaced by other materials. The reserved region and the non-reserved region can be distinguished by image processing technology.

[0052] The reserved area refers to the area that is reserved and not replaced during the process of replacing the area of ​​the image. For example, in the scene of replacing the background area, the foreground area is the reserved area. For example, if the background area (such as the sky area) in the character image needs to be replaced, the foreground area containing the character can be used as the reserved area. In the scene of replacing the foreground area, the background area is the reserved area.

[0053] The reserved area map may include a reserved area cut out from the original image. The size of the reserved area map is the same as the size of the original image. The area in the reserved area map other than the reserved area may be filled with pixels of a preset pixel value. For example, the preset pixel value may be 0, 1, etc.

[0054] The transparency map is a map used to distinguish between the reserved area and the non-reserved area by different transparency values. The numerical value of the pixel in the transparency map indicates the transparency of the corresponding pixel. In some embodiments, the transparency value of the pixel in the transparency map belonging to the reserved area is a first value, and the transparency value of the pixel in the non-reserved area is a second value.

[0055] In different scenarios, the first value and the second value may vary.

[0056] For example, in a scene where the region is replaced and the original region is not retained, the first value of the transparency map may be 1, indicating that the pixels of the retained region are non-transparent, and the second value of the transparency map may be 0, indicating that the pixels of the non-retained region are transparent. The non-retained region is replaced by the transparency, and the original non-retained region is not retained at all. For another example, in a scene where the region is replaced and the original retained region needs to be blurred, the first value of the transparency map may be 1, indicating that the pixels of the retained region are non-transparent, and the second value of the transparency map may be 0.3, indicating that the pixels of the non-retained region are semi-transparent. The non-retained region is replaced by the transparency, and the original retained region can be blurred. In some embodiments, in S102, a tripartite map corresponding to the original image to be processed may be obtained; the numerical value corresponding to each pixel in the tripartite map indicates the probability that the pixel belongs to any one of the retained region, the non-retained region or the region to be determined in the original image; then, the original image may be matted according to the tripartite map to obtain the retained region map and the transparency map.

[0057] The triplicate map has the characteristic of distinguishing the foreground, the background, and the junction area between the foreground and the background in the image. That is, no matter whether the reserved area is the foreground or the background in the original image, the triplicate map is used to distinguish the reserved area, the non-reserved area, and the undetermined area between the reserved area and the non-reserved area in the original image, so as to save the detailed information of the junction position between the reserved area and the non-reserved area.

[0058] In some implementations, a pre-trained cutout network may be used for cutout processing. The cutout network is pre-trained in a supervised manner using training samples annotated with transparency information and reserved area information. The original image is input into the cutout network to obtain the reserved area and transparency map.

[0059] S104, determining a color difference between the reserved area and the target area according to a difference between pixels in the reserved area map and pixels in the material image including the target area; the target area is used to replace the non-reserved area in the original image.

[0060] The material images are generally some images acquired in advance, and these images contain replacement materials for replacing non-reserved areas. The area occupied by these replacement materials in the material images can be called the target area. For example, in a sky replacement scene, the material image can contain some sky materials, and these sky materials can be used to replace the sky in the original image (that is, the non-reserved area in the original image).

[0061] The color difference refers to the difference in pixel values ​​between pixels in the reserved area and pixels in the target area.

[0062] The pixel value of a pixel may indicate the color value of the pixel. Exemplarily, the color difference between the reserved area and the target area may be obtained by calculating the average difference between the pixel values ​​of the pixels in the reserved area map and the pixel values ​​of the pixels in the material image.

[0063] In some implementations, sampling can be performed in the reserved area and the target area, and the color difference can be determined by the pixel values ​​of the sampling points, thereby reducing the amount of calculation for determining the color difference and improving the efficiency of area replacement.

[0064] See also Figure 2 , Figure 2 A schematic flow chart of a method for determining color difference shown in an embodiment of the present application. Figure 2 The steps shown are for explanation of S104. Figure 2 As shown, the determination method may include S202-S204. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0065] S202, sampling pixels in the reserved area map and pixels in the material image respectively to obtain reserved area sampling points and material sampling points.

[0066] In some embodiments, the step size (step) may be preset. For example, the result of dividing the short side of the original image by a preset value (e.g., 10, 20, 30, etc.) may be determined as the step. For the reserved area map, sampling may be performed in a preset order (e.g., from left to right, from top to bottom) and a set step size to obtain some reserved area sampling points; and for the material image, sampling may be performed in a preset order (e.g., from left to right, from top to bottom) and a set step size to obtain some material sampling points.

[0067] S204: Determine a color difference between the reserved area and the target area based on a difference between a pixel value of a sampling point in the reserved area and a pixel value of a sampling point in the material.

[0068] In some implementations, the pixel mean or pixel median value of the reserved area sampling points may be determined based on the pixel values ​​of the reserved area sampling points, and the pixel mean or pixel median value of the material sampling points may be determined based on the pixel values ​​of the material sampling points. Then, the color difference is determined based on the difference between the two pixel mean values ​​or the two pixel median values.

[0069] Therefore, the pixel value of the sampling point can be represented by the pixel mean value or the pixel median value of the sampling point, which can simplify the calculation.

[0070] In some embodiments, when calculating the pixel mean of the sampling points in the reserved area, the transparency of the sampling points in the reserved area can be taken into consideration, so that the determined pixel mean is more accurate, which helps to accurately determine the color difference between the reserved area and the target area, thereby helping to improve the color adjustment effect.

[0071] In this implementation, the pixels in the transparency map may be sampled first to obtain transparency sampling points.

[0072] Specifically, the step disclosed in the aforementioned S202 may be used for sampling to obtain some transparency sampling points.

[0073] See also Figure 3 , Figure 3 A schematic flow chart of a method for determining color difference shown in an embodiment of the present application. Figure 3 The steps shown are supplementary explanations for S204. Figure 3 As shown, the determination method may include S302-S306. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0074] S302: Determine a first pixel mean value of the sampling points in the reserved area based on the pixel values ​​of the sampling points in the reserved area and the transparency values ​​of the transparency sampling points.

[0075] In order to distinguish the pixel means, the present application refers to the pixel means of the sampling points in the reserved area as the first pixel means.

[0076] The embodiment of the present application does not limit the specific formula for determining the first pixel mean value, and the following is only an example:

[0077]

[0078] As in the above formula (1), fg_mean indicates the first pixel mean. FG1 refers to the pixel value of the reserved area sampling point. Alpha1 refers to the transparency value of the transparency sampling point. Formula 1 can be combined with the transparency of the sampling point to obtain an accurate first pixel mean.

[0079] S304: Determine a second pixel mean value of the material sampling point based on the pixel value of the material sampling point.

[0080] In this application, the pixel mean of the material sampling points is referred to as the second pixel mean.

[0081] The second pixel mean bg_mean can be obtained by averaging BG1 using the average calculation formula, where BG1 is the pixel value of the material sampling point.

[0082] S306: Determine a color difference between the reserved area and the target area according to a difference between the first pixel mean and the second pixel mean.

[0083] The pixel value of a pixel can indicate the color information of the pixel. The color difference of the pixel can be determined by the difference in pixel value. The embodiment of the present application does not limit the specific formula for determining the color difference, and the following is only an example:

[0084] diff=bg_mean-fg_mean……………………(2)

[0085] As in the above formula (2), diff indicates the color difference, bg_mean indicates the second pixel mean, and fg_mean indicates the first pixel mean.

[0086] Therefore, according to the steps recorded in S302-S304, on the one hand, the pixel value of the sampling point can be represented by the pixel mean value or the pixel median value of the sampling point, which can simplify the calculation. On the other hand, when calculating the pixel mean value of the sampling point in the reserved area, the transparency of the sampling point is combined, so that the determined pixel mean value is more accurate, which helps to accurately determine the color difference between the reserved area and the target area, and further helps to improve the color adjustment effect.

[0087] S106, adjusting the hue of pixels in the reserved region map according to the color difference to obtain an adjusted reserved region map that matches the hue of the target region.

[0088] Hue refers to the overall tendency of the color of an image. Although an image includes multiple colors, it has an overall color tendency. For example, an image may be blue or red, warm or cold, etc. This color tendency is the hue of the image. That is, the hue of the image can be indicated by the pixel value (color value) of the pixel in the image, and the hue adjustment can be completed by adjusting the pixel value of the image pixel.

[0089] The hue adjustment may be to make the color values ​​of the pixels in the reserved area map closer to the color values ​​of the pixels in the target area.

[0090] The hue matching of two images means that the difference between the color values ​​of pixels in the two images is less than a preset color threshold (empirical threshold), that is, the color values ​​of pixels in the two images are relatively close, presenting an effect of roughly the same hue.

[0091] In S106, the color difference may be merged into the pixel values ​​of the pixels in the reserved region map to achieve an effect of matching the hue of the reserved region map with the hue of the target region, thereby completing the hue adjustment.

[0092] The embodiment of the present application does not limit the specific formula of color fusion, and the following is only an example:

[0093] new_FG=q*diff+FG…………(3)

[0094] As shown in formula (3), FG refers to the pixel value of the pixel in the reserved area map. new_FG refers to the pixel value of the pixel in the adjusted reserved area map. q is a preset adjustment coefficient. This q is a numerical value pre-set according to business needs. diff indicates the color difference between the target area and the reserved area. Through formula (3), the hue of the pixels in the reserved area map can be adjusted based on the color difference to obtain an adjusted reserved area map that matches the hue of the target area, thereby facilitating the improvement of the area replacement effect.

[0095] In some implementations, the color difference between the adjusted reserved area map and the reserved area before adjustment can be integrated, and the pixels in the adjusted reserved area map can be adjusted again to reduce excessive or insufficient pixel values ​​of the adjusted image and improve the tone adjustment effect.

[0096] See also Figure 4 , Figure 4 A flowchart of a color tone adjustment method shown in an embodiment of the present application. Figure 4 The steps shown are supplementary explanations for S106. Figure 4 As shown, the method may include S402-S404. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0097] S402, adjusting pixel values ​​of pixels in the reserved region map based on the color difference to obtain a preliminarily adjusted reserved region map; pixels in the preliminarily adjusted reserved region map incorporate the color difference.

[0098] In order to distinguish the adjustment area, in this application, the area obtained after adjusting the pixels in the reserved area is referred to as the reserved area map after preliminary adjustment.

[0099] In some implementations, the aforementioned formula (3) may be used to obtain the initially adjusted reserved area map.

[0100] S404, adjusting the pixel values ​​of the pixels in the retained area map after the preliminary adjustment based on the difference between the pixel mean value of the pixels in the retained area map after the preliminary adjustment and the pixel mean value of the pixels in the retained area map before the preliminary adjustment, to obtain an adjusted retained area map that matches the color tone of the target area.

[0101] The difference between the pixel means may indicate a color difference between the reserved area map after the preliminary adjustment and the reserved area map before the preliminary adjustment.

[0102] In S404, the difference in pixel values ​​in the reserved area map before and after the preliminary adjustment can be determined based on the pixel mean value of the pixels in the reserved area map before and after the preliminary adjustment, and then the difference in pixel values ​​is fused with the pixels in the reserved area map after the preliminary adjustment to obtain an adjusted reserved area map.

[0103] The embodiment of the present application does not limit the specific formula of color fusion, and the following is only an example:

[0104] new_FG=new_FG+(mean(FG)-mean(new_FG))……………………(4)

[0105] As shown in the above formula (4), new_FG on the left side of the equal sign is the pixel value of the pixel in the retained area map after the final adjustment. The new_FG on the right side of the equal sign is the pixel value of the pixel in the retained area map after the preliminary adjustment obtained by the above formula (3). mean() is a function for finding the average value. The color difference between the retained area map after the preliminary adjustment and the retained area map before the preliminary adjustment can be obtained by mean(FG)-mean(new_FG). Through S402-S404, the color tone of the retained area map can be preliminarily adjusted using formula 3, and then the color tone of the retained area map after the preliminary adjustment can be adjusted using formula 4 to obtain the adjusted retained area map, so that the color tone of the adjusted retained area map is closer to the target area, but it will not deviate too much from the retained area map before the color tone adjustment is performed, reducing the possibility of the color being too bright or too dark due to the adjusted pixel value being too large or too small, improving the color tone adjustment effect, and then improving the area replacement effect.

[0106] S108: Based on the transparency map, perform image fusion on the adjusted reserved area map and the material image to obtain a target image in which the non-reserved area is replaced by the target area.

[0107] The image fusion may include but is not limited to any method such as concatenation, addition, and multiplication of pixel values ​​of pixels in two images.

[0108] In S108, a first result can be obtained based on the fusion of the transparency map and the reserved area map, and a second result can be obtained based on the fusion of the material image and the inverse transparency map corresponding to the transparency map; and then the first result and the second result are fused to obtain the target image.

[0109] The embodiment of the present application does not limit the specific formula of image fusion, and the following is only an example:

[0110] new=new_FG*Alpha+BG*(1-Alpha)……………………(5)

[0111] As shown in the above formula (5), new indicates the pixel value of the pixel in the target image. new_FG indicates the pixel value of the pixel in the adjusted reserved area map obtained in S106. BG indicates the pixel value of the pixel in the material image corresponding to the target area. Alpha indicates the transparency value of the pixel in the transparency map. Among them, 1-Alpha can be expressed as the reverse transparency map corresponding to the transparency map. The first result obtained by fusing the transparency map Alpha with the reserved area new_FG can be expressed as new_FG*Alpha, and the second result obtained by fusing the material image BG with the reverse transparency map can be expressed as BG*(1-Alpha). The result of fusing the first result and the second result is new obtained by formula (5).

[0112] The transparency value of the pixels belonging to the reserved area in the transparency map is a first value, and the transparency value of the pixels belonging to the non-reserved area is a second value. For example, in a scene where the area is replaced and the original area is not retained, the first value may be 1, indicating that the pixel is non-transparent, and the value may be 0, indicating that the pixel is transparent. Through new_FG*Alpha, the pixels belonging to the reserved area in the adjusted reserved area map may be adjusted to non-transparent, and the pixels belonging to the non-reserved area may be adjusted to transparent. Through BG*(1-Alpha), the pixels belonging to the target area in the material image may be adjusted to non-transparent, and the pixels belonging to the non-target area may be adjusted to transparent. The image fusion may be achieved through Formula 5.

[0113] According to the scheme recorded in S102-S108, the color difference between the reserved area and the target area can be determined based on the pixel values ​​of the pixels in the reserved area obtained by cutting out the original image and the pixel values ​​of the pixels in the target area, and then the hue of the pixels in the reserved area can be adjusted based on the color difference to unify the hue of the pixels in the reserved area and the pixels in the target area, so that in the process of image fusion of the adjusted reserved area image with the target area and performing area replacement, it can be ensured that the hue of the target area matches that of the reserved area in the original image, thereby improving the area replacement effect.

[0114] In the related art, a segmentation model is generally used to perform semantic segmentation on the reserved area to obtain a rough mask result of the non-reserved area. Then, the original non-reserved area is replaced based on the mask result and the source image. Since the mask result output by the segmentation model is often rough in the boundary area between the reserved area and the non-reserved area, directly using the mask for replacement will cause obvious artificial traces in the boundary area. For example, in the sky replacement scene, some local details between the sky and the horizon in the original image may be missing.

[0115] In some embodiments, a three-part image clipping method can be used to solve the above-mentioned problem. In S102, the original image can be clipped using the three-part image corresponding to the original image to obtain the reserved area map and the transparency map, so that the three-part image can be used to well distinguish the characteristics of the reserved area, the non-reserved area, and the undetermined area between the reserved area and the non-reserved area in the original image, so that the obtained transparency map can save the detailed information of the intersection position of the reserved area and the non-reserved area. Compared with the solution of directly using the mask result for area replacement, when performing area replacement based on the transparency map, it can be beneficial to improve the connection effect between the reserved area and the target area.

[0116] See also Figure 5 , Figure 5 A schematic flow chart of a method for cutting out an image according to an embodiment of the present application. Figure 5 The steps shown are supplementary explanations for S102. Figure 5 As shown, the cutout method may include S502-S504. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0117] S502, obtaining a trisection map corresponding to the original image to be processed; the numerical value corresponding to each pixel in the trisection map represents the probability that the pixel belongs to any one of the reserved area, the non-reserved area or the area to be determined in the original image.

[0118] The tripartite map has the characteristic of distinguishing the foreground, background, and the junction area between the foreground and the background in the image. That is, no matter whether the reserved area is the foreground or the background in the original image, the tripartite map is used to distinguish the reserved area, the non-reserved area, and the undetermined area between the reserved area and the non-reserved area in the original image, so as to save the detailed information of the junction position between the reserved area and the non-reserved area. The tripartite map in this application can be represented by trimap.

[0119] In some embodiments, editing software can be used to assist in obtaining the tripartite map of the original image. With the reserved area as the foreground area, the non-reserved area (background area), the reserved area (foreground area), and the area to be determined can be marked on the original image through image editing software to obtain the tripartite map.

[0120] In some implementations, the tripartite graph may be obtained by using a tripartite graph extraction network generated based on a neural network. The tripartite graph extraction network may be obtained by pre-training with training samples labeled with tripartite graph information.

[0121] In the above two implementations, either the images are manually annotated by software or trimap is predicted by a prediction network. However, the manual annotation method is too complicated and inconvenient to use; and the network prediction method requires a large number of trimap annotations, which is relatively cumbersome.

[0122] In order to simplify the trimap acquisition process, in some implementations, the user is not required to manually annotate the trimap, nor is a prediction network for predicting the trimap required to be pre-trained. Instead, the trimap can be obtained based on the result of semantic segmentation combined with probability conversion.

[0123] See also Figure 6 , Figure 6 A schematic diagram of a process for obtaining a trisection map according to an embodiment of the present application. Figure 6 The steps shown are supplementary explanations of the method for obtaining the trisection map in S502. Figure 6 As shown, the cutout method may include S602-S604. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0124] S602, performing semantic segmentation processing on the original image to be processed to obtain a semantic probability map corresponding to the original image.

[0125] The image to be cut out can be referred to as the original image. For example, suppose a non-sky area is to be extracted from a person image, and the person image can be referred to as the original image. The non-sky area is the target to be extracted by the cut out process, and can be referred to as the reserved area, which can be the foreground or background in the original image.

[0126] In this embodiment, semantic segmentation processing can be performed on the original image, for example, segmentation processing can be performed through a semantic segmentation network. The semantic segmentation network includes but is limited to commonly used segmentation networks such as SegNet, U-Net, DeepLab, and FCN.

[0127] After semantic segmentation, a semantic probability map of the original image can be obtained, which may include: the probability that each pixel in the original image belongs to the reserved area. Take the reserved area as the foreground as an example. In the semantic probability map, the probability that a certain pixel in the original image belongs to the foreground may be 0.85, and the probability that another pixel belongs to the foreground may be 0.24.

[0128] In order to distinguish from other probabilities that appear in the subsequent description of embodiments, the embodiment of the present application may refer to the probability that each pixel in the semantic probability map belongs to the reserved area as the first probability.

[0129] S604: Perform probability conversion processing based on the semantic probability map to obtain a trisection map corresponding to the original image.

[0130] In this step, a probability conversion process may be performed based on the result of the semantic segmentation process to obtain a tri-map. The tri-map obtained by the probability conversion process in this embodiment may be represented by a soft-trimap.

[0131] The probability conversion process may be performed by mathematical conversion to map the probabilities corresponding to the pixels obtained in the semantic probability map to the values ​​corresponding to the pixels in the soft-trimap.

[0132] Specifically, taking the reserved area as the foreground as an example, the probability in the semantic probability map can be converted into the following two parts:

[0133] 1) Based on the semantic probability graph, the first probability is converted into a second probability.

[0134] The soft-trimap may include three types of areas: "reserved area (foreground)", "non-reserved area (background)" and "to-be-determined area". In this embodiment, the probability that a pixel belongs to the to-be-determined area in the soft-trimap may be referred to as the second probability.

[0135] When converting the first probability that a pixel in the semantic probability map belongs to the reserved area to the second probability, the following probability conversion principle can be followed: the higher the probability that the first probability represents that the pixel belongs to the reserved area (foreground) or the non-reserved area (background), the lower the probability that the second probability represents that the pixel belongs to the to-be-determined area in the trigraph. For example, the closer the first probability is to 1 and 0, the closer the second probability is to 0; the closer the first probability is to 0.5, the closer the second probability is to 1. The above conversion principle means that if the probability that a pixel in the image belongs to the reserved area (foreground) is higher, or the probability that it belongs to the non-reserved area (background) is higher, the probability that the pixel belongs to the to-be-determined area is lower; and when the probability that the pixel belongs to the reserved area (foreground) or the non-reserved area (background) is around 0.5, it means that the probability that the pixel belongs to the to-be-determined area is higher.

[0136] Based on the above principle of probability conversion, the first probability can be converted to obtain the second probability. The embodiment of the present application does not limit the specific formula for how to convert, and the following is only an example:

[0137] un=-k4*score 4 +k3*score 3 –k2*score 2 +k1*score.......(6)

[0138] As shown in the above formula (6), un represents the second probability that the pixel belongs to the area to be determined, and score represents the first probability that the pixel belongs to the reserved area in the semantic probability map. The formula (6) is a polynomial fitting method, through which the first probability of the pixel is fitted to obtain the second probability. This embodiment does not limit the specific values ​​of the above-mentioned coefficients "k1 / k2 / k3 / k4".

[0139] It is understandable that the actual implementation is not limited to the above polynomial fitting, and other functional formulas can also be used as long as the above probability conversion principle is followed. This embodiment uses polynomial fitting to convert the first probability into the second probability, which can make the conversion calculation efficiency of this polynomial higher and more accurately reflect the above conversion principle.

[0140] 2) Generate the tri-map according to the first probability and the second probability of the pixel.

[0141] As mentioned above, by performing semantic segmentation on the original image, a semantic probability map can be obtained. Through the semantic probability map, the retained area (foreground) and the non-retained area (background) in the original image can be roughly distinguished. For example, if the first probability that a pixel belongs to the foreground is 0.96, then the probability that it belongs to the foreground is very high; if the first probability that a pixel belongs to the foreground is 0.14, then the probability that the pixel belongs to the background is very high.

[0142] After obtaining the second probability based on the semantic probability map, the second probability of each pixel belonging to the area to be determined can be obtained. For each pixel in the original image, the first probability corresponding to the pixel in the semantic probability map and the second probability that the pixel belongs to the area to be determined can be combined for probability fusion to obtain the value corresponding to the pixel in the soft-trimap, which can represent the probability that the pixel belongs to any of the determined reserved area (foreground), non-reserved area (background) or the area to be determined in the original image.

[0143] For example, in soft-trimap, if the value corresponding to a pixel is closer to 1, it means that the pixel is more likely to belong to the reserved area (foreground) in the original image; if the value corresponding to the pixel in soft-trimap is closer to 0, it means that the pixel is more likely to be in the non-reserved area (background); if the value corresponding to the pixel in soft-trimap is closer to 0.5, it means that the pixel is more likely to be in the undetermined area. That is, the value corresponding to the pixel in soft-trimap can indicate the probability that the pixel belongs to any of the reserved area, non-reserved area or undetermined area.

[0144] The following formula (7) illustrates a method of obtaining a trisection map according to the first probability and the second probability:

[0145] soft_trimap=-k5*un / k6*sign(score-k7)+(sign(score-k7)+k8) / k9....(7)

[0146] In the above formula (7), soft_trimap represents the value corresponding to the pixel in soft-trimap, un represents the second probability, score represents the first probability, and sign() represents the sign function. Similarly, this embodiment does not limit the specific values ​​of the above coefficients "k5 / k6 / k7 / k8".

[0147] As described in the above example, after converting the first probability to obtain the second probability and combining the first probability and the second probability to generate the trimap, the probability conversion process based on the semantic probability map is implemented to obtain the trimap soft_trimap.

[0148] In some embodiments, before performing the above-mentioned probability conversion process based on the semantic probability map, the semantic probability map may be pooled, and the above-mentioned probability conversion process may be performed on the semantic probability map after the pooling process. Please refer to the following formula (8):

[0149] score_=avgpool2d(score,ks,stride).....(8)

[0150] As shown in formula (8), in one example, the semantic probability map can be average pooled, and pooled according to the convolution step length stride and the convolution kernel size (kernel_size, ks). score_ represents the semantic probability map obtained after the pooling process.

[0151] If the semantic probability map is pooled, then the score in the above formula (6) and formula (7) is replaced by score_, that is, the semantic probability map after pooling is used to perform probability conversion.

[0152] The size of the kernel used in the above pooling process can be adjusted, and the pooling process is performed before the probability conversion of the semantic probability map, which helps to adjust the width of the undetermined area in the soft_trimap to be generated by adjusting the convolution kernel size. For example, the larger the kernel_size, the wider the width of the undetermined area can be.

[0153] In some embodiments, assuming that the semantic segmentation processing of the original image to be processed is performed by a semantic segmentation network, then before the semantic segmentation processing, the image size of the original image can also be preprocessed. The preprocessing can be based on the downsampling multiple of the original image by the semantic segmentation network, and the image size of the original image is processed by an integer multiple of the downsampling multiple. The integer multiple processing makes the processed image size able to divide the above-mentioned downsampling multiple scale_factor. The scale_factor is the downsampling multiple of the original image by the semantic segmentation network, and the specific value is determined by the network structure of the semantic segmentation network.

[0154] The steps of S602-S604 are used to obtain the triplicate map. The triplicate map can be obtained by performing probability conversion based on the semantic probability map obtained by semantic segmentation of the original image, so that the triplicate map can be obtained more quickly and conveniently, and no manual labeling is required, and no prediction network needs to be trained through trimap labeling, thereby making the process of cutout processing easier to implement. Moreover, the method of obtaining the triplicate map through probability conversion is based on the semantic probability map of semantic segmentation, so that the generated triplicate map is more accurate.

[0155] S504: Perform a matting process according to the trisection image and the original image to obtain the matting result.

[0156] In this step, the process of the matting processing may include: taking the trisection image and the original image as inputs of the matting network, obtaining the retained area residual output by the matting network, and the initial transparency map of the original image.

[0157] The retained area residual may be a residual result obtained by a residual processing unit in a matting network. The retained area residual may indicate a difference between a pixel value of a pixel in the retained area extracted by the residual processing unit and a pixel value of a corresponding pixel in the original image.

[0158] The value of a pixel in the initial transparency map indicates the transparency of the pixel.

[0159] Next, the reserved area map can be obtained based on the original image and the reserved area residual (for example, the foreground image is obtained based on the addition of the foreground residual and the original image, or the background image is obtained based on the addition of the background residual and the original image), and the values ​​of the pixels in the initial transparency map can be adjusted according to the tri-map soft_trimap to obtain a transparency map corresponding to the original image.

[0160] Through the above adjustment, first, the transparency value of the pixels in the reserved area of ​​the tri-plot in the initial transparency map can be adjusted to the first value. Second, the transparency value of the pixels in the non-reserved area of ​​the tri-plot in the initial transparency map can be adjusted to the second value. Third, based on the size of the transparency value of the pixels in the to-be-determined area of ​​the tri-plot in the initial transparency map, the pixels in the to-be-determined area in the initial transparency map can be distinguished to distinguish whether the pixels in the to-be-determined area belong to the reserved area or the non-reserved area, and a value can be assigned to them.

[0161] For example, in a scenario where an area is replaced and the original area is not retained, based on the above-mentioned adjustment, the transparency values ​​of the pixels in the initial transparency map that are in the retained area of ​​the tri-map can be adjusted to 1, the transparency values ​​of the pixels in the initial transparency map that are in the non-retained area of ​​the tri-map can be adjusted to 0, and the transparency values ​​of the pixels in the initial transparency map that are in the to-be-determined area that have transparency values ​​greater than 0.5 can be adjusted to 1, and the transparency values ​​less than 0.5 can be adjusted to 0.

[0162] According to the cutout scheme recorded in S502-S504, the three-part image can be used for cutout, so that the detailed information of the intersection between the reserved area and the non-reserved area can be well preserved, which is conducive to improving the connection effect between the reserved area and the target area when performing area replacement.

[0163] In the related art, the region replacement software on the mobile phone mainly uploads the data to the server for processing, and then transmits the region replacement result back to the mobile phone for local reading. In this solution, the security and privacy of the data are difficult to be guaranteed.

[0164] To solve this problem, considering the processing power of the mobile terminal, in some embodiments, the network deployed to the mobile terminal can be miniaturized, and the size of the original image can be scaled so that the running time and memory consumption are within the burden range of the mobile terminal, so that there is no need to replace the area through the server, ensuring the security and privacy of the data. An example of performing image matting on a mobile terminal is described below.

[0165] This embodiment can use a semantic segmentation network and a matting network when performing matting using the method recorded in S502-S504. Among them, the semantic segmentation network can be a network such as SegNet, U-Net, and the matting network can include an encoder and a decoder. The encoder of the matting network can adopt the structural design of mobv2, and before being deployed to the mobile terminal, the matting network can be channel compressed. The channel compression can be to compress the number of channels of the intermediate features of the network of the matting network. For example, the number of output channels of the convolution kernel in the matting network processing process can be reduced. Assuming that the number of output channels of the convolution kernel is originally a, it can be compressed according to 0.35 times the number of channels. After compression, the number of output channels of the convolution kernel is 0.35*a.

[0166] The following describes the replacement of the sky scene in a person image as an example.

[0167] See also Figure 7a , Figure 7a A schematic diagram of a character image shown in an embodiment of the present application.

[0168] Figure 7a The sky area in the illustrated character image is a background area and is also a non-reserved area, and needs to be replaced by another sky area (ie, the target area in the present application) in a pre-acquired material image. Figure 7a The non-sky area in is the retained area that needs to be cut out in this example, that is, the foreground area.

[0169] See also Figure 8 , Figure 8 A schematic diagram of a region replacement process shown in an embodiment of the present application. Fig. 9 Based on Figure 8 Schematic diagram of the regional replacement process. Fig. 9 As shown, the method may include S901-S909. Unless otherwise specified, the present application does not specifically limit the execution order of these steps.

[0170] S901, scaling the original image to be processed.

[0171] The original image of this embodiment can be Figure 7a The character image shown can be taken by the user through the mobile terminal camera, or can also be an image stored in the mobile terminal or received from other devices.

[0172] The purpose of the cutout process in this embodiment may be to extract the non-sky area in the person image. The non-sky area in the original image may be used as the foreground.

[0173] Since the present embodiment performs the cutout processing on the mobile terminal, in order to reduce the processing burden on the mobile terminal and save the amount of calculation on the mobile terminal, the original image can be scaled. Figure 7a The size of the original image in is 1080*1920, and the image can be scaled to 480*288. For example, the scaling can be performed by bilinear interpolation. The scaling can be performed by referring to the following formula (9) and formula (10):

[0174] scale=max(h / basesize,w / basesize)......(9)

[0175] new_h=int(h / scale+k10) new_w=int(w / scale+k11).....(10)

[0176] Wherein, h and w are the length and width of the original image, basesize is the base size, which is 480 in this example, and int(x) indicates rounding x. new_h and new_w are the sizes of the original image after scaling, respectively. The specific values ​​of the coefficients in formula (10) are not limited in this embodiment.

[0177] In addition, the image size of the original image can be processed by an integer multiple of the downsampling factor according to formula (11) and formula (12) to control the scaled image size to be able to divide the downsampling factor scale_factor of the semantic segmentation network. It can be understood that the integer multiple processing can also use other formulas, not limited to the following two formulas.

[0178] new_h=int(int(int(new_h–k12+scale_facor–k13) / scale_factor)*scale_factor)......(11)

[0179] new_w=int(int(int(new_w–k14+scale_facor–k15) / scale_factor)*scale_factor)......(12)

[0180] This embodiment does not limit the specific values ​​of the coefficients in the above formula (11) and formula (12). For example, the values ​​of k12 to k15 can all be set to 1. If the original image before scaling is marked with A, then the original image obtained by scaling to a 480*288 image and then normalizing the image can be marked with B. Figure 8 As shown in , the original image B is the original image after scaling processing.

[0181] S902, performing semantic segmentation processing on the original image after the scaling processing through a semantic segmentation network to obtain the semantic probability map output by the semantic segmentation network.

[0182] Combination Figure 8 As shown, the original image B can be processed by semantic segmentation through the semantic segmentation network 81 to obtain a semantic probability map 82 output by the semantic segmentation network. The semantic probability map can be marked with score, and Figure 7b A semantic probability map is shown in FIG. As can be seen, the semantic probability map score indicates the probability that a pixel belongs to the non-sky area (foreground), which roughly distinguishes the foreground and background in the image, that is, roughly distinguishes the sky area from the non-sky area.

[0183] S903, performing probability conversion processing based on the semantic probability map to obtain a trigraph.

[0184] In this step, the soft-trimap may be generated according to the probability conversion process described in S604. For example, the semantic probability map may be first pooled according to formula (8), and then the pooled semantic probability map may be used to generate the soft-trimap according to formulas (6) and (7). Figure 8 The three-point diagram 83 in the figure.

[0185] See also Figure 7c The indication is that Figure 7c The figure shows a soft-trimap. It can be seen that the probability value of a pixel in the soft-trimap can represent the probability that the pixel belongs to the three areas. According to the probability value, the "sky area", "non-sky area" and "the area to be determined between the sky area and the non-sky area" in the image can be distinguished.

[0186] S904, using the trisection image and the original image as inputs of the matting network, and obtaining the foreground residual and initial transparency map output by the matting network.

[0187] See also Figure 8 As shown, both the trisection image 83 and the original image B can be used as inputs of the matting network 84, which can output a 4-channel result, where the result of one channel is the initial transparency map raw_alpha, and the results of the other three channels are the foreground residual fg_res. Figure 8 The first result 85 output by the matting network in may include "raw_alpha+fg_res".

[0188] S905, obtaining a foreground image including a foreground area based on the original image and the foreground residual, and obtaining a transparency map based on the initial transparency and the tri-map.

[0189] The foreground is the non-sky area in the person image.

[0190] Please continue to combine Figure 8 As shown, the foreground residual fg_res can be enlarged by bilinear interpolation to restore the scale before the scaling process, and then execute formula (13):

[0191] FG=clip(A+fg_res,s1,s2)......(13)

[0192] like Figure 8 As shown, the clipping result 86, i.e., the foreground image FG in the original image, can be obtained based on the foreground residual fg_res after the magnification processing and the original image A. Wherein, (clip(x, s1, s2) is to limit the value of x to [s1, s2]). This embodiment does not limit the specific values ​​of s1 and s2 in the above formula (13), for example, s1 can be 0 and s2 can be 1.

[0193] In addition, the transparency map corresponding to the non-sky area can be calculated according to the following formula (14) and formula (15):

[0194] fs=clip((soft_trimap-s3) / s4,s5,s6)......(14)

[0195] Alpha=clip(fs+un*raw_alpha,s7,s8)....(15)

[0196] Wherein, Alpha represents the transparency corresponding to the non-sky area. After obtaining Alpha, Alpha can be enlarged back to the original size of the original image before scaling by bilinear interpolation. Similarly, this embodiment does not limit the specific values ​​of the coefficients s3 to s8 in the above formula (14) and formula (15).

[0197] Figure 7d The transparency map Alpha is shown, where the non-sky area and the sky area can be clearly distinguished in Alpha. For example, the first value of the pixel in the sky area is 1, indicating non-transparency. The second value of the pixel in the non-sky area is 0, indicating full transparency. Figure 7e The non-sky area cut out, ie, the foreground image FG, is illustrated.

[0198] S906 , sampling the foreground image FG, Alpha and the material image including the target area BG according to the preset step to obtain a foreground sampling point FG1 , a transparency sampling point Alpha1 and a material sampling point BG1 .

[0199] Combination Figure 8In this step, the value obtained by dividing the short side of the character image by 20 can be used as the step to perform the sampling on the cutout result 86 (foreground image FG and Alpha) and the material image 87.

[0200] S907: Determine the color difference between the target area BG and the foreground area FG.

[0201] In this step, we can Figure 3 The method shown results in the color difference.

[0202] Wherein, based on the foreground sampling point FG1 and the transparency sampling point Alpha1, the first pixel mean fg_mean of the foreground sampling point can be obtained according to the above formula (1).

[0203] Based on the material sampling point BG1, a second pixel mean bg_mean of the material sampling point is determined.

[0204] Then, the color difference diff can be obtained according to the above formula (2).

[0205] S908 , adjusting the hue of the foreground image FG based on the color difference to obtain an adjusted foreground image matching the hue of the target area.

[0206] In this step, you can Figure 4 The color tone adjustment method shown performs color tone adjustment.

[0207] Combination Figure 8 , wherein the non-sky area (the foreground image FG in the cutout result) can be adjusted according to the preset adjustment coefficient q according to the aforementioned formula (3) to obtain a preliminarily adjusted foreground image. Then, based on the aforementioned formula (4), the tone of the preliminarily adjusted foreground image is corrected to obtain a final adjustment result 88, i.e., the final adjusted foreground image new_FG.

[0208] S909: Based on the transparency map, perform image fusion on the adjusted foreground image and the material image to obtain a target image in which the non-reserved area is replaced by the target area.

[0209] In this step, the image fusion can be performed according to the above formula (5) to obtain the target image new after replacing the sky area.

[0210] like Figure 7f As shown, the target image after replacing the sky area is obtained through S901-S909. In this method, on the one hand, during the process of region replacement, the color tone of the target region can be guaranteed to match that of the non-sky region in the original image, thereby improving the region replacement effect.

[0211] On the other hand, the trisection map can be used for cutout, so as to obtain the detailed information of the intersection of the sky area and the non-sky area. When replacing the area, it is helpful to improve the connection effect between the sky area and the non-sky area.

[0212] On the other hand, by performing channel compression and other processing on the matting network and scaling the original image, it can be made more suitable for matting on mobile terminals. For example, after a user uses his or her mobile terminal to take an image, he or she can directly perform matting processing on the non-sky area on the mobile terminal and merge it with the target area to complete the replacement of the sky area. In this way, all these processes can be performed locally on the mobile terminal without uploading to the cloud, thereby improving data security and privacy protection. Figure 8 It can be found that the region replacement method can directly obtain the matting result by taking a single original image as input, that is, providing an original image, the region replacement method provided in the embodiment of the present application can obtain a prediction of the foreground in the original image, and less input information is required, thereby making the image processing more convenient.

[0213] In addition, in the process of cutout processing in the embodiment of the present application, the semantic segmentation network and the image cutout network are used, and the present embodiment does not limit the training methods of these two networks. Fig.10 The flowchart of a network training method shown in the embodiment of the present application is shown in FIG. The method can be used to jointly train a semantic segmentation network and a matting network. Fig.10 As shown, the method may include the following processing:

[0214] S1002: Acquire a training sample set, where the training sample set includes a plurality of sample data.

[0215] In some embodiments, each sample data in the training sample set may include a sample image, a first feature label corresponding to the sample image, and a second feature label corresponding to the sample image. Taking the matte scene as an example, the first feature label may be a segmentation label for the sample image, and the second feature label may be a matte label for the sample image.

[0216] S1004, for each sample data in the training sample set, the sample data is processed to obtain a first image containing global image information of the sample image and a segmentation label corresponding to the first image, and a second image containing local image information of the sample image and a matting label corresponding to the second image.

[0217] In some embodiments, the sample image of the sample data may be subjected to image processing to obtain a first image including most of the image information of the sample image, and the first image may be considered to include the global image information of the sample image, and the first feature label corresponding to the sample image may be subjected to the same processing to obtain a segmentation label corresponding to the first image. For example, the sample image may be scaled according to the size requirement of the semantic segmentation network for the input image, but most of the image information of the sample image may still be retained to obtain the first image, and the first feature label may be subjected to the same scaling processing to obtain the segmentation label.

[0218] At the same time, a second image including local image information of the sample image can also be obtained, and the second feature label corresponding to the sample image is processed in the same manner to obtain a matting label corresponding to the second image. For example, the sample image can be partially cropped to obtain a second image including local image information of the sample image, and the second feature label can be subjected to the same local cropping process to obtain the matting label.

[0219] S1006, performing semantic segmentation processing on the first image through a semantic segmentation network to obtain the semantic probability map output by the semantic segmentation network.

[0220] S1008, performing probability conversion processing based on the semantic probability map to obtain a trigraph.

[0221] The probability conversion process of this step can refer to the above-mentioned embodiment and will not be described in detail. The soft-trimap of the embodiment of the present application can be obtained through the probability conversion process.

[0222] S1010, performing matting processing on the trisection image and the second image to obtain a matting result. The matting result may indicate a matting result for a reserved area in the sample.

[0223] S1012, adjusting network parameters of the semantic segmentation network according to the difference between the semantic probability map and the segmentation label, and adjusting network parameters of the matting network based on the difference between the matting result and the matting label.

[0224] From the above, it can be seen that in the implementation mode of the present application, by processing each sample data, the first sub-network is trained using the obtained first image and first label including global image information, and the second sub-network is trained using the second image and second label including local image information, so as to improve the joint training effect and reduce the risk of network effect degradation.

[0225] In addition, in the above training method, the generation of soft-trimap adopts probability conversion processing, which can assist the network training to a certain extent.

[0226] Specifically, soft-trimap can be adaptively adjusted during the network training process. For example, in the process of adjusting the network parameters of the semantic segmentation network according to the difference between the semantic probability map and the segmentation label, and adjusting the network parameters of the matting network based on the difference between the matting result and the matting label, the network parameters of the semantic segmentation network will be updated, and then the semantic probability map output by the semantic segmentation network will also be updated.

[0227] Furthermore, the generation of soft-trimap is based on the semantic probability map, so the update of the semantic probability map will bring about the update of the soft-trimap of the trimap, and then the matting result will also be updated. That is, the network training process usually iterates multiple times, and after each iteration, if the parameters of the semantic segmentation network are updated, even if the input is the same image, the semantic probability map, soft-trimap and matting result will be adaptively updated, and the network parameters will continue to be adjusted according to the updated results. This adaptive adjustment of the soft-trimap will help to dynamically optimize the generated soft-trimap and matting results as the semantic segmentation network is adjusted, so that the training effect of the final model is better and the retained area in the image can be extracted more accurately.

[0228] Fig.11 An image processing device is illustrated, and the device can be used to implement the image processing method of any embodiment of the present application. Fig.11 As shown, the device may include: a cutout module 1110 , a determination module 1120 , an adjustment module 1130 and a fusion module 1140 .

[0229] like Fig.11 As shown, the device 1100 includes:

[0230] The cutout module 1110 is used to perform cutout processing on the original image to be processed to obtain a cutout result, wherein the cutout result includes a reserved area map containing the reserved area in the original image and a transparency map corresponding to the original image; the reserved area is the foreground or background in the original image; the value of the pixel in the transparency map indicates the transparency of the pixel;

[0231] A determination module 1120, configured to determine a color difference between the reserved area and the target area according to a difference between pixels in the reserved area map and pixels in a material image including the target area; the target area is used to replace a non-reserved area in the original image;

[0232] An adjustment module 1130 is used to adjust the color tone of pixels in the reserved area map according to the color difference to obtain an adjusted reserved area map that matches the color tone of the target area;

[0233] The fusion module 1140 is used to perform image fusion on the adjusted reserved area map and the material image based on the transparency map to obtain a target image in which the non-reserved area is replaced by the target area.

[0234] In some embodiments, the determining module 1120 is specifically configured to:

[0235] The pixels in the reserved area map and the pixels in the material image are sampled respectively to obtain reserved area sampling points and material sampling points;

[0236] Based on the difference between the pixel values ​​of the sampling points of the reserved area and the pixel values ​​of the material sampling points, the color difference between the reserved area and the target area is determined.

[0237] In some embodiments, the apparatus 1100 further includes:

[0238] A module is used to sample pixels in the transparency map to obtain transparency sampling points;

[0239] The determining module 1120 is specifically configured to:

[0240] Determine a first pixel mean value of the sampling points in the reserved area based on the pixel values ​​of the sampling points in the reserved area and the transparency values ​​of the transparency sampling points;

[0241] Determining a second pixel mean value of the material sampling point based on the pixel value of the material sampling point;

[0242] The color difference between the reserved area and the target area is determined according to the difference between the first pixel mean value and the second pixel mean value.

[0243] In some embodiments, the adjustment module 1130 is specifically configured to:

[0244] Based on the color difference, pixel values ​​of pixels in the reserved area map are adjusted to obtain a preliminarily adjusted reserved area map; pixels in the preliminarily adjusted reserved area map incorporate the color difference;

[0245] Based on the difference between the pixel mean value of the pixels in the retained area map after the preliminary adjustment and the pixel mean value of the pixels in the retained area map before the preliminary adjustment, the pixel values ​​of the pixels in the retained area map after the preliminary adjustment are adjusted to obtain an adjusted retained area map that matches the color tone of the target area.

[0246] In some embodiments, the fusion module 1140 is specifically used to:

[0247] Obtaining a first result based on fusing the transparency map with the reserved area map;

[0248] Obtaining a second result based on fusing the material image with an inverse transparency map corresponding to the transparency map;

[0249] The first result and the second result are fused to obtain the target image.

[0250] In some embodiments, the cutout module 1110 is specifically used to:

[0251] Obtaining a trisection map corresponding to the original image to be processed; a value corresponding to each pixel in the trisection map represents a probability that the pixel belongs to any of the reserved area, the non-reserved area or the area to be determined in the original image;

[0252] A matting process is performed according to the trisection image and the original image to obtain the matting result.

[0253] In some embodiments, the cutout module 1110 is specifically used to:

[0254] Performing semantic segmentation processing on the original image to be processed to obtain a semantic probability map of the original image, wherein the value of a pixel in the semantic probability map indicates a first probability that the pixel belongs to the reserved area in the original image;

[0255] Probability conversion processing is performed based on the semantic probability map to obtain a trisection map corresponding to the original image.

[0256] In some embodiments, the cutout module 1110 is specifically used to:

[0257] Performing semantic segmentation processing on the original image through a semantic segmentation network to obtain the semantic probability map output by the semantic segmentation network;

[0258] The performing matting processing according to the trisection image and the original image comprises: performing matting processing according to the trisection image and the original image through a matting network.

[0259] In some embodiments, the cutout module 1110 is specifically used to:

[0260] The first probability of each pixel in the semantic probability map is converted by probability to obtain a second probability that the pixel belongs to the area to be determined in the trigraph;

[0261] The tri-map is generated according to the first probability and the second probability of the pixel.

[0262] In some embodiments, the higher the probability that the first probability represents that the pixel belongs to the foreground or the background, the lower the probability that the second probability obtained by probability conversion represents that the pixel belongs to the to-be-determined area in the trigraph;

[0263] The generating the tri-map according to the first probability and the second probability of the pixel includes: for each pixel in the original image, performing probability fusion according to the first probability and the second probability corresponding to the pixel, and determining a value corresponding to the pixel in the tri-map.

[0264] In some embodiments, the cutout module 1110 is specifically used to:

[0265] Performing matting processing on the trisection image and the original image to obtain a residual of a reserved area and an initial transparency map of the original image; the value of a pixel in the initial transparency map indicates the transparency of the pixel;

[0266] Based on the original image and the retained area residual, obtaining the retained area map;

[0267] According to the trisection map, the values ​​of the pixels in the initial transparency map are adjusted to obtain a transparency map corresponding to the original image.

[0268] In some embodiments, the apparatus 1100 further includes:

[0269] A scaling module, used for scaling the original image;

[0270] The cutout module 1110 is specifically used for:

[0271] Amplifying the residual of the retained area obtained by performing a matting process on the trisection image and the original image to a scale before the scaling process;

[0272] The reserved area map is obtained according to the enlarged reserved area residual and the original image.

[0273] In some embodiments, the non-retained area includes a sky area in the original image; and the target area includes a sky area in the material image.

[0274] The embodiment of the image processing device shown in the present application can be applied to an electronic device. Accordingly, the present application discloses an electronic device, which may include: a processor.

[0275] Memory used to store processor-executable instructions.

[0276] The processor is configured to call the executable instructions stored in the memory to implement the image processing method shown in any of the aforementioned embodiments.

[0277] See also Fig.12 , Fig.12 A schematic diagram of the hardware structure of an electronic device shown in an embodiment of the present application.

[0278] like Fig.12 As shown, the electronic device may include a processor for executing instructions, a network interface for network connection, a memory for storing operating data for the processor, and a non-volatile memory for storing instructions corresponding to the image processing device.

[0279] The embodiments of the device can be implemented by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, in addition to Fig.12 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device in the embodiment is located may also include other hardware according to the actual function of the electronic device, which will not be described in detail.

[0280] It is understandable that, in order to improve the processing speed, the corresponding instructions of the image processing device can also be directly stored in the memory, which is not limited here.

[0281] The present application proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can be used to enable a processor to execute the image processing method shown in any of the aforementioned embodiments.

[0282] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0283] The term "and / or" in the present application means at least one of the two. For example, "A and / or B" may include three options: A, B, and "A and B".

[0284] Each embodiment in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0285] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0286] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware that may include the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0287] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0288] A computer suitable for executing a computer program may include, for example, a general and / or special purpose microprocessor, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer may include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, a computer does not necessarily have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0289] Computer readable media suitable for storing computer program instructions and data may include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0290] Although the application includes many specific implementation details, these should not be interpreted as limiting any disclosed scope or the scope of protection claimed, but are mainly used to describe the features of specific disclosed specific embodiments. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0291] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0292] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0293] The above are only preferred embodiments of one or more embodiments of the present application and are not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the scope of protection of one or more embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing a matting process on the original image to be processed to obtain a matting result, wherein the matting result includes a reserved area map including a reserved area in the original image and a transparency map corresponding to the original image; the reserved area is a foreground or background in the original image; and the value of a pixel in the transparency map indicates the transparency of the pixel; Determining a color difference between the reserved area and the target area according to a difference between pixels in the reserved area map and pixels in a material image containing a target area; the target area is used to replace a non-reserved area in the original image; Adjusting the color tone of pixels in the reserved area map according to the color difference to obtain an adjusted reserved area map matching the color tone of the target area; Based on the transparency map, the adjusted reserved area map is fused with the material image to obtain a target image in which the non-reserved area is replaced by the target area.

2. The method according to claim 1, characterized in that The determining the color difference between the reserved area and the target area according to the difference between the pixels in the reserved area map and the pixels in the material image including the target area comprises: The pixels in the reserved area map and the pixels in the material image are sampled respectively to obtain reserved area sampling points and material sampling points; Based on the difference between the pixel values ​​of the sampling points of the reserved area and the pixel values ​​of the material sampling points, the color difference between the reserved area and the target area is determined.

3. The method according to claim 2, characterized in that The method further comprises: Sampling pixels in the transparency map to obtain transparency sampling points; The determining the color difference between the reserved area and the target area based on the difference between the pixel value of the sampling point of the reserved area and the pixel value of the sampling point of the material includes: Determine a first pixel mean value of the sampling points in the reserved area based on the pixel values ​​of the sampling points in the reserved area and the transparency values ​​of the transparency sampling points; Determining a second pixel mean value of the material sampling point based on the pixel value of the material sampling point; The color difference between the reserved area and the target area is determined according to the difference between the first pixel mean value and the second pixel mean value.

4. The method according to any one of claims 1 to 3, characterized in that: The step of adjusting the hue of pixels in the reserved area map according to the color difference to obtain an adjusted reserved area map matching the hue of the target area includes: Based on the color difference, pixel values ​​of pixels in the reserved area map are adjusted to obtain a preliminarily adjusted reserved area map; pixels in the preliminarily adjusted reserved area map incorporate the color difference; Based on the difference between the pixel mean value of the pixels in the retained area map after the preliminary adjustment and the pixel mean value of the pixels in the retained area map before the preliminary adjustment, the pixel values ​​of the pixels in the retained area map after the preliminary adjustment are adjusted to obtain an adjusted retained area map that matches the color tone of the target area.

5. The method according to any one of claims 1 to 4, characterized in that: The step of fusing the adjusted reserved area map with the material image based on the transparency map to obtain a target image in which the non-reserved area is replaced with the target area includes: Obtaining a first result based on fusing the transparency map with the reserved area map; Obtaining a second result based on fusing the material image with an inverse transparency map corresponding to the transparency map; The first result and the second result are fused to obtain the target image.

6. The method according to any one of claims 1 to 5, characterized in that: The step of performing a cutout process on the original image to be processed to obtain a cutout result includes: Obtaining a trisection map corresponding to the original image to be processed; a value corresponding to each pixel in the trisection map represents a probability that the pixel belongs to any of the reserved area, the non-reserved area or the area to be determined in the original image; A matting process is performed according to the trisection image and the original image to obtain the matting result.

7. The method according to claim 6, characterized in that The step of obtaining a tri-image corresponding to the original image to be processed includes: Performing semantic segmentation processing on the original image to be processed to obtain a semantic probability map of the original image, wherein the value of a pixel in the semantic probability map indicates a first probability that the pixel belongs to the reserved area in the original image; Probability conversion processing is performed based on the semantic probability map to obtain a trisection map corresponding to the original image.

8. The method according to claim 7, characterized in that The processing of semantic segmentation on the original image to be processed to obtain a semantic probability map corresponding to the original image includes: performing semantic segmentation on the original image through a semantic segmentation network to obtain the semantic probability map output by the semantic segmentation network; The performing matting processing according to the trisection image and the original image comprises: performing matting processing according to the trisection image and the original image through a matting network.

9. The method according to claim 7 or 8, characterized in that: The performing probability conversion processing based on the semantic probability map to obtain a trisection map corresponding to the original image includes: The first probability of each pixel in the semantic probability map is converted by probability to obtain a second probability that the pixel belongs to the area to be determined in the trigraph; The tri-map is generated according to the first probability and the second probability of the pixel.

10. The method according to claim 9, characterized in that The higher the probability that the first probability represents that the pixel belongs to the foreground or the background, the lower the probability that the second probability obtained by probability conversion represents that the pixel belongs to the to-be-determined area in the trigraph; The generating the tri-map according to the first probability and the second probability of the pixel includes: for each pixel in the original image, performing probability fusion according to the first probability and the second probability corresponding to the pixel, and determining a value corresponding to the pixel in the tri-map.

11. The method according to any one of claims 6 to 10, characterized in that: The performing matting processing according to the trisection image and the original image to obtain the matting result includes: Performing matting processing on the trisection image and the original image to obtain a residual of a reserved area and an initial transparency map of the original image; the value of a pixel in the initial transparency map indicates the transparency of the pixel; Based on the original image and the retained area residual, obtaining the retained area map; According to the trisection map, the values ​​of the pixels in the initial transparency map are adjusted to obtain a transparency map corresponding to the original image.

12. The method according to claim 11, characterized in that Before performing semantic segmentation processing on the original image to be processed, the method further includes: performing scaling processing on the original image; The step of obtaining the reserved area map based on the original image and the reserved area residual comprises: Amplifying the residual of the retained area obtained by performing a matting process on the trisection image and the original image to a scale before the scaling process; The reserved area map is obtained according to the enlarged reserved area residual and the original image.

13. The method according to any one of claims 1 to 12, characterized in that: The non-reserved area includes a sky area in the original image; and the target area includes a sky area in the material image.

14. An image processing device, characterized in that: The device comprises: A cutout module is used to perform cutout processing on the original image to be processed to obtain a cutout result, wherein the cutout result includes a reserved area map containing the reserved area in the original image and a transparency map corresponding to the original image; the reserved area is the foreground or background in the original image; the value of the pixel in the transparency map indicates the transparency of the pixel; a determination module, configured to determine a color difference between the reserved area and the target area according to a difference between pixels in the reserved area map and pixels in a material image containing the target area; the target area is used to replace a non-reserved area in the original image; An adjustment module, configured to adjust the hue of pixels in the reserved area map according to the color difference, so as to obtain an adjusted reserved area map matching the hue of the target area; A fusion module is used to fuse the adjusted reserved area map with the material image based on the transparency map to obtain a target image in which the non-reserved area is replaced by the target area.

15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the image processing method as described in any one of claims 1-13 by running the executable instructions.

16. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to enable a processor to execute the image processing method according to any one of claims 1 to 13.

Images