Image processing method, readable medium and electronic equipment

Abstract

The invention relates to the technical field of computer vision, and discloses an image processing method, a readable medium and electronic equipment. According to the method, on the basis that a neural network-based method is adopted for an original image to obtain a foreground mask, a modification area is generated based on the foreground mask, and then in the modification area, a neural network model is used for repairing a modifiable area based on the original image corresponding to a foreground area (or a background area) of the foreground mask. Therefore, the edge (background area) of the foreground image obtained after repairing is more accurate, and a more accurate segmentation effect is further obtained. According to the image processing method provided by the invention, on the basis of the foreground mask obtained through the neural network model, the foreground content (or background content) is repaired in the edge region (modification region) of the foreground mask, so that the accuracy of segmentation of an ambiguous image can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, a readable medium and an electronic device. Background Art

[0002] Image matting is a method of separating the foreground object from the background of an original image or an original image in a video sequence. It is widely used in the fields of film making and digital image editing.

[0003] For example, taking the user-defined wallpaper scene of mobile phone 10 as an example, Figure 1a As shown, in response to the user clicking on the "select control" 101 in the custom wallpaper setting interface 10a, the mobile phone 10 displays the following Figure 1b In response to the user clicking on the picture 102 in the picture selection interface 10b, the mobile phone 10 displays the following Figure 1c The wallpaper adjustment interface 10c shown in the figure. It can be understood that the foreground 103 in the wallpaper adjustment interface 10c is obtained by cutting out the picture 102, and the background 104 can be another picture or a virtual background (such as a solid color background, a cartoon background, etc.). In response to the user adjusting the size and position of the foreground 103 and the background 104 and clicking the "Confirm" control 105, the mobile phone 10 re-integrates the foreground 103 and the background 104 to obtain a new picture, and displays as shown in the figure. Figure 1d The custom wallpaper setting interface 10d shown in the figure includes a new image 106 obtained by merging and an "apply" control 107. In response to the user clicking the "apply" control 107, the mobile phone 10 displays the following Figure 1e In the wallpaper display interface shown, the wallpaper image is the new image 106 obtained by fusion.

[0004] However, due to the ambiguity of some images, for example, a part of the background of the image may be identified as the foreground, or a part of the foreground of the image may be identified as the background, resulting in inaccurate image edge segmentation, which in turn affects the user experience.

[0005] For example Figure 2a The image shown is composed of a foreground "little girl" 201 and a background "star" 202. According to the neural network model, Figure 2b The foreground mask 203 shown is based on the foreground mask 203. Figure 2a The original image shown in the figure can be further processed to obtain Figure 2c The foreground "little girl" 204 is shown. It can be understood that due to the inaccurate image edge segmentation, the hair edge area 205 of the foreground "little girl" 204 is incomplete, resulting in a poor final display effect. Summary of the invention

[0006] The purpose of the present application is to provide an image processing method, a readable medium and an electronic device.

[0007] The first aspect of the present application provides an image processing method, comprising: determining a first area and a second area of ​​a first image to be processed; generating a modified area of ​​the first area, wherein the modified area includes at least part of the boundary of the first area; correcting the first area based on the modified area to obtain a first corrected area; and segmenting the first corrected area from the first image as a first sub-image of the first image.

[0008] In an embodiment of the present application, the first image may be an original image, the first area may be a foreground area (or background area), the second area may be a background area (or foreground area), the modified area may be a foreground modifiable area (or background modifiable area), the first corrected area is a modified foreground area (or modified background area), and the first sub-image is a foreground image (or background image). That is, in an embodiment of the present application, the electronic device 100 determines a foreground area and a background area from the original image to be processed; generates a foreground modifiable area (or background modifiable area), the modified area includes at least part of the boundary of the foreground area or the boundary of the background area; patches the modified area, and corrects the foreground area (or background area) based on the patched modified area to obtain a patched foreground area (or patched background area); and segments the patched foreground area (or patched background area) from the original image as the foreground image (or background image) of the original image.

[0009] In a possible implementation of the first aspect above, the first area is the foreground area of ​​the first image, and the second area is the background area of ​​the first image; or the first area is the background area of ​​the first image, and the second area is the foreground area of ​​the first image.

[0010] In an embodiment of the present application, the first area may be a foreground area, and the second area may be a background area; or, the first area may be a background area, and the second area may be a foreground area.

[0011] In a possible implementation of the first aspect above, determining a first region and a second region of a first image to be processed includes: determining a first mask of the first image based on a segmentation model, the first mask including a first region having a first color and a second region having a second color.

[0012] In the embodiment of the present application, the first mask may be a primary mask, the first color region may be a white region (or a black region), and the second color region may be a black region (or a white region). The electronic device 100 may determine the primary mask through the segmentation model, and determine the foreground region and the background region based on the primary mask.

[0013] In a possible implementation of the first aspect above, the segmentation model includes at least one of the following: depth matting, path aggregation network model, geodesic distance algorithm, and random walk algorithm.

[0014] In the embodiment of the present application, the segmentation model includes but is not limited to depth cutout, path aggregation network model, geodesic distance algorithm, and random walk algorithm.

[0015] In a possible implementation of the first aspect, generating a modified region of the first region includes: expanding an edge of the first region of the first mask to obtain the modified region.

[0016] In the embodiment of the present application, the electronic device 100 determines the foreground modifiable area by using the edge expansion method, or expands the edge of the foreground area of ​​the first mask to obtain the modified area.

[0017] In a possible implementation of the first aspect, the edge of the first area of ​​the first mask is expanded to obtain a modified area, including: binarizing the pixel values ​​of each pixel in the first mask to obtain a first binary image; determining an edge of a first area with a first color in the first binary image; enlarging the first area according to a first ratio to obtain a first enlarged area, wherein a center of the first enlarged area coincides with a center of the first area; and taking a portion of the first enlarged area that does not include the first area as the modified area.

[0018] In an embodiment of the present application, the first magnified area may be an enlarged foreground area (or an enlarged background area). The electronic device 100 performs binarization processing on the pixel values ​​of each pixel point in the first mask to obtain a first binary image; determines the edge of the foreground area (or the edge of the background modification area) having a white area (or a black area) in the first binary image; enlarges the foreground area (or the background area) according to a first ratio to obtain an enlarged foreground area (or an enlarged background area), wherein the center of the enlarged foreground area (or the enlarged background area) coincides with the center of the foreground area (or the enlarged background area); uses the portion of the enlarged foreground area (or the enlarged background area) that does not include the foreground area (or the background area) as a foreground modifiable area (or a background modifiable area), that is, uses the portion of the enlarged foreground area (or the enlarged background area) that does not overlap with the foreground area (or the background area) as a foreground modifiable area (or a background modifiable area).

[0019] In a possible implementation of the above first aspect, generating a modified region of the first region further includes: determining a lowest threshold and a highest threshold; using the set of pixel points in the first mask whose pixel values are between the lowest threshold and the highest threshold as the modified region.

[0020] In the embodiments of the present application, the lowest threshold may be the foreground lowest threshold (or background lowest threshold), and the highest threshold may be the foreground highest threshold (or background highest threshold). The electronic device 100 finds the set of pixel points (i.e., TL < V(P) < TH) whose probability values (or pixel values) V(P) are between the foreground lowest threshold (TL) and the foreground highest threshold (TH) from the foreground mask (initial mask) as the filling region A (foreground modifiable region); alternatively, the electronic device 100 can also find the set of pixel points (i.e., TL’ < V(P) < TH’) whose probability values (or pixel values) V(P) are between the background lowest threshold (TL’) and the background highest threshold (TH’) from the foreground mask (initial mask) as the filling region B (background modifiable region).

[0021] In a possible implementation of the above first aspect, correcting the first region based on the modified region to obtain a first corrected region includes: correcting the first region based on a generative neural network model to obtain a first corrected region.

[0022] In the embodiments of the present application, the electronic device 100 performs an image repair operation on the foreground modifiable region (or background modifiable region) through a generative neural network model (such as a Stable Diffusion model) to generate a patched foreground region (patched background region).

[0023] In a possible implementation of the above first aspect, the generative neural network model includes at least one of the following: a Stable Diffusion model, a generative adversarial network model, and a partial convolution model.

[0024] In a possible implementation of the above first aspect, it further includes: obtaining a second corrected region based on the generative neural network model.

[0025] In the embodiments of the present application, the second corrected region may be the background region corresponding to the patched foreground. The electronic device 100 obtains the background region corresponding to the patched foreground based on the generative neural network model.

[0026] In a possible implementation of the above first aspect, the generative neural network model obtains the first corrected region and the second corrected region in the following manner: determining the edge of the modified region as the edge of the first corrected region; filling the first corrected region based on the pixel values of the first region to obtain the first corrected region; using the part of the first image that does not include the first corrected region as the second corrected region.

[0027] In an embodiment of the present application, the foreground area to be patched is first determined: for example, the edge of the foreground area to be modified is used as the edge of the foreground area to be patched; a repair operation is performed on the foreground area to be modified, such as filling the foreground area to be modified based on the pixel value of the foreground area; and then the part of the original image that does not include the patched foreground area is used as the background area corresponding to the foreground patch.

[0028] In a possible implementation of the first aspect above, segmenting a first correction area from a first image as a first sub-image of the first image includes: setting a pixel value of a second correction area in the first image to a first pixel value to obtain a second image, wherein a difference between the first pixel value and a pixel value of a pixel point in the first correction area is greater than a pixel threshold; and inputting the second image into a segmentation model to obtain a first sub-image of the first image.

[0029] In an embodiment of the present application, the background area corresponding to the foreground patch is filled with color: the pixel value of the background area corresponding to the foreground patch is set to the first pixel value to obtain a patched image. Finally, the patched image is input into the segmentation model to obtain the patched foreground area. The difference between the first pixel value and the pixel value of the foreground area is greater than the pixel threshold, that is, the foreground is analyzed for color or color layout, and the complementary color of the foreground or the color with a large contrast with the foreground color is calculated as the background fill color.

[0030] In a possible implementation of the first aspect above, the first area is corrected based on the modified area to obtain the first correction area and the second correction area, including: acquiring text control information, where the text control information is related to the image object in the first area; based on the text control information, the image information of the first area, and the modified area, obtaining the first correction area and the second correction area, wherein the image information includes at least one of color features, texture features, shape features, and spatial relationship features.

[0031] In the embodiment of the present application, the text control information may be a text indication, and the image object may be a foreground object. The electronic device 100 obtains the text indication (text control information), and based on the text indication, the image information of the foreground area, and the foreground modifiable area, obtains the patched foreground area and the corresponding background area after the foreground is patched. The text indication is related to the foreground object in the foreground area; the image information includes but is not limited to color features, texture features, shape features, and spatial relationship features.

[0032] In a possible implementation of the first aspect above, the first area is corrected based on the modified area to obtain the first correction area and the second correction area, and it also includes: acquiring image information of the first image and semantic information of the first image, wherein the image information includes at least one of color features, texture features, shape features and spatial relationship features, and the semantic information includes at least one of visual information, object information and conceptual information; based on the image information of the first image and the semantic information of the first image, determining the first correction area and the second correction area.

[0033] In the embodiment of the present application, the semantic information may be a depth map. The electronic device 100 obtains a patched foreground area and a corresponding background area after the foreground is patched based on the image information and semantic information of the original image.

[0034] In a possible implementation of the first aspect above, the semantic information includes at least one of the following: a depth map, a posture map, an edge map, a sketch map, and a normal map.

[0035] In a possible implementation of the first aspect above, determining a first region and a second region of a first image to be processed includes: determining a semantic segmentation map of the first image, and determining a plurality of first sub-regions from the first image based on the semantic segmentation map; taking a first sub-region that satisfies a region condition as a first region, and taking a portion of the first image that does not include the first region as a second region; wherein the region condition includes: an image object in the first sub-region is a target object.

[0036] In an embodiment of the present application, the semantic segmentation map may be a semantic segmentation mask, the first sub-region may be a semantic region, and the image object may be an object. The electronic device 100 generates a semantic segmentation mask, which includes multiple semantic regions; the semantic region that meets the region condition is used as the object to be segmented (i.e., the foreground region), and the part of the original image that does not include the foreground region is used as the background region. Among them, the region condition may be that the object in the semantic region is the target object.

[0037] In a possible implementation of the first aspect above, determining the first area and the second area of ​​the first image to be processed includes: determining a semantic segmentation map of the first image, and determining a plurality of second sub-areas from the first image based on the semantic segmentation map; taking a second sub-area having a third color among the plurality of second sub-areas as the first area; or displaying a plurality of second sub-areas to a user, and taking a second sub-area selected by the user as the first area.

[0038] In an embodiment of the present application, the second sub-region may be a semantic region. The electronic device 100 uses the region corresponding to the target color in the multiple semantic regions as the foreground region. Alternatively, the electronic device 100 displays the multiple semantic regions in the semantic segmentation mask to the user, and uses the semantic region selected by the user as the foreground region.

[0039] A second aspect of the present application provides a readable medium on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the first aspect.

[0040] The third aspect of the present application provides an electronic device, comprising a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing any one of the methods in the above-mentioned first aspect.

[0041] A fourth aspect of the present application provides a program product, which includes instructions. When the instructions are executed by an electronic device, the electronic device can implement any one of the methods in the first aspect.

[0042] The fifth aspect of the present application provides a chip, characterized in that the chip system includes a processing circuit and a storage medium, and the storage medium stores computer program code; when the computer program code is executed by the processing circuit, any one of the methods in the above-mentioned first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figures 1a to 1e According to an embodiment of the present application, a schematic diagram of interface changes of a custom wallpaper scene is shown;

[0045] Figure 2a According to an embodiment of the present application, a schematic diagram of an original image is shown;

[0046] Figure 2b According to an embodiment of the present application, a schematic diagram of a foreground mask obtained according to a neural network model is shown;

[0047] Figure 2c A schematic diagram of a foreground image is shown according to an embodiment of the present application;

[0048] Figure 3aA schematic diagram of a foreground mask is shown according to an embodiment of the present application;

[0049] Figure 3b According to an embodiment of the present application, a schematic diagram of a binary segmentation map is shown;

[0050] Figure 3c According to an embodiment of the present application, a schematic diagram of a semantic segmentation graph is shown;

[0051] Figure 3d According to an embodiment of the present application, a schematic diagram of repairing an input image by generating a neural network model is shown;

[0052] Figure 3e According to an embodiment of the present application, a schematic diagram of a foreground image obtained based on a generative neural network model is shown;

[0053] Figure 4a According to an embodiment of the present application, a schematic diagram of a modification region generated by a dilation operation is shown;

[0054] Figure 4b According to an embodiment of the present application, a schematic diagram of a modification area generated by a double threshold method is shown;

[0055] Figure 4c According to an embodiment of the present application, a schematic diagram of a repaired image is shown;

[0056] Figure 4d According to an embodiment of the present application, a schematic diagram of an image after replacing a background area is shown;

[0057] Figure 4e According to an embodiment of the present application, a schematic diagram of an image after replacing a background area for repair is shown;

[0058] Figure 4f According to an embodiment of the present application, a schematic diagram of another foreground image is shown;

[0059] Figure 4g According to an embodiment of the present application, a flow chart of an image processing method is shown;

[0060] Figure 5 According to an embodiment of the present application, a flow chart of an image processing method for synthesizing a target image based on foreground and background is shown;

[0061] Figure 6a According to an embodiment of the present application, a flow chart of another image processing method for synthesizing a target image based on foreground and background is shown;

[0062] Figure 6b A schematic diagram of an expansion operation is shown according to an embodiment of the present application;

[0063] Figure 7 According to an embodiment of the present application, a schematic diagram of color filling is shown;

[0064] Figure 8a A schematic diagram of content generation is shown according to an embodiment of the present application;

[0065] Figure 8b A schematic diagram of a wallpaper interface is shown according to an embodiment of the present application;

[0066] Fig. 9 According to an embodiment of the present application, a schematic diagram of a process for generating a primary mask is shown;

[0067] Fig.10 According to an embodiment of the present application, a flow chart of an image processing method for generating content based on a control diagram is shown;

[0068] Fig.11 According to an embodiment of the present application, a flow chart of another image processing method for generating content based on a control diagram is shown;

[0069] Figures 12a to 12c According to an embodiment of the present application, a schematic diagram of generating a dynamic wallpaper based on an original image and a depth image is shown;

[0070] Fig.12d According to an embodiment of the present application, a schematic diagram of a target image generated based on a depth image is shown;

[0071] Fig.12e According to an embodiment of the present application, a schematic diagram of the effect of a dynamic wallpaper generated based on a depth image and a target image is shown;

[0072] Figures 13a to 13c According to an embodiment of the present application, a schematic diagram of stretching a target image into a three-dimensional graphic based on a depth image is shown;

[0073] Fig.14 According to an embodiment of the present application, a structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0074] Illustrative embodiments of the present application include, but are not limited to, an image processing method, a readable medium, and an electronic device.

[0075] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and comprehensively described below in conjunction with the accompanying drawings.

[0076] Before introducing the technical solutions involved in the embodiments of the present application, some of the terms included in the embodiments of the present application are explained first.

[0077] (1) Cutout

[0078] Cutting is a method of separating the foreground object from the background of an original image or an original image in a video sequence. Usually, it refers to the method of segmenting an image into foreground and background.

[0079] In some embodiments, due to the blurriness of the edges of foreground objects, an alpha matte is usually used to represent the output result of the cutout. The foreground mask is a probability map (or grayscale map). The probability value of each pixel in the probability map is in the range of [0.0, 1.0]. A probability value of 1.0 indicates that the probability that the corresponding pixel belongs to the foreground is 100%, and a probability value of 0.0 indicates that the probability that the corresponding pixel belongs to the background is 100%. The numerical value between 0.0 and 1.0 indicates the probability that the corresponding pixel belongs to the foreground. The grayscale value of each pixel in the grayscale map is in the integer range of [0, 255]. Each grayscale value indicates the degree to which the corresponding pixel belongs to the foreground object. White (255) indicates that the corresponding pixel belongs to the foreground, and black (0) indicates that the corresponding pixel belongs to the background. Exemplarily, refer to Figure 3a The schematic diagram of the foreground mask shown in FIG. 30 is the foreground mask corresponding to FIG. 301 , wherein each grayscale value indicates the degree to which the corresponding pixel point belongs to the foreground object, white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0080] In some embodiments, for ease of operation, a binary segmentation map is used to represent the output result of the cutout, that is, compared with the foreground mask, the pixel value of each pixel in the binary segmentation map is only 1 or 0 (or 255 or 0). In other words, white (1) indicates that the corresponding pixel belongs to the foreground, and black (0) indicates that the corresponding pixel belongs to the background; or, white (255) indicates that the corresponding pixel belongs to the foreground, and black (0) indicates that the corresponding pixel belongs to the background. For example, refer to Figure 3b The schematic diagram of the binary segmentation map shown is that picture 303 is the binary segmentation map corresponding to picture 301, wherein white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0081] (2) Semantic segmentation

[0082] Semantic segmentation is classification at the pixel level, for example, marking pixels belonging to the same type of object with the same color. The neural network that implements semantic segmentation usually includes two parts: an encoding module (encoder) and a decoding module (decoder). The encoding module is used to extract input image features. For example, image feature extraction can be based on a convolutional neural network (CNN), or feature extraction networks such as a residual network (ResNet), a dense convolutional network (DenseNet), and a feature pyramid network (FPN). The decoding module is used to fuse features of different resolutions proposed by the encoding module and restore them to the size of the original image, which can be achieved through a convolution group.

[0083] For example, reference Figure 3c The schematic diagram of the semantic segmentation map shown is that picture 305 is the semantic segmentation map corresponding to picture 304, wherein "puppy" 306, "little girl" 307 and "little boy" 308 are identified as three different objects and are marked with different colors.

[0084] (3) Mask

[0085] A mask refers to the part that covers part of an image, usually represented by all-black pixels, also known as a mask or a mask.

[0086] (4) Threshold method and double threshold method

[0087] The basic principle of the threshold method is to first determine a threshold, and then divide all pixels in the image into two categories according to the relationship between the grayscale value and the threshold. For example, if the grayscale value of a pixel is greater than the threshold, the pixel is determined to be a foreground class; conversely, if the grayscale value of a pixel is less than or equal to the threshold, the pixel is determined to be a background class. It can be understood that by determining a suitable threshold, the foreground and background of the image can be separated.

[0088] The basic principle of the double threshold method is to first determine two thresholds (low threshold and high threshold), and then divide all pixels in the image into three categories according to the relationship between the grayscale value and the threshold. For example, if the grayscale value of the pixel is less than or equal to the low threshold, the pixel is determined to be a background class; if the grayscale value of the pixel is greater than or equal to the high threshold, the pixel is determined to be a foreground class; if the grayscale value of the pixel is between the low threshold and the high threshold, that is, the grayscale value of the pixel is greater than the low threshold and less than the high threshold, it is necessary to make further judgments according to specific rules. In an embodiment of the present application, if the grayscale value of the pixel is greater than the low threshold and less than the high threshold, it is determined that the pixel belongs to the modified area.

[0089] (5) Generative neural network (GNN)

[0090] Generative neural networks, also known as graph generative networks, are a type of neural network model used to generate graph data. They are widely used in image segmentation, target detection, image restoration and other fields. They use certain rules to recombine the image information (such as pixels) of the input image and output it as the target image.

[0091] In some embodiments, when applied to image restoration, a neural network is generated to extract image features of the non-defective area of ​​the input image, and a restoration operation such as filling or diffusion is performed on the defective area (modifiable area), and the restored target image is output. For example, a regional patch matching the defective area is obtained from the non-defective area of ​​the input image, and the defective area is filled. Or, for example, by diffusing the pixels at the junction of the non-defective area and the defective area, the image information of the non-defective area is transferred to the defective area.

[0092] For example, reference Figure 3d The schematic diagram of repairing the input image by generating a neural network model is shown. The data input to the generating neural network is the input image 30a and the mask 30b corresponding to the modified area. It can be understood that the white area in the mask 30b corresponding to the modified area is the area that can be repaired by the generating neural network. The generating neural network determines the masked image 30c, and repairs (fills or diffuses, etc.) the defective area based on the image information of the non-defective area in the mask image 208. Among them, the foreground 30c1 is the non-defective area, and the white area 30c2 is the defective area (modified area). The output data of the generating neural network is the repaired image 30d.

[0093] It can be understood that the generative neural network models include stable diffusion model (SD), generative adversarial network (GAN), partial convolution (PConv) and other neural network models as well as variants of various neural network models.

[0094] Currently, a neural network-based method is usually used to input a scaled color image (e.g., an image with a resolution of 512x512) into a neural network model and output a foreground mask. Then, the original image is further processed based on the foreground mask to obtain a foreground image, for example, the pixels in the original image corresponding to the background area (black part) of the foreground mask are deleted to obtain the foreground image.

[0095] For example, reference Figure 3bThe picture 303 shown in FIG. 303 is the foreground mask of the picture 301. The pixel points corresponding to the background area (black part) of the picture 303 in the picture 301 are deleted, and the following can be obtained: Figure 3e The picture 309 shown is the foreground image obtained by cutting out the image.

[0096] It can be understood that the neural network model can be a computing model composed of convolutional layers, pooling layers, batch normalization layers, multi-head attention layers, fully connected layers, etc., including but not limited to convolutional neural networks, fully convolutional neural networks (fully convolutional network, FCN), instance segmentation algorithm models (such as Mask-RCNN) and other neural network models.

[0097] In some embodiments, the foreground mask output by the neural network model has a low resolution, and an upsampling method is required to obtain a higher resolution foreground mask.

[0098] However, as mentioned earlier, due to the ambiguity of some images, segmentation through a neural network model may identify part of the background as the foreground, or identify part of the foreground as the background, resulting in inaccurate image edge segmentation, which in turn affects the user experience.

[0099] For example Figure 2a The original image shown is composed of a foreground "little girl" 201 and a background "star" 202. According to the neural network model, Figure 2b The foreground mask 203 shown is based on the foreground mask 203. Figure 2a The original image shown in the figure can be further processed to obtain Figure 2c The foreground "little girl" 204 is shown. It can be understood that due to the ambiguity of the original image, such as the fact that the colors of the foreground "little girl" 201 and the background "stars" 202 are too close, the edge of the obtained foreground mask 203 is not accurate enough, and the foreground and background in the original image are segmented based on the foreground mask 203, resulting in an incomplete hair edge area 205 of the foreground "little girl" 204, resulting in a poor final display effect.

[0100] To solve the above problems, an embodiment of the present application provides an image processing method. Based on obtaining a foreground mask for an original image by using a neural network-based method, a foreground modifiable region of the original image is generated by performing a dilation operation or a double-threshold method on the foreground mask. Then, in the foreground modifiable region, the foreground region of the original image is repaired. For example, a generative neural network model is used to repair the foreground modifiable region of the original image based on the image information of the foreground region of the original image (such as diffusion, filling, etc.) to obtain a repaired foreground region. The repaired foreground region is segmented from the original image as the foreground image of the original image, making the edge of the foreground image more accurate, and thus obtaining a more accurate segmentation effect.

[0101] In some embodiments, the dilation operation may be to expand the white region of the boundary of the foreground region by adding pixels to the perceived boundary of the foreground region. For example, Figure 2b the edge of the white region in the foreground mask 203 shown is expanded to obtain Figure 4a the mask corresponding to the foreground modifiable region shown. The region of the original image at the corresponding position of the white region (region 402) is the foreground modifiable region.

[0102] In some embodiments, generating the foreground modifiable region by the double-threshold method may be: finding the set of pixel points whose probability value (or pixel value) V(P) is between the foreground lowest threshold (TL) and the foreground highest threshold (TH) from the foreground mask (i.e., TL < V(P) < TH) as the foreground modifiable region, where the foreground lowest threshold and the foreground highest threshold are preset values. For example, the foreground lowest threshold is set to 50 and the foreground highest threshold is set to 80. It can be understood that for different images, the foreground lowest threshold and the foreground highest threshold may be different. For example, referring to Figure 4b the schematic diagram of the foreground modifiable region generated by the double-threshold method shown, the set of pixel points with V(P) ≤ TL is the background region, the set of pixel points with TL < V(P) < TH is the foreground modifiable region, and the set of pixel points with V(P) ≥ TH is the foreground region.

[0103] It can be understood that in some other embodiments, in addition to determining the corresponding foreground region through the foreground mask, such as a semantic segmentation map, etc., the present application places no restrictions on this.

[0104] In some embodiments, using a generative neural network model to patch the modified area may be: based on the content (image information) of the foreground area of ​​the original image, performing patching operations such as diffusion of image information and pixel filling on the foreground modifiable area, so that the patched foreground modifiable area and the foreground area form a complete foreground content (patched foreground area). The generative neural network model includes but is not limited to: stable diffusion model stablediffusion, generative adversarial network GAN, partial convolution PConv and other neural network models and variants of various neural network models.

[0105] It can be understood that in other embodiments, during the process of performing the patching operation on the foreground modifiable area, the patching of the image can also be controlled by adding control text or the like.

[0106] For example, Figure 2a As an example, the original image shown in the figure is firstly processed by a neural network-based method to obtain the following Figure 2b As the edge of the foreground mask 203 is not accurate enough, a foreground modifiable area is generated based on the foreground mask 203, for example, the edge of the foreground area (white part) corresponding to the foreground mask 203 is expanded to obtain the following: Figure 4a The foreground mask 401 is shown, wherein the white area 402 corresponds to the foreground modifiable area, and the area 403 corresponds to the foreground area.

[0107] Then, based on the foreground area corresponding to area 403, the foreground modifiable area corresponding to area 402 is drawn through the generative neural network model, and the repaired image is obtained as follows: Figure 4c Image 404 is shown.

[0108] Or, first Figure 2a The background area of ​​the original image shown is replaced with a solid color background, e.g. Figure 4d The image 405 shown in FIG. 4 is then based on the foreground area corresponding to the area 403, the foreground modifiable area corresponding to the area 402 is drawn through the generative neural network model, and the repaired image is obtained as follows: Figure 4e Image 406 is shown.

[0109] Alternatively, based on the foreground area corresponding to area 403, the foreground modifiable area corresponding to area 402 is drawn through the generative neural network model to obtain a patched image as follows: Figure 4c The image 404 is shown, and the background area of ​​the image 404 is replaced with a solid color background, so that Figure 4e Image 406 is shown.

[0110] Then, the foreground area in image 405 or image 406 can be segmented out to obtain Figure 4f The foreground image 407 is shown. It can be understood that Figure 2c Compared to the foreground "little girl" 204 shown, Figure 4f The edge of the foreground image 407 shown is relatively complete, and the display effect is better.

[0111] In some application scenarios, the cutout result does not need to be completely consistent with the original image, such as Figures 1a to 1e In the custom wallpaper scene shown, there is no high requirement for the consistency between the edge of the cutout result (e.g., foreground 103) and the edge of the main object in picture 102, that is, the edge of the foreground 103 does not need to be exactly the same as the edge of the main object in picture 102, and it is only necessary to obtain a foreground image with complete edges that meets user needs.

[0112] It can be understood that for an ambiguous image, the edge of the foreground mask obtained by the neural network model may not be accurate enough, which in turn causes the edge of the foreground image directly determined from the original image by the foreground mask to be inaccurate. Therefore, the image processing method provided in the embodiment of the present application, based on the foreground mask obtained by the neural network model, repairs the foreground content in the edge area (modified area) of the foreground mask, which can improve the accuracy of edge segmentation of ambiguous images.

[0113] It can be understood that the above only introduces the image processing method provided by the embodiment of the present application by taking the foreground image as an example. In other embodiments, if a complete background needs to be obtained, the background can also be repaired by the above-mentioned image processing method to obtain a complete background image. Specifically, it includes: on the basis of obtaining a foreground mask by a neural network-based method for the original image, a background modification area of ​​the original image is generated by performing a dilation operation on the foreground mask or by a double threshold method. Then, in the background modification area, the background area of ​​the original image is repaired, for example, the background modification area of ​​the original image is repaired based on the image information of the background area of ​​the original image using a generative neural network model to obtain a repaired background area, and the repaired background area is segmented from the original image as the background image of the original image, so that the background image is more complete, thereby obtaining a more accurate segmentation effect.

[0114] For example, Figure 4g According to an embodiment of the present application, a flowchart of an image processing method is shown.

[0115] Understandably, Figure 4g Each step of the process shown in the figure is performed by the electronic device 100. Figure 4g The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 4gAs shown, the process includes but is not limited to the following steps:

[0116] S401: Determine a first region and a second region of a first image to be processed.

[0117] In some embodiments, the electronic device 100 determines the first region and the second region from a first image (original image) to be processed.

[0118] It can be understood that the first area can be a foreground area, and the second area can be a background area; or, the first area can be a background area, and the second area can be a foreground area.

[0119] In some embodiments, the electronic device 100 may determine a primary mask (first mask) through a segmentation model, and determine a foreground area and a background area based on the primary mask.

[0120] It can be understood that the foreground area corresponds to the white area in the first mask, and the background area corresponds to the black area in the first mask.

[0121] It can be understood that the segmentation model includes but is not limited to deep cutout, path aggregation network model, geodesic distance algorithm, and random walk algorithm.

[0122] S402: Generate a modified area of ​​the first area.

[0123] In some embodiments, the electronic device 100 generates the modification area through a dilation operation or through a dual threshold method.

[0124] It can be understood that the modification area can be a foreground modifiable area or a background modifiable area. The modification area includes at least part of the boundary of the foreground area or the boundary of the background area.

[0125] S403: Correct the first region based on the modified region to obtain a first corrected region.

[0126] In some embodiments, the electronic device 100 patches the modified area, and corrects the foreground area or the background area based on the patched modified area to obtain a first corrected area (the patched foreground area or the patched background area).

[0127] S404: Segment a first correction region from the first image as a first sub-image of the first image.

[0128] In some embodiments, the electronic device 100 segments the repaired foreground area or the repaired background area from the original image as the first sub-image (foreground image or background image) of the original image.

[0129] The image processing method of the present application can be applied to custom wallpaper scenes, face recognition, image search and other scenes where the cutout results do not need to be completely consistent with the original image.

[0130] For example, it is applied to face recognition to determine the face area from the image and then perform recognition.

[0131] For another example, it is applied to image search to segment the main object from the image, and then further obtain the image features of the main object.

[0132] For another example, when replacing the background of a subject (the subject may be a human body, a building, the sky, a river, etc.) in an image or video, the subject and the background in the original image are first segmented, and then the segmented subject and the background are merged. The background merged with the subject may be a virtual background (not a real image), such as a cartoon background, an automatically generated background, a dynamic background, a solid color background, etc.; the background merged with the subject may also be a background from the same original image as the subject; or the background merged with the subject may also be a background from a different original image than the subject.

[0133] Specifically, you can refer to the custom wallpaper scene Figures 1a to 1e The interface changes shown in the figure. Figure 1a As shown, in response to the user clicking on the "select control" 101 in the custom wallpaper setting interface 10a, the mobile phone 10 displays the following Figure 1b In response to the user clicking on the picture 102 in the picture selection interface 10b, the mobile phone 10 displays the following Figure 1c In the wallpaper adjustment interface 10c shown, it can be understood that the foreground 103 and the background 104 in the wallpaper adjustment interface 10c are obtained by cutting out the picture 102. In response to the user adjusting the size and position of the foreground 103 and the background 104 and clicking the "confirm" control 105, the mobile phone 10 re-integrates the foreground 103 and the background 104 to obtain a new picture, and displays the following. Figure 1d The custom wallpaper setting interface 10d shown in the figure includes a new image 106 obtained by merging and an "apply" control 107. In response to the user clicking the "apply" control 107, the mobile phone 10 displays the following Figure 1e In the wallpaper display interface shown, the wallpaper image is a fused new image 106. It can be understood that the foreground 103 and the background 104 in the wallpaper adjustment interface 10c can also be obtained by cutting out different images.

[0134] The image processing method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, smart large screens, tablet computers, laptop computers, netbooks, personal digital assistants (PDAs), etc. The exemplary embodiments of the electronic devices include but are not limited to those equipped with Or electronic devices with other operating systems. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0135] In order to better understand the technical solution of the embodiment of the present application, Figures 1a to 1e Taking the custom wallpaper scene shown as an example, some technical solutions of the present application are introduced in detail.

[0136] Figure 5 According to an embodiment of the present application, a flow chart of an image processing method for synthesizing a target image based on foreground and background is shown. It can be understood that Figure 5 Each step of the process shown in the figure is performed by the electronic device 100. Figure 5 The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 5 As shown, the process includes but is not limited to the following steps:

[0137] S501: Acquire an original image.

[0138] In some embodiments, the electronic device 100 obtains the image to be processed as an original image, for example Figure 5 The original image 50a is shown.

[0139] It is understood that in some embodiments, the image to be processed may be a high-resolution high-definition picture, and the image to be processed may be preprocessed and the preprocessed image may be used as the original image. The preprocessing includes but is not limited to: format conversion, scaling, cropping, rotation, translation, noise removal, brightness adjustment, grayscale adjustment, contrast adjustment, color temperature adjustment, etc.

[0140] S502: Generate a primary mask.

[0141] In some embodiments, the electronic device 100 segments the original image using a segmentation model to obtain a primary mask, for example Figure 5 The primary mask 50c is shown, and the white area in the primary mask 50c corresponds to the portrait in the image 50b.

[0142] It can be understood that the initial mask 50c is a binary segmentation image, in which white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0143] In some other embodiments, the primary mask may also be a foreground mask, for example Figure 3a Schematic diagram of the foreground mask shown. Picture 302 is the foreground mask corresponding to Picture 301. Among them, each grayscale value represents the degree to which the corresponding pixel belongs to the foreground object. White indicates that the corresponding pixel belongs to the foreground, and black indicates that the corresponding pixel belongs to the background. This application does not limit this.

[0144] It can be understood that the segmentation model can be a computational model composed of convolutional layers, pooling layers, batch normalization layers, multi-head attention layers, fully connected layers, etc., including but not limited to neural network models such as CNN, FCN, Mask-RCNN, etc.

[0145] It can be understood that compared with the original image 50a, the segmentation edge of the initial mask 50c is not accurate enough, and the edge lines of the portrait in the corresponding image 50b are not smooth enough, and the display effect is not good. Therefore, further processing is required.

[0146] S503: Generate the filling area A according to the initial mask.

[0147] In some embodiments, the electronic device 100 generates the filling area A (foreground modifiable area) according to the initial mask. The filling area A is the area where the foreground of the original image can be modified, such as the edge area of the foreground of the original image, such as the white part area in Image 50d.

[0148] In some embodiments, the filling area A can be the area where the foreground of the original image can be modified determined by performing a dilation operation along the edge contour of the initial mask. The dilation operation is to add pixels to the perceived boundary of the object in the image and expand the bright white area in the image. In other embodiments, the filling area A can also be the area determined by the double-threshold method, such as finding the set of pixel points (i.e., TL < V(P) < TH) where the probability value (or pixel value) V(P) is between the foreground minimum threshold (TL) and the foreground maximum threshold (TH) from the foreground mask as the filling area A.

[0149] It can be understood that this application does not specifically limit the method for determining the filling area A. The filling area A can cover part of the foreground area and part of the background area, or can cover the entire background area.

[0150] S504: Generate the filling area B according to the initial mask.

[0151] In some embodiments, the electronic device 100 generates the filling area B (background modifiable area) according to the initial mask. The filling area B is the area where the background of the original image can be modified, such as the white part area in Image 50e.

[0152] In some embodiments, the filled region B can be a region where the background of the original image determined by performing a dilation operation along the contour of the primary mask edge can be modified.

[0153] In other embodiments, the filled region B can also be a region determined by a double-threshold method. For example, a set of pixels (i.e., TL’ < V(P) < TH’) where the probability value (or pixel value) V(P) of the foreground mask is between the background minimum threshold (TL’) and the background maximum threshold (TH’) is found as the filled region B.

[0154] It can be understood that the present application does not specifically limit the method for determining the filled region B. The filled region B can cover part of the foreground region and part of the background region, or can cover the entire foreground region. The filled region B and the filled region A can partially overlap or be completely the same.

[0155] S505: Generate foreground content within the filled region A.

[0156] In some embodiments, within the filled region A, the electronic device 100 uses a generative neural network model such as a stable diffusion model or a generative adversarial network to generate new foreground content. For example, the generated foreground can refer to Figure 5 the image 50f in

[0157] In some embodiments, taking the stable diffusion model as an example, an inpainting operation is performed on the filled region A to generate a new image. The inpainting operation targets the foreground modifiable region (such as the filled region A), and uses partial differential equations to diffuse the image, transferring the image information of the known region to the region to be repaired. For example, the filled region A is patched based on the image information of the foreground region of the original image.

[0158] It can be understood that to make the generated foreground easy to segment, the region to be generated and the background region of the original image can be filled with a green screen background or a background of other colors together.

[0159] S506: Generate background content within the filled region B.

[0160] In some embodiments, within the filled region B, the electronic device 100 uses a stable diffusion model, a generative adversarial network, or other neural network models to generate new background content. For example, the generated background can refer to Figure 5 the image 50g in

[0161] For example, taking the stable diffusion model as an example, an image restoration operation is performed on the filled area B to generate a new image. The image restoration operation is performed on the background modifiable area (such as the filled area B), and the image is diffused using a partial differential equation to transfer the image information of the known area to the area to be restored. For example, the filled area B is repaired based on the image information of the background area of ​​the original image.

[0162] It can be understood that during the generation process, in order to avoid interference from the foreground, the filling area B can be set to include the entire foreground area, or the foreground area can be filled with a solid color.

[0163] S507: Update the mask.

[0164] In some embodiments, the electronic device 100 generates a target mask according to the result of the foreground filling in step S505. For example, the target mask can refer to Figure 5 The target mask in 50h.

[0165] It can be understood that since the foreground is generated on a simple background such as a green screen, an accurate target mask can be obtained; since the background is relatively simple, more complex methods such as neural network models can be used for segmentation instead of using classic fast algorithms such as geodesic distance, random walk, and confidence propagation algorithms.

[0166] It can be understood that, compared with the initial mask 50c, the edge lines of the segmentation edges of the target mask 50h are smoother, the segmentation effect is better, and the display effect is better.

[0167] S508: Synthesize a new image.

[0168] In some embodiments, the electronic device 100 synthesizes a new image (target image) based on the generated foreground, background, and target masks. For example, the synthesized new image can refer to Figure 5 Image 50i.

[0169] In some embodiments, the target image can be obtained by using the foreground-background separation equation I=F*M+B*(1-M).

[0170] Among them, I represents the target image, F represents the foreground image, B represents the background image, and M represents the target mask.

[0171] Understandably, Figure 5 The process shown only takes the example of segmenting the foreground and background of the same original image and then fusing them to obtain the target image to introduce the image processing method provided in the embodiment of the present application. In other embodiments, the target image can also be obtained by fusing the foreground and background of different original images, and the present application does not impose any restrictions on this.

[0172] In order to better understand the technical solution of the embodiment of the present application, the image processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0173] Embodiment 1

[0174] Figure 6a According to an embodiment of the present application, a flow chart of an image processing method for synthesizing a target image based on foreground and background is shown. It can be understood that Figure 6a Each step of the process shown in the figure is performed by the electronic device 100. Figure 6a The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 6a As shown, the process includes but is not limited to the following steps:

[0175] S601: Acquire an original image.

[0176] In some embodiments, the electronic device 100 obtains an image to be processed, performs preprocessing and resolution reduction operations on the image to be processed, and uses the preprocessed image as the original image, for example Figure 6a The original image 60a is shown.

[0177] It can be understood that preprocessing includes but is not limited to: format conversion, scaling, cropping, rotation, translation, noise removal, brightness adjustment, grayscale adjustment, contrast adjustment, color temperature adjustment, etc.

[0178] S602: Generate a primary mask.

[0179] In some embodiments, the electronic device 100 segments the original image through deep image matting to obtain a primary mask (ie, mask image P), for example Figure 6a A primary mask 60b is shown.

[0180] It can be understood that the primary mask 60b is a foreground mask, for example Figure 3a The schematic diagram of the foreground mask shown in FIG. 30 is the foreground mask corresponding to FIG. 301 , wherein each grayscale value indicates the degree to which the corresponding pixel point belongs to the foreground object, white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0181] In some other embodiments, the initial mask may also be a binary segmentation image, in which white indicates that the corresponding pixel belongs to the foreground, and black indicates that the corresponding pixel belongs to the background. This application does not impose any limitation on this.

[0182] In some other embodiments, the electronic device 100 also segments the original image through artificial neural network models such as the path aggregation network (PANet) and fast algorithms such as geodesic distance algorithms and random walks to obtain a primary mask, and the present application does not make specific limitations thereto.

[0183] It can be understood that since the subsequent steps will also determine the modified area of the original image and perform further processing, the requirement for the edge accuracy of the primary mask is relatively low. Therefore, in step S602, fast algorithms such as geodesic distance algorithms and random walks can be used to segment the original image to obtain the primary mask, thereby improving the execution rate of the entire image processing process.

[0184] S603: Generate a filling area A according to the primary mask.

[0185] In some embodiments, the electronic device 100 determines the filling area A (foreground modifiable area) through a dual-threshold method to obtain a foreground mask (i.e., a mask MA), such as the foreground mask shown in Image 60c. Among them, the filling area A is the area where the foreground of the original image can be modified, such as the edge area of the foreground of the original image, such as the white part area in Image 60c.

[0186] It can be understood that determining the filling area A through the dual-threshold method may include: finding the set of pixel points whose probability value (or pixel value) V(P) is between the foreground minimum threshold (TL) and the foreground maximum threshold (TH) from the foreground mask (i.e., TL < V(P) < TH), as the filling area A. The filling area A may cover part of the foreground area and part of the background area, or may cover the entire background area.

[0187] It can be understood that in some other embodiments, the electronic device 100 may also find the set of pixel points whose pixel value V(P) belongs to the foreground from the foreground mask (i.e., V(P) < TH), as the filling area A. That is, directly take the foreground area as the foreground modifiable area.

[0188] In some other embodiments, the electronic device 100 may also determine the filling area A through an edge dilation method, and determine the area where the foreground of the original image can be modified by performing a dilation operation along the edge contour of the primary mask. The dilation operation is to add pixels to the perceived boundary of the object in the image and expand the bright white area in the image. For example Figure 6b As shown in the schematic diagram of the dilation operation, perform binary processing on the pixel value V(P) using a fixed threshold or an adaptive threshold method, detect the edge contour C of the object, and perform a filling operation along the edge contour C of the object using the rolling circle method (the radius of the circle is r) to obtain the filling area A, such as the white part area in Image 60c.

[0189] It can be understood that the binarization of the pixel value V(P) using a fixed threshold can be: determining the pixel threshold, if the pixel value V(P) is greater than the pixel threshold, setting the value of the pixel value V(P) to 255 (white); if the pixel value V(P) is less than or equal to the pixel threshold, setting the value of the pixel value V(P) to 0 (black).

[0190] It can be understood that the binarization processing of the pixel value V(P) using the adaptive threshold method can be: dividing the original image into multiple areas, determining the pixel threshold of each area, and binarizing the pixel value of each area. If the pixel value V(P) is greater than the pixel threshold of the corresponding area, the value of the pixel value V(P) is set to 255 (white); if the pixel value V(P) is less than or equal to the pixel threshold of the corresponding area, the value of the pixel value V(P) is set to 0 (black).

[0191] In other embodiments, the electronic device 100 determines the foreground modifiable area by the edge expansion method or by expanding the edge of the foreground area of ​​the first mask to obtain the modified area. Specifically, it includes: the electronic device 100 performs binarization processing on the pixel values ​​of each pixel point in the first mask to obtain a first binary image; determines the edge of the white area in the first binary image (the edge of the foreground area); enlarges the foreground area according to a first ratio to obtain an enlarged foreground area, wherein the center of the enlarged foreground area coincides with the center of the foreground area; and uses the part of the enlarged foreground area that does not include the foreground area as the foreground modifiable area, that is, uses the part of the enlarged foreground area that does not overlap with the foreground area as the foreground modifiable area. It can be understood that the first ratio can be a number greater than 1.

[0192] It can be understood that by determining the filling area A through the double threshold method or the edge dilation method, an accurate foreground modifiable area can be obtained, thereby improving the generation effect of the foreground content.

[0193] S604: Generate a filling area B according to the initial mask.

[0194] In some embodiments, the electronic device 100 determines and generates a fill area B (background modifiable area) by a double threshold method to obtain a background mask (i.e., a mask image MB), such as the background mask shown in image 60d. The fill area B is an area in the original image where the background can be modified, such as the white area in image 60d.

[0195] It can be understood that determining the filling area B through the double-threshold method may include: finding, from the foreground mask, the set of pixel points whose probability values (or pixel values) V(P) are between the background lowest threshold (TL’) and the background highest threshold (TH’), that is, TL’ < V(P) < TH’), as the filling area B. The filling area B may cover part of the foreground area and part of the background area, or may cover the entire foreground area. The filling area B and the filling area A may partially overlap or may be exactly the same.

[0196] It can be understood that in some other embodiments, the electronic device 100 may also find, from the foreground mask, the set of pixel points whose pixel values V(P) belong to the background (that is, V(P) > TL’), as the filling area B. That is, directly use the background area as the background modifiable area.

[0197] In some other embodiments, the electronic device 100 may also determine the filling area B by the method of edge dilation. The method of edge dilation may refer to the relevant description in step S603 above and will not be elaborated here.

[0198] It can be understood that determining the filling area B through the double-threshold method or the method of edge dilation can obtain an accurate background modifiable area, thereby improving the generation effect of the background content.

[0199] S605: Generate foreground content within the filling area A.

[0200] In some embodiments, within the filling area A, the electronic device 100 performs an image restoration operation on the filling area A through a generative neural network model (such as the StableDiffusion model) to generate the foreground F (the repaired foreground area). For example, the generated foreground F may refer to Figure 6a the image 60e in

[0201] It can be understood that the image restoration operation is performed on the modifiable area (such as the filling area A), and the image is diffused using partial differential equations to transfer the image information of the known area to the area to be repaired. For example, the filling area A is repaired based on the image information of the foreground area of the original image.

[0202] It can be understood that in some embodiments, to make the generated foreground easy to segment, the area to be generated and the background area of the original image can be filled with a green screen background or a background of other colors. When generating foreground content, the foreground can be pre-identified or the color layout of the foreground can be analyzed, and a complementary color or a color with a large contrast can be used as the background filling color to replace the background area in the original image with the background filling color.

[0203] For example Figure 7The schematic diagram of color filling is shown. The double-threshold method is used for the original image 70a to determine the filling area shown in the image 70b. It can be understood that the set of pixel points where V(P) ≤ TL is the background area, the set of pixel points where TL < V(P) < TH is the foreground modifiable area, and the area of pixel points where V(P) ≥ TH is the foreground area; referring to the reference image 70c, the foreground is recognized as a person by an artificial intelligence method, and the background is filled with a solid color background (such as green); referring to the reference image 70d, foreground content is generated within the filling area A.

[0204] It can be understood that if the foreground is recognized as green plants, the background filling color usually uses blue or red; it is also possible not to recognize the foreground, but only to perform color or color layout analysis on the foreground, such as the main color method or the color layout method, and calculate the complementary color of the foreground or a color with a large contrast to the foreground color as the background filling color.

[0205] It can be understood that in some other embodiments, the background filling color may not be a solid color either. The present application does not limit the specific color of the background filling color, as long as the foreground and the background can be distinguished.

[0206] In some other embodiments, it is also possible to first determine the foreground area after repair: for example, take the edge of the foreground area to be modified as the edge of the foreground area after repair; perform a repair operation on the foreground area to be modified, such as filling the foreground area to be modified based on the pixel values of the foreground area; then take the part of the original image that does not include the foreground area after repair as the background area corresponding to the foreground after repair. Then, perform color filling on the background area corresponding to the foreground after repair: set the pixel values of the background area corresponding to the foreground after repair to the first pixel value to obtain the repaired image. Finally, input the repaired image into the segmentation model, and the foreground area after repair can be obtained.

[0207] It can be understood that the difference between the first pixel value and the pixel values of the foreground area is greater than the pixel threshold, that is, perform color or color layout analysis on the foreground, and calculate the complementary color of the foreground or a color with a large contrast to the foreground color as the background filling color.

[0208] It can be understood that by performing color filling on the background, a large contrast is generated between the generated foreground and the background, so that the segmentation result is more accurate. And after generating the foreground content on the picture after background color filling, the foreground can be segmented out through a simple segmentation algorithm.

[0209] In some embodiments, when generating the foreground F through the Stable Diffusion model, positive text instructions (or positive prompts) and negative text instructions (or negative prompts) can also be input, where the positive text instructions represent the content that is expected to appear, and the negative text instructions represent the content that is not expected to appear, so as to control the output result.

[0210] That is, the electronic device 100 obtains the text indication (text control information), and obtains the repaired foreground area and the corresponding background area after the foreground is repaired based on the text indication, the image information of the foreground area, and the foreground modifiable area.

[0211] It can be understood that the text indication is related to the foreground object in the foreground area; the image information includes but is not limited to color features, texture features, shape features and spatial relationship features.

[0212] It can be understood that the text indication can be in the form of a template library, which can be pre-stored on an electronic device or the cloud. When the content of the original image is recognized, the relevant text indication or the user's preferred text indication is queried in the template library as the input of the smooth diffusion model, and the original image is processed.

[0213] For example Figure 8a As shown in the generation schematic diagram, the original image refers to the original image 80a, and the mask of the original image refers to the mask 80b; the image generated after inputting "positive indication: long-tailed puppy, negative indication: cartoon" refers to the image 80c, and the corresponding mask refers to the mask 80d; the image generated after inputting "positive indication: short-legged puppy, negative indication: cartoon" refers to the image 80e, and the corresponding mask refers to the mask 80f.

[0214] It can be understood that by adding text indicators in the process of generating foreground content to control the generation effect of the foreground content, such as inputting user-preferred labels as positive indicators, the generation result can be closer to the user's preference, thereby improving the user experience.

[0215] For example, reference Figure 8b The schematic diagram of the wallpaper interface shown in FIG. 801 shows the foreground "puppy" in the interface 801. Figure 8b The image 80c in the interface 802 is the foreground "puppy". Figure 8b Image 80e in .

[0216] S606: Generate background content in the filling area B.

[0217] In some embodiments, the electronic device 100 performs an image restoration operation on the fill area B using a stable diffusion model to generate a background B. For example, the generated background B can refer to Figure 6a Image 60f in.

[0218] In other embodiments, the electronic device 100 can also generate a foreground F in the filled area A by generating adversarial networks, partial convolutional neural networks and other neural network models and variants of various neural network models (such as StyleGAN, DragGAN), and the present application does not impose any restrictions on this.

[0219] It can be understood that the image restoration operation is for the modifiable area (such as the filled area B), using the partial differential equation to diffuse the image and transfer the image information of the known area to the area to be restored. For example, the filled area B is repaired based on the image information of the background area of ​​the original image.

[0220] It is understood that in order to avoid interference from the foreground during the generation process, the filling area B can be set to include the entire foreground area, or the foreground area can be filled with a solid color. For filling the foreground area with a solid color, reference can be made to the relevant description in the above step S605, which will not be repeated here.

[0221] It is understood that in other embodiments, the foreground fill color may not be a pure color. The present application does not limit the specific color of the foreground fill color, as long as the foreground and the background can be distinguished.

[0222] It can be understood that by filling the foreground with color, the generated background has a greater contrast with the foreground, thereby making the segmentation result more accurate.

[0223] S607: Update the mask.

[0224] In some embodiments, the electronic device 100 generates a target mask (mask image P') by using the geodesic distance method according to the result of the foreground filling in step S605, and sets the pixel points in the foreground F corresponding to the pixel values ​​greater than or equal to TH in the initial mask (i.e., the mask image P) as the foreground, and the pixel points in the foreground F corresponding to the pixel values ​​less than or equal to TL as the background. For example, the target mask can refer to Figure 6a The target mask 60g is shown in FIG.

[0225] It can be understood that since the foreground is generated on a simple background such as a green screen, an accurate target mask can be obtained; since the background is relatively simple, more complex methods such as neural network models can be used for segmentation, but the geodesic distance method can be used for segmentation.

[0226] S608: Synthesize a new image.

[0227] In some embodiments, the electronic device 100 synthesizes a new image (target image) based on the generated foreground F, background B and target mask P'. For example, the synthesized new image can refer to Figure 6a Image in 60h.

[0228] In some embodiments, the target image can be obtained by the foreground-background separation equation I=F*P'+B*(1-P').

[0229] Among them, I represents the target image, F represents the foreground image, B represents the background image, and P' represents the target mask.

[0230] Understandably, Figure 6a The process shown only takes the example of segmenting the foreground and background of the same original image and then fusing them to obtain the target image to introduce the image processing method provided in the embodiment of the present application. In other embodiments, the target image can also be obtained by fusing the foreground and background of different original images, and the present application does not impose any restrictions on this.

[0231] Understandably, Figure 6a The flowchart shown only takes the scenario of customizing wallpaper as an example. In other application scenarios, such as image search, the above steps S604 and S606 can be deleted, and step S608 can be set to segment the foreground F based on the target mask (mask image P') to obtain the foreground image as the target image.

[0232] In some embodiments, when the foreground includes multiple subjects, Figure 6a The process of generating the initial mask in step S602 may also adopt a multi-instance segmentation method, where the electronic device 100 or the user selects the object to be segmented and generates a corresponding mask.

[0233] Fig. 9 According to an embodiment of the present application, a schematic diagram of a process for generating a primary mask is shown. It can be understood that Fig. 9 Each step of the process shown in the figure is performed by the electronic device 100. Fig. 9 The execution entities of each step will not be described repeatedly in the steps of the process shown. Fig. 9 As shown, the process includes but is not limited to the following steps:

[0234] S901: Acquire an original image.

[0235] In some embodiments, the electronic device 100 obtains an image to be processed, performs preprocessing and resolution reduction operations on the image to be processed, and uses the preprocessed image as the original image, for example Fig. 9 The original image 90a is shown.

[0236] It can be understood that preprocessing includes but is not limited to: format conversion, scaling, cropping, rotation, translation, noise removal, brightness adjustment, grayscale adjustment, contrast adjustment, color temperature adjustment, etc.

[0237] S902: Generate a semantic segmentation mask.

[0238] In some embodiments, the electronic device 100 outputs a semantic segmentation mask through a path aggregation network model, and the semantic segmentation mask includes multiple semantic regions. Fig. 9 The semantic segmentation mask 90b is shown, which includes “tree”, “person” and “sky”.

[0239] S903: Determine the object to be segmented and generate a primary mask.

[0240] In some embodiments, the electronic device 100 determines an object to be segmented and generates a corresponding mask. For example, the electronic device 100 determines that the “person” in the semantic segmentation mask 90b is an object to be segmented and generates a corresponding mask 90c.

[0241] In some embodiments, the electronic device 100 uses the semantic region that meets the region condition as the object to be segmented (i.e., the foreground region), and uses the portion of the original image that does not include the foreground region as the background region. The region condition may be that the object in the semantic region is the target object.

[0242] In some other embodiments, the electronic device 100 uses the area corresponding to the target color in the multiple semantic areas as the foreground area.

[0243] In other embodiments, the electronic device 100 displays multiple semantic regions in the semantic segmentation mask to the user, and uses the semantic region selected by the user as the foreground region.

[0244] It can be understood that the first mask is the foreground mask, e.g. Figure 3a The schematic diagram of the foreground mask shown in FIG. 30 is the foreground mask corresponding to FIG. 301 , wherein each grayscale value indicates the degree to which the corresponding pixel point belongs to the foreground object, white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0245] In some other embodiments, the initial mask may also be a binary segmentation image, in which white indicates that the corresponding pixel belongs to the foreground, and black indicates that the corresponding pixel belongs to the background. This application does not impose any limitation on this.

[0246] It can be understood that the above-mentioned multi-instance segmentation method, in which the electronic device 100 or the user selects the object to be segmented and generates a corresponding mask, can segment multiple foreground objects in the original image containing multiple foreground objects in turn to obtain a foreground image corresponding to each foreground object.

[0247] Embodiment 2

[0248] Fig.10 According to an embodiment of the present application, a flow chart of an image processing method for generating content based on a control diagram is shown. It can be understood that Fig.10 Each step of the process shown in the figure is performed by the electronic device 100. Fig.10 The execution entities of each step will not be described repeatedly in the steps of the process shown. Fig.10 As shown, the process includes but is not limited to the following steps:

[0249] S1001: Acquire an original image.

[0250] In some embodiments, the electronic device 100 obtains an image to be processed, performs preprocessing and resolution reduction operations on the image to be processed, and uses the preprocessed image as the original image, for example Fig.10 The original image 100a is shown.

[0251] It can be understood that preprocessing includes but is not limited to: format conversion, scaling, cropping, rotation, translation, noise removal, brightness adjustment, grayscale adjustment, contrast adjustment, color temperature adjustment, etc.

[0252] S1002: Generate a control chart.

[0253] In some embodiments, the electronic device 100 outputs a semantic segmentation mask through a path aggregation network model, for example Fig.10 The semantic segmentation mask 100b shown includes “tree”, “person” and “sky”.

[0254] S1003: Determine the object to be segmented.

[0255] In some embodiments, the electronic device 100 determines an object to be segmented and determines a corresponding object mask. For example, the “person” in the semantic segmentation mask 100b is determined as an object to be segmented, and an object mask 100c corresponding to the “person” is determined.

[0256] It can be understood that the object mask is a foreground mask, e.g. Figure 3a The schematic diagram of the foreground mask shown in FIG. 30 is the foreground mask corresponding to FIG. 301 , wherein each grayscale value indicates the degree to which the corresponding pixel point belongs to the foreground object, white indicates that the corresponding pixel point belongs to the foreground, and black indicates that the corresponding pixel point belongs to the background.

[0257] In some other embodiments, the object mask may also be a binary segmentation map, in which white indicates that the corresponding pixel belongs to the foreground, and black indicates that the corresponding pixel belongs to the background. This application does not impose any limitation on this.

[0258] S1004: Generate filling area A.

[0259] In some embodiments, the electronic device 100 determines the fill area A (foreground modifiable area) by a double threshold method to obtain a foreground mask (i.e., a mask image MA), such as the foreground mask shown in image 100d. The fill area A is an area of ​​the foreground of the original image that can be modified, such as an edge area of ​​the foreground of the original image, such as the white area in image 100d.

[0260] It can be understood that determining the filling area A through the double-threshold method may include: finding the set of pixel points in the foreground mask whose probability value (or pixel value) V(P) is between the foreground lowest threshold (TL) and the foreground highest threshold (TH) (i.e., TL < V(P) < TH), and using it as the filling area A. The filling area A may cover part of the foreground area and part of the background area, or may cover the entire background area.

[0261] It can be understood that in some other embodiments, the electronic device 100 may also find the set of pixel points in the foreground mask whose pixel value V(P) belongs to the foreground (i.e., V(P) < TH), and use it as the filling area A. That is, directly use the foreground area as the foreground modifiable area.

[0262] S1005: Generate content within the filling area A.

[0263] In some embodiments, the electronic device 100 generates a target image through the ControlNet method. For example Fig.10 the new image 100e shown.

[0264] It can be understood that the foreground mask MA is used to control the fidelity between the target image and the original image, so that content generation only occurs in the local area at the edge of the foreground, and the main bodies of the foreground and the background will not change.

[0265] It can be understood that compared with the method corresponding to the Figure 5 flowchart shown, Fig.10 in the method shown, the object mask is obtained by primary segmentation, directly controls the content to generate the target image, and does not require secondary segmentation, simplifying the segmentation steps.

[0266] Embodiment III

[0267] Fig.11 According to the embodiments of the present application, a schematic flowchart of another image processing method for generating content based on a control graph is shown. It can be understood that Fig.11 the execution subject of each step in the shown process is the electronic device 100. For the sake of simplicity of description, the execution subject of each step will not be repeatedly described hereinafter when introducing Fig.11 each step of the shown process. As Fig.11 shown, this process includes but is not limited to the following steps:

[0268] S1101: Obtain the original image.

[0269] In some embodiments, the electronic device 100 obtains an image to be processed, performs preprocessing and downsampling operations on the image to be processed, and uses the preprocessed image as the original image. For example Fig.11 the original image 110a shown.

[0270] It can be understood that preprocessing includes but is not limited to: format conversion, scaling, cropping, rotation, translation, noise removal, brightness adjustment, grayscale adjustment, contrast adjustment, color temperature adjustment, etc.

[0271] S1102: Generate a primary mask.

[0272] In some embodiments, the electronic device 100 segments the original image through depth matting to obtain a primary mask (ie, mask image P), for example Fig.11 The primary mask 110b is shown.

[0273] It can be understood that the primary mask 110b is a foreground mask, for example Figure 3a The schematic diagram of the foreground mask shown in FIG. 30 is the foreground mask corresponding to FIG. 301 , wherein each grayscale value indicates the degree to which the corresponding pixel point belongs to the foreground object, white indicates that the corresponding pixel point belongs to the foreground area, and black indicates that the corresponding pixel point belongs to the background area.

[0274] In some other embodiments, the initial mask may also be a binary segmentation map, in which white indicates that the corresponding pixel point belongs to the foreground area, and black indicates that the corresponding pixel point belongs to the background area. This application does not impose any limitation on this.

[0275] In other embodiments, the electronic device 100 also segments the original image to obtain a primary mask through artificial neural network models such as path aggregation network models and fast algorithms such as geodesic distance algorithm and random walk, and the present application does not impose specific restrictions on this.

[0276] S1103: Generate a control chart.

[0277] In some embodiments, the electronic device 100 generates a depth image based on the original image, for example, by using a depth estimation algorithm (such as the Midas algorithm). Fig.11 Depth image 110c is shown.

[0278] It can be understood that a depth image is also called a range image, which refers to an image in which the distance (depth) value of each point in the image collected by the image collector, or the distance (depth) value of each point in the image calculated by the depth estimation algorithm, is used as a pixel value. The closer the distance (the closer the object is), the brighter the corresponding pixel points (the larger the pixel value).

[0279] It can be understood that the control graph contains the semantic information of the original image, and the semantic information includes but is not limited to visual information, object information and concept information. In other embodiments, the control graph can also be other semantic information images such as posture graph, edge graph, sketch graph, normal graph, etc., and this application does not limit this.

[0280] S1104: Generate the filling area A.

[0281] In some embodiments, the electronic device 100 determines the filling area A (foreground modifiable area) by the dual-threshold method to obtain a foreground mask (i.e., the mask image MA), such as the foreground mask shown in the image 110d. Among them, the filling area A is the area where the foreground of the original image can be modified, such as the edge area of the foreground of the original image, such as the white area in the image 50c.

[0282] It can be understood that determining the filling area A by the dual-threshold method may include: finding the set of pixel points whose probability value (or pixel value) V(P) is between the foreground lowest threshold (TL) and the foreground highest threshold (TH) from the foreground mask (i.e., TL < V(P) < TH), as the filling area A. The filling area A may cover part of the foreground area and part of the background area, or may cover the entire background area.

[0283] It can be understood that in some other embodiments, the electronic device 100 may also find the set of pixel points whose pixel value V(P) belongs to the foreground from the foreground mask (i.e., V(P) < TH), as the filling area A. That is, directly use the foreground area as the foreground modifiable area.

[0284] S1105: Generate content within the filling area A.

[0285] In some embodiments, the electronic device 100 generates a target image by controlling the diffusion model method, and generates the target image based on the foreground mask and the control map. For example Fig.11 the new image 110e shown.

[0286] That is, the electronic device 100 obtains the repaired foreground area and the background area corresponding to the repaired foreground based on the image information and semantic information of the original image.

[0287] It can be understood that the foreground mask MA and the control map are used to control the fidelity of the target image to the original image, so that content generation only occurs in the local edge area of the foreground, and the main bodies of the foreground and background will not change.

[0288] S1106: Update the mask.

[0289] In some embodiments, the electronic device 100 generates a target mask (mask image P') by a segmentation method of a four-channel image (RGBD) based on depth information according to the target image generated in the above step S1105. For example, the target mask may refer to Fig.11 the target mask 110f in.

[0290] It is understandable that in other embodiments, the segmentation result is not needed and step S1106 can be deleted, for example, a scene of generating a three-dimensional (3D) wallpaper based on a single image, where the 3D wallpaper can be a 3D relief wallpaper or a 3D animated wallpaper.

[0291] For example, Figures 12a to 12c A schematic diagram of generating a target image based on a depth map is shown. For example, the original image is Fig.12a The image 120a shown in FIG. 1 and the depth image corresponding to the image 120a are shown in FIG. Figure 12b As shown in the depth image 120b, it can be understood that, according to the depth value (distance value), the elements in the depth image 120b are sorted from far to near according to the distance: water, lotus leaf, fish.

[0292] It is understandable that the depth estimation based on the original image may not be accurate enough, resulting in inaccurate depth information in the depth map. For example, the color of the tail of the element fish in image 120a is too close to the color of the element water (background). When estimating the depth, part of the tail of the fish may be identified as water content in depth, resulting in the missing tail of the element fish in the depth image 120b obtained based on the depth estimation. If a 3D dynamic wallpaper is generated directly based on image 120a and depth image 120b, the dynamic effect in the obtained 3D dynamic wallpaper may be fragmented, for example Fig.12c As shown in a frame image 120c of the dynamic wallpaper, the fish body 1201 and the fish tail 1202 are visually separated.

[0293] Therefore, it is necessary to control the modification of the original image based on the depth image to generate the target image, such as Fig.12d The target image 120d is shown. It can be understood that the depth image 120b corresponds to the accurate depth information of the target image 120d. The dynamic effect of generating a 3D dynamic wallpaper based on the target image 120d and the depth image 120b can be referred to Fig.12e As shown in the effect diagram, the segmentation of the fish 1203 is accurate without any fragmentation.

[0294] In other embodiments, a stretching (extrude) operation of a target image is controlled based on the depth image to obtain a three-dimensional graphic corresponding to the target image, and the three-dimensional graphic is applied to a wallpaper to obtain a 3D relief wallpaper.

[0295] For example, Figures 13a to 13c A schematic diagram of stretching a target image into a three-dimensional image based on a depth image is shown. For example, the target image is Fig.13a As shown in 130a, the depth image corresponding to the image 130a is as follows Fig.13bAs shown in the depth image 130b, it can be understood that according to the depth value (distance value), the elements in the depth image 130b are sorted from far to near in terms of distance: water, lotus leaf, fish. Based on the depth image 130b, the target image 130a is stretched to obtain the following: Fig.13c The three-dimensional graph 130c shown in FIG. 130c can be understood that controlling the segmentation of the image based on the depth image can improve the accuracy of edge segmentation.

[0296] It can be understood that by controlling the content generation of the image modification area and the segmentation of the image through semantic information images, such as depth maps, posture maps, edge maps, sketch maps, normal maps, etc., the generation and segmentation results can be enriched, and thus applied to more scenarios, such as the above-mentioned scenario of generating 3D wallpaper based on a single image.

[0297] In summary, the image processing method provided in the embodiment of the present application performs a primary segmentation on the original image to obtain a primary mask, and then generates a modified area based on the primary mask, and repairs the foreground and / or background of the original image in the modified area to obtain a more accurate segmentation effect. By repairing the foreground and / or background content based on the primary mask obtained by the primary segmentation and then performing the segmentation, the accuracy of the edge segmentation of the ambiguous image can be improved.

[0298] It can be understood that the image processing method provided in the embodiment of the present application can also be used for video processing. For example, in the case of barrage obstruction during video playback, the main object needs to be separated and set to be displayed on the upper layer of the layer where the barrage is located. By performing frame processing on the video data, multiple video frame data are obtained, and each video frame data is used as a single image, thereby converting the video processing problem into an image processing problem.

[0299] In some embodiments, on the basis of the image processing method provided in the present application, a timing constraint may be added (eg, the time for generating a new image must be after the foreground content filling and the background content filling are completed).

[0300] In order to better understand the technical solutions of the embodiments of the present application, the structure of the electronic device involved in the present application is introduced below with reference to the accompanying drawings.

[0301] For example, Fig.14 According to an embodiment of the present application, a structural schematic diagram of an electronic device 100 is shown.

[0302] like Fig.14As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, an interface module 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0303] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0304] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. The processor 110 can be used to execute the image processing method provided in the embodiment of the present application.

[0305] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0306] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0307] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0308] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 charges the battery 142, it can also power the electronic device through the power management module 141.

[0309] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.

[0310] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0311] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0312] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0313] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TDSCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0314] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0315] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-Led, Micro-Led, Micro-oLed, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In some embodiments, the display screen 194 is used to display images such as original images and target images.

[0316] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0317] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0318] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. In some embodiments, the electronic device 100 may obtain an original image through a camera.

[0319] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0320] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0321] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, etc. In some embodiments, electronic device 100 can implement image segmentation, repair and other functions by running a neural network model through NPU. For example, the image of the modified area is repaired by running a stable diffusion model through NPU.

[0322] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0323] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image playback function), etc. The data storage area may store data (such as image data) created during the use of the electronic device 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. In some implementation examples, the processor 110 executes the image processing method provided in the embodiment of the present application by running instructions stored in the memory 120.

[0324] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 and the application processor.

[0325] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0326] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons.

[0327] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0328] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0329] The SIM card interface 195 is used to connect a SIM card.

[0330] The various embodiments of the mechanisms disclosed in this application may be implemented in hardware, software, firmware, or a combination of these implementation methods.

[0331] The embodiment of the present application also provides a program product, which includes instructions. When the instructions are executed by an electronic device, the electronic device can implement the image processing method provided by the embodiment of the present application.

[0332] An embodiment of the present application also provides a chip, wherein the chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program code; when the computer program code is executed by the processing circuit, the image processing method provided by the embodiment of the present application is implemented.

[0333] Embodiments of the present application may be implemented as a computer program or program code executed on a programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0334] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0335] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0336] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0337] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0338] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0339] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0340] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image processing method, characterized in that: include: Determining a first area and a second area of ​​a first image to be processed; generating a modified region of the first region, wherein the modified region includes at least a portion of a boundary of the first region; Correcting the first region based on the modified region to obtain a first corrected region; The first correction area is segmented from the first image as a first sub-image of the first image.

2. The method according to claim 1, characterized in that The first area is a foreground area of ​​the first image, and the second area is a background area of ​​the first image; or The first area is a background area of ​​the first image, and the second area is a foreground area of ​​the first image.

3. The method according to claim 1 or 2, characterized in that: The determining of the first area and the second area of ​​the first image to be processed includes: A first mask of the first image is determined based on a segmentation model, the first mask including the first region having a first color and the second region having a second color.

4. The method according to claim 3, characterized in that The segmentation model includes at least one of the following: Deep mapping, path aggregation network model, geodesic distance algorithm, random walk algorithm.

5. The method according to claim 3, characterized in that: The step of generating a modified region of the first region comprises: The edge of the first area of ​​the first mask is expanded to obtain the modified area.

6. The method according to claim 5, characterized in that The step of expanding the edge of the first area of ​​the first mask to obtain the modified area includes: Binarizing the pixel values ​​of each pixel point in the first mask to obtain a first binary image; Determine an edge of the first region having a first color in the first binary image; Enlarging the first region according to a first ratio to obtain a first enlarged region, wherein a center of the first enlarged region coincides with a center of the first region; A portion of the first enlarged region that does not include the first region is used as the modified region.

7. The method according to claim 3, characterized in that The generating of the modified area of ​​the first area further comprises: Determine minimum and maximum thresholds; A set of pixel points in the first mask whose pixel values ​​are between the lowest threshold and the highest threshold is used as the modification area.

8. The method according to claim 1 or 2, characterized in that: The correcting the first region based on the modified region to obtain a first corrected region includes: The first region is corrected based on the generative neural network model to obtain the first corrected region.

9. The method according to claim 8, characterized in that The generation of a neural network model includes at least one of the following: Stable diffusion model, generative adversarial network model, partial convolution model.

10. The method according to claim 8, characterized in that Also includes: A second correction region is obtained based on the generative neural network model.

11. The method according to claim 10, characterized in that The generative neural network model obtains the first correction region and the second correction region in the following manner: determining an edge of the modified area as an edge of the first correction area; Filling the first correction area based on the pixel values ​​of the first area to obtain the first correction area; A portion of the first image that does not include the first correction area is used as the second correction area.

12. The method according to claim 11, characterized in that Segmenting the first correction region from the first image as a first sub-image of the first image includes: Setting a pixel value of the second correction area in the first image to a first pixel value to obtain a second image, wherein a difference between the first pixel value and a pixel value of a pixel point in the first correction area is greater than a pixel threshold; The second image is input into a segmentation model to obtain a first sub-image of the first image.

13. The method according to claim 1 or 2, characterized in that: The correcting the first area based on the modified area to obtain a first corrected area and a second corrected area includes: Acquire text control information, where the text control information is related to the image object in the first area; A first correction area and a second correction area are obtained based on the text control information, the image information of the first area, and the modified area, wherein the image information includes at least one of color features, texture features, shape features, and spatial relationship features.

14. The method according to claim 1 or 2, characterized in that: The correcting the first area based on the modified area to obtain a first corrected area and a second corrected area also includes: Acquire image information of the first image and semantic information of the first image, wherein the image information includes at least one of color features, texture features, shape features, and spatial relationship features, and the semantic information includes at least one of visual information, object information, and concept information; A first correction region and a second correction region are determined based on the image information of the first image and the semantic information of the first image.

15. The method according to claim 14, characterized in that The semantic information includes at least one of the following: Depth map, pose map, edge map, sketch map, normal map.

16. The method according to claim 1 or 2, characterized in that: The determining of the first area and the second area of ​​the first image to be processed includes: Determining a semantic segmentation map of the first image, and determining a plurality of first sub-regions from the first image based on the semantic segmentation map; The first sub-region that meets the region condition is taken as the first region, and taking a portion of the first image that does not include the first area as the second area; The area condition includes: the image object in the first sub-area is a target object.

17. The method according to claim 1 or 2, characterized in that: The determining of the first area and the second area of ​​the first image to be processed includes: Determining a semantic segmentation map of the first image, and determining a plurality of second sub-regions from the first image based on the semantic segmentation map; taking a second sub-region having a third color in the plurality of second sub-regions as the first region; or The plurality of second sub-regions are displayed to the user, and a second sub-region selected by the user is used as the first region.

18. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 17.

19. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of an electronic device, and is used to execute the method according to any one of claims 1 to 17.

20. A chip, characterized in that: The chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program code; when the computer program code is executed by the processing circuit, the method according to any one of claims 1 to 17 is implemented.

Images