Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an initial image, and a background blurring image and a target part segmentation image corresponding to the initial image; the initial image comprises a target object to which a target part belongs; according to the initial image, the background blurring image and the target part segmentation image, obtaining image data after target part adjustment; and fusing the adjusted image data of the target part with the background blurred image to obtain a target image. By adopting the method, the target image can have a background blurring effect, and the structure and the color of the target part can be displayed more accurately.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] In traditional technology, when a blurred background image is captured by an electronic device, the blurred background image has a problem of missing details, which may cause image distortion.

[0003] Currently, the definition of the blurred background image can be improved by adjusting the blurred background image. However, after the adjustment is directly based on the blurred background image, there is still a lack of details between the target object and the background area of ​​the target object. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can make a target image have a background blur effect and can more accurately display the information of the target part.

[0005] In a first aspect, the present application provides an image processing method, comprising:

[0006] Acquire an initial image, and a background blur image and a target part segmentation map corresponding to the initial image; the initial image includes a target object to which the target part belongs;

[0007] Obtaining image data of the target part after adjustment according to the initial image, the background blur image and the target part segmentation map;

[0008] The adjusted image data of the target part is fused with the background blurred image to obtain a target image.

[0009] In a second aspect, the present application further provides an image processing device, comprising:

[0010] An acquisition module, used to acquire an initial image, a background blur image corresponding to the initial image, and a target part segmentation map; the initial image includes a target object to which the target part belongs;

[0011] An adjustment module, used for obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map;

[0012] The fusion module is used to fuse the adjusted image data of the target part with the background blurred image to obtain a target image.

[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the image processing method of the first aspect is implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method of the first aspect.

[0015] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the image processing method of the first aspect when executed by a processor.

[0016] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product obtain an initial image, a background blur image and a target part segmentation map corresponding to the initial image; the initial image includes a target object to which the target part belongs, so the initial image provides structural information and color information of the target part by containing the target part and the target object that are not blurred, forming information adjustment under these two dimensions, which helps to accurately provide the real information of the target part; at the same time, the background blur image has the characteristic of background blur, which can provide blurred background information; in addition, since the target part segmentation map focuses on the characteristics of the target part, the target part segmentation map can focus on providing information of the target part. Then, according to the initial image, the background blur image and the target part segmentation map, the image data after the target part is adjusted is obtained, so as to realize the adjustment process of the target part by gathering the information carried by the three images; then, the image data after the target part is adjusted is fused with the background blur image to obtain the target image, which can make the target image have the effect of background blur and can more accurately show the structure and color of the target part. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0019] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0020] Figure 3is a schematic diagram of a process of adjusting a cropped image in one embodiment;

[0021] Figure 4 A schematic diagram of a flow chart of adjusting through combined data in one embodiment;

[0022] Figure 5 A schematic diagram of a process flow through model adjustment in one embodiment;

[0023] Figure 6 is a schematic diagram of a portrait with a clear background in one embodiment;

[0024] Figure 7 is a schematic diagram of a portrait with blurred background in one embodiment;

[0025] Figure 8 A schematic diagram of a framework of a target site adjustment model in one embodiment;

[0026] Fig. 9 A schematic diagram of a process for obtaining a fusion weight in one embodiment;

[0027] Fig.10 A schematic diagram of a model selection interface in one embodiment;

[0028] Fig.11A is a schematic diagram of a model management system in one embodiment;

[0029] Fig. 11B is a schematic diagram of a model management system in one embodiment;

[0030] Fig.12 is a flow chart of an implementation scheme of a testing process for a neural network module in one embodiment;

[0031] Fig.13 A flowchart of a facial detection process implemented by a neural network module on a device in one embodiment;

[0032] Fig.14 is a structural block diagram of an image processing device in one embodiment;

[0033] Fig.15 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] In one embodiment, the image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the electronic device 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the electronic device 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0036] In some exemplary embodiments, Figure 2 As shown, an image processing method is provided, which is applied to Figure 1 The electronic device 102 in the example is used for illustration, and can also be applied to the server 104. The method includes the following steps 202 to 206:

[0037] Step 202, obtaining an initial image, a background blur image corresponding to the initial image, and a target part segmentation map; the initial image includes a target object to which the target part belongs.

[0038] The initial image is image data in which the clarity of the background area is higher than that of the background blur image. The initial image includes the area where the target object is located and the background area other than the area where the target object is located; the area where the target object is located includes the area where the target part is located. The initial image can provide relatively accurate information for the target part and the target object.

[0039] The blurred background image is image data that contains the same target object as the initial image, and the clarity of the background area is lower than that of the initial image. The blurred background image includes the area where the target object is located, and the background area other than the area where the target object is located; the area where the target object is located includes the area where the target part is located. Compared with the background area in the initial image, the background area of ​​the blurred background image is relatively blurred. The background area of ​​the blurred background image can be the background area of ​​the target object, the background area of ​​the target object is the area other than the area where the target object is located, and the area where the target object is located includes the target part area.

[0040] The target part segmentation map is image data containing the target part. The target part segmentation map includes at least the area where the target part is located, and the background area in the target part segmentation map is the area outside the area where the target part is located. The target part segmentation map is image data used to indicate the area where the target part is located. Through the target part segmentation map, the image data of the initial image and the background blurred image at the target part can be determined.

[0041] The target object is an object containing a target part. From the perspective of quantity and range, the target object can be one object, or multiple objects in the initial image, or the object with the largest area among multiple objects. The target object in the background blurred image has the phenomenon of missing details of the target part; the phenomenon of missing details of the target part includes but is not limited to the existence of breakpoints in the lines used to constitute the target part, and the color distortion of the pixels used to represent the target part. For example, the target object is a face, and the target part is the hair area of ​​the face; in order to avoid problems such as broken hair and light hair color in the background blurred portrait, and to improve the filming effect of the portrait mode, a variety of information needs to be processed to make the hair complete and the hair color more realistic. In addition, the target object can be a plant, and the target part can be the roots of the plant; the target object can be an animal, and the target part can be the hair of the animal.

[0042] The initial image can be acquired by a focused camera, or it can be transmitted by other devices. The blurred background image can be obtained by fusing multiple images acquired by cameras of different focal lengths, or it can be acquired from other devices according to the identification of the initial image and certain corresponding relationships, or it can be acquired by blurring the background of the initial image. The target part segmentation map can be determined according to the area where the target part is located in the initial image, so as to ensure that the corresponding target part segmentation map can more accurately reflect the target part through higher clarity; the target part segmentation map can also be determined according to the area where the target part is located in the blurred background image; the target part segmentation map can also be acquired from other devices according to the identification of the initial image and certain corresponding relationships.

[0043] Step 204 , obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map.

[0044] The adjusted image data of the target part at least includes the image data obtained by adjusting the target part. Since the initial image, the background blurred image and the target part segmentation map can provide information of three kinds of image data, the information of the three kinds of image data can be combined to make the target part closer to the real scene.

[0045] The channels of each pixel in the initial image, the background blur image and the target part segmentation map can be used to extract information, and then the image data of the target part can be adjusted based on the information carried by the three images through the information extracted by more channels; the channels in the initial image, the background blur image and the target part can also be directly merged to integrate the information carried by the three images through the merged channels to form combined data, and the combined data can be adjusted to obtain the adjusted image data of the target part to reduce the adjustment process. Exemplarily, the initial image, the background blur image and the target part segmentation map all contain their own channels, and the data contained in their own channels can be combined to obtain the combined data; then the combined data is adjusted through the target part adjustment model to obtain the adjusted image data of the target part.

[0046] Since the initial image, the background blur image and the target part segmentation map are three kinds of image data containing the target part, the information contained in the three kinds of image data can be fully utilized by adjusting the three kinds of image data, so that the target part is closer to the real shooting scene. Specifically, the semantic information provided by the target part segmentation map can be used as a guide to associate the target part in the initial image with the target part in the background blur image, and the target part in the background blur image is adjusted through the target part in the initial image to achieve the goals of accurate image correction, low information loss compression, etc. The initial image and the target part segmentation map can be mutually verified to more accurately determine the verified information of the target part, and then adjust the target part of the background blur image according to the verified information of the target part to obtain the adjusted image data of the target part. For example, the adjusted image data of the target part can be obtained through some neural network models. Thus, based on the information gathered by the three kinds of image data, the adjusted image data of the target part is obtained, so as to comprehensively utilize the information carried by the three kinds of images to achieve the effects of image restoration of the target part, so that the adjusted image data of the target part is closer and closer to the real scene.

[0047] Step 206: Fusing the adjusted image data of the target part with the background blurred image.

[0048] The target image has comprehensive information provided by the adjusted image data of the target part and the background blur image, so the target image has the effect of background blur, and can more accurately provide the structure and color of the target part through the adjusted image data of the target part, so that the target image is closer to the real shooting scene.

[0049] For example, the target part contained in the image data after the target part is adjusted can be added to the target part in the background blur image to obtain the target image. Alternatively, the fusion weight of each point in the image data after the target part is adjusted can be determined first, and the image data after the target part is adjusted can be fused with the target part data in the background blur image according to the fusion weight of each point in the image data after the target part is adjusted to obtain the target image.

[0050] In the above image processing method, an initial image, a background blur image and a target part segmentation map corresponding to the initial image are obtained; the initial image includes the target object to which the target part belongs, so the initial image provides structural information and color information of the target part by containing the target part and the target object that are not blurred, forming information adjustment in these two dimensions, which is helpful to accurately provide the real information of the target part; at the same time, the background blur image has the characteristic of background blur, which can provide blurred background information; in addition, since the target part segmentation map focuses on the characteristics of the target part, the target part segmentation map can focus on providing information of the target part. Then, according to the initial image, the background blur image and the target part segmentation map, the adjusted image data of the target part is obtained to gather the information carried by the three images; then, the adjusted image data of the target part is fused with the background blur image to obtain the target image, which can make the target image have the effect of background blur and can more accurately show the structure and color of the target part.

[0051] In some embodiments, the target part segmentation map is a binary map, the first pixel value in the binary map represents the area where the target part is located, and the second pixel value in the binary map represents the area outside the target part.

[0052] A binary image is image data that represents information through two numerical values. When the target part segmentation map is a binary image, the target part segmentation map can use two numerical values ​​in one channel to represent information respectively, so the required storage space and computing space are relatively small. When the target part segmentation map is used as a binary image, the target part is represented by the area where the target part is located, so that the accuracy of the target part is controlled at the regional level, thereby avoiding the problem of too much data due to too many details of the target part, thereby occupying less resources with relatively less data.

[0053] The first pixel value and the second pixel value are different pixel values. The first pixel value and the second pixel value may both be two pixel values ​​of a grayscale channel, so as to distinguish different regions. When the first pixel value is 1 or 255, the second pixel value is 0, so that the black pixel reflected by 1 or 255 represents the region where the target part is located, and the white pixel reflected by 0 represents the region outside the region where the target part is located.

[0054] When the target part segmentation map is a binary image, the difference between the target part segmentation map and the initial image is that the background area in the target part segmentation map is the area outside the area where the target part is located; the difference between the target part segmentation map and the background blurred image is that the background area in the target part segmentation map is the area outside the area where the target part is located, and the background area in the target part segmentation map does not involve blurring processing.

[0055] Exemplarily, the target part may be hair, and the target part segmentation map may be a hair segmentation map; and the pixels in the binary map include white pixels and black pixels, the white pixels represent the area where the hair is located, and the black pixels represent the area outside the hair. Thus, the pixels at the corresponding position of the hair segmentation map are white, otherwise they are black, so as to control the degree of detail of the edge hair in the hair segmentation map through the data at the regional level, selectively retain the degree of detail of the features, and ignore the features that are too small. The target part may be the hair on the head of a male lion, and the target part segmentation map may be a hair segmentation map; and the pixels in the binary map include white pixels and black pixels, the white pixels represent the area where the hair on the head of a male lion is located, and the black pixels represent the area outside the hair on the head of a male lion. In addition, the target part may also be the rhizome of asparagus fern, and the target part segmentation map may be a rhizome segmentation map.

[0056] In this embodiment, when the target part segmentation map is a binary map, the binary map data volume is relatively small, and the target part segmentation map can efficiently and accurately reflect the structural information of the target part to clarify the correlation between the pixels contained in the initial image and the background blurred image at the target part. When the fusion weight is obtained based on the target part segmentation map, the advantage of the relatively small amount of binary map data can be fully utilized to obtain the fusion weight with less calculation.

[0057] In some embodiments, the target part segmentation map is confidence image data, and the pixel value of the target part segmentation map represents the confidence that each pixel is located in the area where the target part is located.

[0058] The confidence image data includes the probability that each pixel belongs to the target part. Each pixel in the confidence image data has its own confidence, so as to indicate whether each pixel is located in the target part area through the confidence.

[0059] The pixel values ​​of each point in the initial image can be classified and identified to obtain the probability that each pixel belongs to the target part; according to the probability that each pixel belongs to the target part, the confidence of the pixel is determined.

[0060] Exemplarily, when the pixel value of the target pixel in the confidence image data is greater than the preset confidence, the corresponding pixel of the target pixel in the initial image and the virtual background image is located in the area where the target part is located; when the pixel value of the target pixel in the confidence image data is less than the preset confidence, the corresponding pixel of the target pixel in the initial image and the virtual background image is not located in the area where the target part is located.

[0061] In this embodiment, when the target part segmentation map is confidence image data, it can more finely reflect whether each pixel belongs to the target part, and can more accurately control the details of the target part.

[0062] In some embodiments, the target part includes hair and the target object includes a human face.

[0063] A hair is at least one strand of hair; a face is at least one face of a person. When the electronic device is in face mode, the target part includes hair, and the target object includes a face, and corresponding steps 202 to 206 are performed to implement the hair image processing steps.

[0064] In this embodiment, the target part includes hair, and the target object includes a face. The detail retention degree of the hair can be controlled by segmenting the image of the target part to accurately retain the information of the hair and to produce a blurring effect in the area outside the face.

[0065] In some embodiments, an initial image, a background blur image and a target part segmentation map corresponding to the initial image are obtained, including: obtaining the initial image; blurring the background area based on the target object in the initial image to obtain a background blur image; and extracting a target part segmentation image for representing the target part from the initial image.

[0066] You can first acquire the initial image, and then perform blur processing on the background area outside the target object, which can reduce the adjustment process of hardware devices such as the focal length and aperture of the image, and still obtain a background blur image. You can also first acquire the initial image, and then adjust the image acquisition parameters based on the initial image during the acquisition process, and then acquire the background blur image again based on the adjusted image acquisition parameters.

[0067] The target part in the initial image can be identified by using u-net or other target detection models, and then the target area and the area outside the target area can be segmented to obtain a target area segmentation image. The target area and the area outside the target area can also be segmented by matching feature points to obtain a target area segmentation image.

[0068] Exemplarily, the pixels in the region where the target part is located may be set to a first pixel value, and the pixels in the region where the target part is located may be set to a second pixel value, so as to achieve segmentation, thereby obtaining a target part segmentation image.

[0069] In this embodiment, an initial image is first acquired, and then the background area based on the target object in the initial image is blurred, so that there is no blur between the target part and the target object, thereby ensuring that the overall composition of the target object and the target part is more realistic; and because the initial image is not blurred, even if the target part is particularly small, it can be identified. Therefore, a target part segmentation image used to represent the target part is extracted from the initial image, which can make the target part segmentation image more accurate, thereby ensuring that the overall composition of the target object and the target part is more realistic.

[0070] In some embodiments, the background area is blurred based on the target object in the initial image to obtain a background blurred image, including: obtaining an image of the target object in a blurred background area according to the focal length and aperture values; the image of the target object in a blurred background area includes the image after background blur.

[0071] The target object is in the blurred background area, which means that the background area of ​​the target object is blurred. From the perspective of data processing, the target object is in the blurred background area, which is similar to using a blur function to blur the background of the initial image. The blur effect is better when the focal length and aperture are used.

[0072] The focal length is the interval of the zoom ratio that matches the camera. The field of view of the target image can be controlled by the zoom ratio; the smaller the zoom ratio, the wider the image range corresponding to the field of view angle and the deeper the depth of field; the larger the zoom ratio, the smaller the image range corresponding to the field of view angle and the shallower the depth of field. The aperture can control the depth of field of the camera for image acquisition. The larger the aperture, the deeper the depth of field received by the camera, and the smaller the aperture, the shallower the depth of field. Since both the focal length and the aperture can reflect the depth of field, and the depth of field can change the degree of blur, the values ​​of the focal length and the aperture can be used to control the degree of blur of the background area of ​​the target object. Exemplarily, in a certain mode, the longer the focal length used to collect images and the larger the aperture, the stronger the degree of blur of the image after the background blur is obtained, and the blurrier the background; the smaller the focal length used to collect images and the smaller the aperture, the weaker the degree of blur of the image after the background blur is obtained, and the clearer the background, so as to collect image data through the parameters of the camera, thereby dynamically controlling the degree of blur of the background area of ​​the target object.

[0073] The shallower the depth of field, the clearer the object at the first distance from the electronic device, and the blurrier the object at other distances from the electronic device, resulting in a blur effect; correspondingly, the deeper the depth of field, the clearer the object at the second distance from the electronic device, and the blurrier the object at other distances from the electronic device, resulting in a blur effect; the first distance is greater than the second distance. Exemplarily, the focal length and aperture used to obtain the initial image can be adjusted to obtain an adjusted focal length and an adjusted aperture; according to the adjusted focal length and the adjusted aperture, an image with blurred background corresponding to the initial image is obtained.

[0074] In this embodiment, the degree of blurring of the background blurred image is reflected according to the values ​​of the focal length and aperture, so that the degree of background blurring can be dynamically changed with the focal length and aperture, forming a variety of display effects. And by controlling the camera acquisition method through the values ​​of the focal length and aperture to obtain the background blurred image, the background blurred image can more accurately retain more information of the shooting scene.

[0075] In some embodiments, image data of the target part after adjustment is obtained according to the initial image, the background blur image and the target part segmentation map, including: merging channels of the initial image, the background blur image and the target part segmentation map to obtain combined data; wherein the channels of the combined data include channels of the initial image, channels of the background blur image and channels of the target part segmentation map; adjusting the target part in the combined data to obtain image data of the target part after adjustment.

[0076] Channel merging means that, on the basis of keeping the original channels and channel values ​​of the three image data unchanged, the original channels of the three image data are stacked simultaneously, so that the channels contained in the stacked three image data form the channels of the combined data. Exemplarily, the channels of the initial image include channel A, the background blur image includes channel B, and the target part segmentation map includes channel C, then the channels of the combined data include channel A, channel B and channel C.

[0077] In the case where the combined data is obtained by merging channels, the combined data is a whole data, which can be in the form of a matrix or a vector, etc., so the combined data can be directly input into the target part adjustment model for the corresponding channel for adjustment, so as to reduce the model preprocessing steps such as data conversion and reconstruction. For example, the combined data is downsampled to obtain downsampled features; feature fusion processing is performed based on the downsampled features to obtain fused features; upsampled features are upsampled to obtain upsampled fused features; the downsampled features are fused with the upsampled fused features to obtain the image data after the target part is adjusted.

[0078] In this embodiment, three types of image data are integrated into the combined data to avoid the use of multiple image data to merge with the background blur image, thereby ensuring processing efficiency; on this basis, the combined data is obtained by channel merging, so that the combined data can carry the information carried by all channels of the three images, so that the combined data can provide comprehensive information of the target part, so as to more finely adjust the target part in the background blur image, so that the image data of the target part after adjustment is more realistic.

[0079] In some embodiments, the channels of the initial image and the channels of the background blurred image are multiple, and the channel of the target part segmentation map is one.

[0080] When the target part segmentation map has only one channel, the amount of data of the target part segmentation map can be reduced. Regardless of whether this channel is a channel of a binary image or a channel used to represent confidence, it can represent the structural information of the target part and can accurately provide the structural information of the target part with a smaller amount of data to ensure a better fusion effect.

[0081] Exemplarily, since the RGB image data includes multiple pixels, the RGB channels of each pixel include a red channel, a green channel, and a blue channel, and thus, according to the multiple pixels of the RGB image data in length H and width W, a channel of a piece of RGB image data can be represented by a matrix of HxWx3; correspondingly, since the grayscale image data includes multiple pixels, a channel of each pixel includes a brightness channel and a grayscale channel, and thus, according to the multiple pixels of the RGB image data in length H and width W, a channel of a piece of grayscale image data can be represented by a matrix of HxWx1. At this time, the clear image and the blurred image are two types of RGB image data, and the hair segmentation image is a grayscale image; the channel dimensions of the two RGB image data and a grayscale image are combined to obtain a matrix of HxWx7, and the matrix of HxWx7 is the combined image data.

[0082] In this embodiment, multiple channels of the initial image are used to provide more comprehensive structural information and color information of the target object and the target part contained in the target object; multiple channels of the background blur image are used to provide more comprehensive information of the blurred background area. At the same time, since the target part segmentation map has only one channel, the structural information of the target part can be accurately provided with less data, thereby ensuring a better fusion effect.

[0083] In some embodiments, Figure 3 As shown, according to the initial image, the background blur image and the target part segmentation map, the image data after the target part is adjusted is obtained, including steps 302-306; wherein:

[0084] Step 302 : cropping the region where the target part of the target object in the initial image is located according to the size corresponding to the target part segmentation map, to obtain a cropped initial image.

[0085] The size corresponding to the target part segmentation map is the size of the area where the target part is located. Since the target part segmentation map can divide the area where the target part is located from the area outside the target part, the size corresponding to the target part segmentation map is used for cropping, so that the area where the target part is located can be reduced, and the area where the target part is located is reduced. The size corresponding to the target part segmentation map can be a fixed size or an adaptively adjusted size.

[0086] When the size corresponding to the target part segmentation map matches the actual size of the target part segmentation map, the target part segmentation map can be directly cropped according to the actual size to obtain a cropped initial image and a cropped background blurred image. When the size corresponding to the target part segmentation map is smaller than the actual size of the target part segmentation map, the target part region of the target part segmentation map can be cropped according to the size corresponding to the target part segmentation map to obtain a cropped target part segmentation map; and then the cropped initial image, the cropped background blurred image and the cropped target part segmentation map are fused.

[0087] In the case where the initial image, the background blur image, and the target part segmentation map have the same resolution, the cropped initial image, the cropped background blur image, and the target part segmentation map are image data of the same size.

[0088] The cropped initial image includes the target part region in the initial image. When the target part segmentation map has different corresponding sizes, the target part region of the target object is also different. The target part region may include the target part region itself, and may also include a portion of the background region outside the target part region, which is relative to the background region of the target part, and may include a portion of the target object region, and may also include a portion of the background region of the target object.

[0089] Exemplarily, the target part of the target object in the initial image can be cropped according to the size corresponding to the target part segmentation map, to obtain the target part image cropped from the initial image; the target part image cropped from the initial image is a cropped initial image. The target part of the target object in the initial image and the background area of ​​the target size can be cropped according to the size corresponding to the target part segmentation map, to obtain the target part and background image cropped from the initial image; the target part and background image cropped from the initial image is a cropped initial image.

[0090] Step 304 , cropping the area where the target part of the target object in the background blur image is located according to the size corresponding to the target part segmentation map, to obtain a cropped background blur image.

[0091] The cropped background blur image includes the target part area in the background blur image. When the corresponding sizes of the target part segmentation map are different, the target part area of ​​the target object is also different. The target part area may include the target part area itself, and may also include a part of the background area outside the target part area, which is relative to the background area of ​​the target part, and may include a part of the target object area, and may also include a part of the background area of ​​the target object.

[0092] Exemplarily, the target part of the target object in the background blur image can be cropped according to the size corresponding to the target part segmentation map, to obtain the target part image cropped from the background blur image; the target part image cropped from the background blur image is a cropped background blur image. The target part of the target object in the background blur image can be cropped according to the size corresponding to the target part segmentation map, to obtain the target part and background image cropped from the background blur image; the target part and background image cropped from the background blur image is a cropped background blur image.

[0093] Step 306 , adjusting according to the cropped initial image, the cropped background blur image and the target part segmentation map to obtain adjusted image data of the target part.

[0094] In one embodiment, adjustments are made based on a cropped initial image, a cropped background blur image, and a target part segmentation map to obtain image data after the target part is adjusted, including: merging channels of the cropped initial image, the cropped background blur image, and the target part segmentation map to obtain combined data; wherein the channels of the combined data include channels of the initial image, channels of the background blur image, and channels of the target part segmentation map; and adjusting the target part in the combined data to obtain image data after the target part is adjusted.

[0095] In one example, the target part is hair and the target object is a face; at this time, based on the hair segmentation map, the portrait with a clear background, the portrait with a blurred background, and the hair segmentation map are cropped to obtain the hair area and the background area of ​​the hair area of ​​the portrait with a clear background, the portrait with a blurred background, and the hair segmentation map; the target part is adjusted for the hair area and the background area of ​​the hair area to obtain the image data after the target part is adjusted. Therefore, since hair only occupies a part of the image in many portrait scenes, it is necessary to use the information of the hair segmentation map to crop the three images to reduce redundant interference information and save computing power.

[0096] In this embodiment, the target part segmentation map is used for both image cropping and fusion to the image data after the target part is adjusted; when cropping is performed based on the size corresponding to the target part segmentation map, the size is reduced, thereby reducing the area outside the target part through a smaller size, reducing the interference information generated by the area outside the target part to the target part, and saving computing power through the small size after cropping.

[0097] In some embodiments, the method further includes: determining a size corresponding to the target part segmentation map based on an area ratio of the target part in the initial image.

[0098] The area ratio of the target part in the initial image includes the area ratio between the area of ​​the target part and the area of ​​the initial image. The area ratio indicates the percentage of information used to characterize the target part in the information carried by the initial image. In the case that the size of the initial image changes, the target part is determined by the area ratio, so as to adaptively adjust the size corresponding to the target part segmentation map according to various sizes of the initial image or the target part.

[0099] In one example, a size product result can be obtained based on the area ratio of the target part in the initial image and the size of the initial image without cropping; and the size corresponding to the target part segmentation map is determined according to the size product result. For example, the size product result can be used as the size corresponding to the target part segmentation map, or the size product result can be adjusted based on the background area in the initial image to obtain an adjusted size product result; and the adjusted size product result is used as the size corresponding to the target part segmentation map.

[0100] Exemplarily, when the area ratio of the target part in the initial image is a first percentage value, the size corresponding to the segmentation map of the target part can be determined to be the size corresponding to the first percentage value; and when the area ratio of the target part in the initial image is a second percentage value, the size corresponding to the segmentation map of the target part can be determined to be the size corresponding to the second percentage value; wherein the first percentage value and the second percentage value are different percentage values, and the sizes corresponding to the first percentage value and the second percentage value are different. When the area ratio of the target part in the initial image is in a first percentage interval, the size corresponding to the segmentation map of the target part can be determined to be the size corresponding to the first percentage interval; and when the area ratio of the target part in the initial image is in a second percentage interval, the size corresponding to the segmentation map of the target part can be determined to be the size corresponding to the second percentage interval; wherein the first percentage interval and the second percentage interval are different percentage values, and the sizes corresponding to the first percentage interval and the second percentage interval are different.

[0101] In this embodiment, based on the area ratio of the hair in the initial image, the size corresponding to the target part segmentation map is adaptively adjusted so that the information carried by the cropped initial image and the cropped background blur image is adaptively adjusted with the size, which can not only ensure that the amount of information used to characterize the target part is sufficient, but also avoid the problem of excessive interference information, so as to further ensure the accuracy of the information.

[0102] In some embodiments, the size corresponding to the target part segmentation map is determined based on the area ratio of the target part in the initial image, including: determining the size corresponding to the target part segmentation map based on the area ratio of the target part in the initial image and the background complexity of the initial image; wherein the background complexity is negatively correlated with the size corresponding to the target part segmentation map.

[0103] Background complexity indicates the degree of similarity between the background area of ​​the target part and the target part. The higher the background complexity, the greater the degree of similarity between the background area and the target part, and the processing of the target part in the background area requires more computing resources; the lower the background complexity, the less similarity between the background area and the target part, and the processing of the target part in the background area requires less computing resources.

[0104] Exemplarily, a background area with high background complexity may have a lot of complex textures; taking the target part being the hair of the target person as an example, an area with high background complexity is difficult to distinguish whether it is hair or background. An area with high background complexity may be weeds or the hair of an object other than the target object; an area with low background complexity is an area with a relatively pure background and no texture, such as the sky, a solid-color wall, etc.

[0105] The more complex the background is, the greater the probability of the algorithm making mistakes. Therefore, the complexity is negatively correlated with the size of the cropped area to avoid introducing abnormal errors due to the background area being too complex. For example, if the target part is the hair of the target person, in the case of extremely high background complexity, there is no hair in the background area where the background complexity is higher than the preset value.

[0106] In one example, a size product result can be obtained by multiplying the area ratio of the target part in the initial image and the size of the initial image without cropping; the size product result is adjusted based on an adjustment parameter negatively correlated with the background complexity to obtain an adjusted size product result; and the adjusted size product result is used as the size corresponding to the target part segmentation map. The adjustment parameter negatively correlated with the background complexity can be a parameter inversely proportional to the background complexity.

[0107] In this embodiment, based on the background complexity, the information carried by the cropped initial image and the cropped background blurred image is adaptively adjusted, and the background complexity is negatively correlated with the size corresponding to the target part segmentation map, so as to avoid excessive computational effort due to the background area of ​​the target part being too complex, and abnormal information in the image processing process due to excessive complexity.

[0108] In some embodiments, Figure 4 As shown, according to the initial image, the background blur image and the target part segmentation map, the image data after the target part is adjusted is obtained, including steps 402 to 410, wherein:

[0109] Step 402, combining the initial image, the background blur image and the target part segmentation map to obtain combined data.

[0110] The combined data is obtained by combining the data of the three images. The combined data can simultaneously represent the information carried by the initial image, the background blur image and the target part segmentation map, and serve as the overall object for adjustment.

[0111] The combined data can be obtained by merging channels. The data of the initial image, the background blur image and the target part segmentation map can also be adjusted by sampling and feature splicing to combine the data of multiple channels to form a new feature dimension, which can be fused with the background blur image through the new feature dimension.

[0112] Step 404, down-sampling the combined data to obtain down-sampling features.

[0113] The downsampled features are features at a downsampled scale, and the downsampled scale is a scale that is smaller than the scale of the combined data. The downsampled features include features contained in each channel in the combined data. When the combined data is downsampled, the downsampled features can better reflect the global information of the combined data to form a new information dimension. Exemplarily, the combined data can be downsampled based on the sampling rate, or pooling, convolution downsampling, etc. can be used as the downsampling layer, and the combined data can be downsampled at least once through the downsampling layer.

[0114] When the channels of the combined data include channels of the initial image, channels of the background blur image and channels of the target part segmentation map, the downsampled features can reflect the respective features of the initial image, the background blur image and the target part segmentation map, thereby forming three image data features contained in the downsampled features.

[0115] The down-sampled features include the features of the combined data at the down-sampled scale to characterize the information of the initial image, the background blur image and the target part segmentation map at the down-sampled scale, that is, three down-sampled results of the information carried by the three image data.

[0116] Step 406: Perform feature fusion processing based on the downsampled features to obtain fused features.

[0117] The fused features include the fusion results of downsampled features at at least one scale. When the feature fusion processing is performed according to the downsampled features, the fused features can integrate the data of the downsampled features in each channel to form new dimensional information. The information of the initial image, the background blur image and the target part segmentation map at the downsampled scale, and the information of the fused features at the upsampled scale can be fused into a unified feature, so as to form a new information dimension through the information of the fused features.

[0118] Exemplarily, channel fusion may be performed on the minimum scale down-sampled features to obtain fused features; channel data fusion may be performed on the minimum scale down-sampled features to obtain fused features.

[0119] Step 408, upsampling the fused features to obtain upsampled fused features.

[0120] The upsampled fused features are features at the upsampled scale, and the upsampled scale is a scale larger than the fused features. The upsampled features include the features contained in each channel in the fused features. When the fused features are upsampled, the upsampled features can better reflect the detailed information of the fused features to form a new information dimension. Exemplarily, the fused features can be upsampled based on the sampling rate, or interpolation, transposed convolution upsampling, etc. can be used as the upsampling layer, and the fused features can be upsampled at least once through the upsampling layer.

[0121] Step 410, fusing the down-sampled features with the up-sampled fused features to obtain image data of the target part after adjustment.

[0122] The adjusted image data of the target part at least includes the image data obtained by adjusting the target part. Since the down-sampled features and the up-sampled fused features can provide information of different dimensions, the target part can be made closer to the real scene by combining the information of the two dimensions.

[0123] In one example, both the up-sampled features and the down-sampled features are features of multiple scales, and the features of each scale are fused by different means to make the adjusted image data of the target part more accurate; the down-sampled features include the down-sampled features obtained by down-sampling the down-sampling layers at multiple scales, and the up-sampled fused features include the up-sampled fused features obtained by up-sampling the up-sampling layers at multiple scales; correspondingly, the down-sampled features and the up-sampled fused features are fused to obtain the adjusted image data of the target part, including: splicing the down-sampled features at the i-th scale and the up-sampled fused features at the i-th scale to obtain the fused features of the (i+n)-th scale; i and n are positive integers; when the (i+n)-th scale is the time to stop adjusting the scale, the adjusted image data of the target part is determined according to the fused features of the (i+n)-th scale. The downsampled features and upsampled features at the same scale are concatenated to form fused features at the corresponding scale, and the fused features of the next set of scales are obtained through n. Finally, when the (i+n) scale is the stop scale adjustment, the above processing steps are stopped, and the adjusted image data of the target part is determined based on the fused features of the stop scale adjustment.

[0124] When steps 402 to 410 are executed based on the target part repair model; at this time, the steps for obtaining the down-sampled features, the fused features, the up-sampled fused features and the image data after adjustment of the target part are implemented based on the target part adjustment model; for this situation, the training goal of the target part adjustment model is to output a background blurred image containing the target part.

[0125] Exemplarily, step 404 can be performed based on the downsampling path of the U-net model to obtain upsampling features; based on the original downsampling path, fused features are generated based on step 406; the fused features are then substituted into the upsampling path of the U-net model to execute step 408 to obtain upsampling features; step 410 is then executed through direct splicing (skip connections) of the U-net model to obtain image data after adjustment of the target part; finally, the image data after adjustment of the target part is output to another output layer, and the target image is obtained with the output result of the other output layer.

[0126] In this embodiment, the combined data is first obtained, and then the combined data is downsampled to obtain the downsampled features; the combined data is transformed from the original scale to the downsampled scale to form three downsampled results of the information carried by the three image data; at the same time, feature fusion processing is performed according to the downsampled features to obtain fused features, thereby forming a new information dimension. Then, the fused features are upsampled to obtain upsampled fused features, so that the new information dimension has features under the upsampled scale, and the feature fusion of the three image data is realized; then, the downsampled features and the upsampled fused features are fused to obtain the image data of the target part after adjustment, and the image data of the target part after adjustment is formed; then, the image data of the target part after adjustment is fused with the background blurred image, and the obtained target image can more accurately reflect the real scene.

[0127] In some embodiments, feature fusion processing is performed based on the downsampled features to obtain fused features, including: fully connecting the downsampled features at the target scale to obtain fused features; wherein the number of channels of the fused features is less than the number of channels of the downsampled features.

[0128] The target scale is at least one scale in the downsampling process. When the downsampled features at the target scale are obtained, the number of channels of the downsampled features can be directly reduced by fully connecting them, so that the information of the initial image, the background blur image and the target part segmentation map at the downsampled scale can be fused.

[0129] Exemplarily, the channel values ​​contained in the downsampled features at the target scale can be weighted by using a fully connected matrix to obtain fused features.

[0130] In this embodiment, the downsampled features at the target scale are fully connected, so that the features of specific channels can be fused to integrate the data of the respective channels in the downsampled features. When full connection is used, the target part segmentation map is used as the guided semantic information to fuse the target part in the initial image and the background area of ​​the target part in the background blurred image, so that the downsampled features can reduce noise, detect the initial image with a clear background and the background blurred image, and the association between the target part segmentation map, and obtain the fused features more accurately.

[0131] In some embodiments, the target image is obtained by fusing the adjusted image data of the target part with the blurred background image, including: performing blur processing based on the target part segmentation map to obtain the fusion weight of each point contained in the target part; and fusing the adjusted image data of the target part with the blurred background image according to the fusion weight to obtain the target image.

[0132] The fusion weight includes the weight of the image data of each point contained in the target part. The fusion weight can be used to fuse the processing result of the adjusted image data of the target part with the background blurred image. Blurring methods such as blur function and Gaussian blur can be used to map the pixel values ​​of each point contained in the target part segmentation map to obtain the pixel value mapping result of each point contained in the target part segmentation map; the pixel value mapping result of each point contained in the target part segmentation map is used as the fusion weight of each point contained in the target part.

[0133] For example, a Gaussian function can be used to determine and calculate the Gaussian weight value corresponding to each position in the Gaussian kernel, and the image data of each point contained in the target part is Gaussian blurred by the Gaussian weight value to obtain the fusion weight of each point contained in the target part. Thus, the slight color difference and brightness difference caused by the fusion weight map can be reduced by the gradual weight distribution of high-speed blur.

[0134] In one embodiment, the image data of the target part after adjustment is fused with the background blurred image according to the fusion weight, including: adjusting the target part in the image data after adjustment of the target part through the fusion weight of each point to obtain the adjusted target part; and then pasting the adjusted target part back to the target object in the background blurred image to obtain the target image.

[0135] In this embodiment, based on the fuzzy result of the target part segmentation map, the fusion weight of each point contained in the target part is determined to form the fusion weight for the target part, so as to avoid the introduction of the color difference or brightness difference of the target part into the target image during the target part adjustment process, and reduce the target part color information lost in the fusion process, so as to make the target part in the target image more realistic and ensure color adaptability. Since the target part segmentation map can be single-channel image data, the amount of data contained in the target part segmentation map is relatively small, so the use of the target part segmentation map can more efficiently determine the fusion weight.

[0136] In some embodiments, fuzzy processing is performed based on the target part segmentation map to obtain the fusion weight of each point contained in the target part, including: performing external expansion processing on the target part segmentation map to obtain the expanded target part segmentation map; and performing fuzzy processing on the expanded target part segmentation map to obtain the fusion weight.

[0137] The external expansion process is used to expand the outer edge of the target part segmentation map. The target part segmentation map after the external expansion can be obtained by using the dilation operation, or the external expansion process can be realized by using the upsampling or interpolation method.

[0138] When the target part segmentation map uses the dilation operation, the pixel value represents the area where the target part is located, the pixel value represents the background area of ​​the target part, and the pixel value represents the background area of ​​the target part. The image is expanded outward. For example, the target part segmentation map is a binary image, the area with a pixel value of the first pixel value is enlarged, the area with a pixel value of the second pixel value is unchanged or reduced, and the image is expanded outward.

[0139] In this embodiment, the edge area outside the target part segmentation map is enlarged through external expansion processing, so that the area involved in the fusion weight is wider, and the target part not involved in the target part segmentation map can be determined, so as to avoid the situation where the target part segmentation map cannot completely cover the target part, thereby ensuring that more areas can be fused based on the fusion weight, so as to further avoid the introduction of color difference or brightness difference of the target part into the target image during the processing of the combined data.

[0140] In an exemplary embodiment, the target object includes a human portrait as a target, the target part includes the hair of the human portrait, the initial image includes a human portrait with a clear background, the background blur image includes a human portrait with a blurred background, and the target part segmentation map includes a hair segmentation map. At this time, in the portrait mode, the real and natural hair effect can reduce the sense of discontinuity between the person and the blurred background, thereby improving the quality of the film.

[0141] In one embodiment, the hair effect of the portrait mode usually consists of two steps: hair cutout and pasting. The hair cutout obtains the hair weight map from the clear background portrait, and the pasting is to merge the hair area of ​​the clear background portrait to the corresponding position of the blurred background portrait through the hair weight map. However, this technical solution based on hair cutout and then pasting will lose the hair structure information in the cutout process and the hair color information in the pasting process, which will affect the realism of the final film.

[0142] In one embodiment, Figure 5 As shown in FIG. 1 , the background clear portrait is used to obtain the background blur portrait and hair segmentation map; then the background clear portrait, background blur portrait and hair segmentation map are obtained. Figure 1 The hair repair model is input to obtain the adjusted image data of the target part; finally, the data output by the hair repair model is pasted back and fused into the background blurred portrait to obtain the background blurred portrait after hair repair.

[0143] like Figure 6 As shown, in the portrait with a clear background, the background of the portrait is clear, and the hair details 601 at the edge of the portrait are relatively complete; correspondingly, Figure 7 As shown, in the blurred background portrait, the background of the portrait is blurred and the hair details 701 at the edge of the portrait are missing.

[0144] In the case where the target part adjustment model includes a hair restoration model, the combined image data may be input to the hair restoration model for processing.

[0145] In this case, step 202 specifically includes: obtaining a portrait with a clear background, based on the blurred background portrait and the hair segmentation map corresponding to the portrait with a clear background; based on the hair segmentation map, determining whether the portrait with a clear background contains hair, and skipping subsequent processing steps if there is no hair in the picture.

[0146] The steps before step 204 include: based on the hair segmentation map, cropping the portrait with a clear background, the portrait with a blurred background, and the hair segmentation map, and only processing the hair and its vicinity; and,

[0147] The cropped background clear portrait, background blur portrait and hair segmentation map are merged in the channel dimension to obtain the combined data.

[0148] Step 204 specifically includes: sending the combined data to a hair restoration model, and the model outputting a background blurred portrait after hair restoration;

[0149] Step 206 specifically includes: pasting the blurred background portrait after hair restoration back to the cropped area of ​​the original blurred background portrait to obtain the final blurred background portrait containing real and natural hair information, that is, obtaining the target image.

[0150] In an exemplary embodiment, Figure 8 As shown, the target part adjustment model includes downsampling layer 1, downsampling layer 2, downsampling layer 3, downsampling layer 4, downsampling layer 5, and downsampling layer 6 forming a downsampling path, that is, a feature extraction module; upsampling layer 1, upsampling layer 2, upsampling layer 3, upsampling layer 4, upsampling layer 5, and upsampling layer 6 forming an upsampling path, that is, a feature fusion module; and a fully connected layer between the minimum-sized downsampling layer 6 and the minimum-sized upsampling layer 1.

[0151] Correspondingly, the combined data is input to the downsampling layer 1, and the spatial resolution of the feature map is gradually reduced by the downsampling path until it reaches the downsampling layer 6 of the smallest size. The output result of the downsampling layer 6 of the smallest size is flattened through the fully connected layer to obtain the fused feature obtained by the fusion of the downsampling features; the fused feature is input to the upsampling path, and the spatial resolution of the feature map is gradually increased by the upsampling path. At the same time, except for the downsampling layer 6 of the smallest size and the upsampling layer 1, the downsampling layers and upsampling layers of the same level are directly spliced ​​until the upsampling layer 6 of the largest size is fused, and the output result of the upsampling layer 6 is passed to the output layer, and the target image is output by the output layer.

[0152] Taking the case where the target part adjustment model is a hair repair model as an example, the feature extraction module in the hair repair model includes six downsampling layers, each downsampling layer has convolutional map features, and performs feature extraction on the input model's clear background portrait, blurred background portrait, and hair segmentation map to obtain downsampling features; after the feature extraction is completed, the features of the clear background portrait, blurred background portrait, and hair segmentation map are fused using a fully connected layer to obtain a fused feature obtained by fusion of the downsampling features.

[0153] Correspondingly, the feature fusion module in the hair repair model contains six upsampling layers, which are used to upsample the fused features. After each upsampling step is completed, it will be fused with the feature layer corresponding to the feature extraction module and then upsampled. The output layer of the network model will perform convolution and activation operations on the fused features obtained by the upsampling module to obtain a blurred background portrait containing hair information.

[0154] At this time, the training data of the hair repair model includes reference data (ground truth, GT), and the reference data includes a blurred background portrait with hair information, which is an RGB image and the output of the model; since the reference data of the training data is a blurred background portrait (RGB) containing hair information, after the training is completed, the blurred background portrait output by the hair repair model contains both the structural information and color information of the hair.

[0155] Therefore, in the process of cutting out, the portrait with a clear background, the blurred background portrait corresponding to the portrait with a clear background and the hair segmentation map are used to avoid loss of hair structure information.

[0156] In one embodiment, in order to avoid slight color difference and brightness difference that may exist between the model input and output, a specific fusion weight is used to perform transition processing on the repainted area.

[0157] like Fig. 9 As shown in FIG, the method includes: binarizing the hair segmentation image and expanding it; Gaussian blurring the expanded hair segmentation image and normalizing it to obtain a fusion weight map; using the value of the fusion weight map as the weight of the background blurred portrait after hair restoration output by the model, and fusing it to the corresponding position of the initial background blurred portrait. Figure 2 After valorization and expansion, there are only two values, 0 and 1. After Gaussian blur, a weight with numerical transition can be obtained; pasting the weight map back makes the blurred background portrait with hair information pasted to the initial blurred background portrait (without hair) to prevent color difference / brightness difference caused by the model.

[0158] Based on this, in this embodiment, based on the clear background portrait, the blurred background portrait and the hair segmentation results, the hair enhancement model is used to directly restore the lost hair in the blurred background portrait. Compared with the solution of cutting out and then pasting, the background information of the blurred background portrait and the hair structure and color information of the clear background portrait are considered, the hair perception of the blurred background portrait is optimized, and the problems of broken hair and light hair color in the blurred background portrait are reduced, thereby improving the filming effect of the portrait mode. From the user's perspective, after taking a photo using the portrait mode, the user will first pay attention to the face and the surrounding area in the photo. The continuous and natural hair information in the blurred background portrait can enhance the realism of the image and provide users with a better telephoto and large aperture portrait shooting experience.

[0159] Based on this, this embodiment uses the clear background portrait, the blurred background portrait and the hair segmentation map with a hair masking function to directly restore the hair information in the blurred background portrait. Compared with the solution of cutting out the hair and then pasting it back, this embodiment will introduce the structural information and color information of the blurred background portrait. Compared with cutting out the clear background portrait and then pasting it back to the blurred background portrait, it can more accurately restore the structure and color of the hair in the blurred background portrait, thereby greatly improving the filming effect of the portrait mode.

[0160] In addition, the current function of the hair repair model is to extract the hair structure and color information from the portrait with clear background and integrate it into the portrait with blurred background. The model input includes the portrait with clear background, the portrait with blurred background and the hair segmentation map. The hair segmentation map here refers to a single-channel image containing some hair position information, which provides the model with the general structural information of the hair, including but not limited to the following images: the binary image of hair segmentation, that is, the hair segmentation map contains only two values, one of which represents the non-hair area and the other represents the hair area. The confidence map of the hair cutout, that is, the value of each pixel in the hair segmentation map corresponds to the weight of the hair at the pixel position.

[0161] The image input into the hair repair model can be cropped without the need to input the entire image. In the actual cropping process, the size of the cropping area can be controlled based on the area ratio of the hair in the image and the complexity of the background near the clear portrait, as long as the cropping area can completely cover the hair area in the hair segmentation image.

[0162] like Fig.10 As shown, in some embodiments, the model selection interface 1000 may be presented within a web browser. For example, a user may enter a website and present the model selection interface 1000 to the user within a web browser of the website. As depicted, the model selection interface 1000 may provide a variety of menus 1002-1010; options within these menus and sub-options or categories 1010-1014, which the user may select in order to implement the AI ​​functionality required for their application. It should be noted that the user interface shown here is for exemplary purposes and is not limited in any way to the menus, options, and sub-options shown herein. A variety of other menus, options, and sub-options are possible and within the scope of the present application.

[0163] like Fig.10As shown, the model selection interface 1000 may include a task menu 1002, a device type menu 1004, and a constraint menu 1006-1010. The task menu 1002 may include a variety of AI tasks, and the user may select the desired task from these AI tasks to be incorporated into the user's application. By way of example and not limitation, these tasks may include scene recognition, image tagging, object detection, object tracking, object segmentation, human posture estimation, image enhancement, behavior recognition, human emotion recognition, speech recognition, text recognition, natural language understanding and / or any kind of data processing tasks. In a possible embodiment, one or more categories 1010 may be included in one or more of these tasks. The one or more categories in the task may include specific objects or items of interest that the user may wish to pay special attention to. For example, the categories in the object detection task may include people, trees, cars, bicycles, dogs, buildings, etc. For another example, the categories in the scene recognition task may include beaches, snow-capped mountains, flowers, buildings, autumn leaves, waterfalls, night scenes, etc. For another example, the categories in the image enhancement task may include denoising, super-resolution, etc.

[0164] The device type menu 1004 may include different types of user devices on which a given task (e.g., the task selected from the task menu 1002) may be deployed. By way of example and not limitation, user devices may include smartphones, tablets, smart watches, ARM-based platforms, embedded sensors, cameras, intel-based platforms, drones, snapdragon-based platforms, NVIDIA-based platforms (e.g., various GPUs), X-86 series, ambarella platforms, etc. From the list of user devices, the user may select the device on which they wish to deploy the selected task.

[0165] Constraint menus 1006-1010 may include a memory constraint menu 1006, a delay constraint menu 1008, and a power constraint menu 1010. The memory constraint menu 1006 may enable a user to specify the amount of memory to be used for the task selected by the user. For example, as shown in the figure, the user may choose to allocate 100MB of memory to the object detection task. The delay constraint menu 1008 may enable a user to specify the refresh rate or frames per second (FPS) at which they wish their task to run. For example, as shown in the figure, the user may choose to run the object detection task at 10FPS. The power constraint menu 1010 may enable a user to specify the amount of power to be utilized by the task selected by the user. For example, as shown in the figure, the user may choose to allocate 1JPS (joule / second) power to the object detection task.

[0166] Fig.11A 1 is a schematic diagram of various components and their associated functions of a model management system 1100 for assigning an appropriate AI model based on user specifications in some exemplary embodiments. Fig.10 The interface discussed in defines its specifications (e.g., tasks, device types, memory constraints, power constraints, latency constraints), and the user specifications can be sent to the query generator server 1102. The query generator server 1102 receives the user specifications and generates a query 1104, which can be used to retrieve one or more original models 1108 (or device-level models) from the database 1106. The device-level model can be defined in certain programming languages. For example, the device-level model can be defined in Python. The database 1106 may include multiple AI models that can be defined and stored based on previous user specifications. The previous user specification may be related to the current user specification or not. For example, the previous specification may be related to an image labeling task for an application intended to operate on an Android phone, and the model management system 1100 generates a model according to the specification and stores the model in the database 1106 for future access and / or retrieval. It is advantageous to store previous models or have a database of models because this avoids creating models from scratch, and existing models can be reused and adjusted to match current user specifications.

[0167] The original or high-level model 1108 retrieved from the database 1106 may be processed by the model compiler 1110 to make it appropriate according to the current user specifications. For example, the model compiler 1110 may compile the model to remove any unnecessary modules / components and add any required components to match the user specifications. For example, the original model may be pre-generated based on user specifications such as task: object detection; device type: Android; memory constraint: 1GB; latency constraint: 100FPS; power constraint: 1JPS; and some additional constraints. The model compiler 1110 may compile the original model to remove the additional constraints, and may change the device type to a mobile phone and the memory constraint to 100MB to make the model suitable for the current user specifications. In a possible embodiment, the output of the model compiler 1110 is a compact model package or model binary 1112. In a possible embodiment, compiling the original model essentially converts the high-level or original model into a model binary, which may also include user queries, such as Fig.11A shown.

[0168] The interface binder 1114 may bind the compact model / model binary to one or more interface functions. The interface function may be a function that actual users will use in an application or software once the model is deployed into the application. For example, to perform an image tagging task, the interface function may process an input image, and the output of the function will be a bunch of labels within the image.

[0169] Once one or more interface functions are bound to the model binary, the final software binary (a combination of the model binary and the interface functions) can be sent to a distributor 1118, which creates a link 1120 for the user to download the final software binary 1116 on their device. Upon download, the user can incorporate the received binary into their application, which can then be ultimately released for use by the end user. The end user can use the developer's software to perform the AI ​​functions.

[0170] In a possible implementation, an automatic benchmarking component built into the model can automatically evaluate the performance of the model based on, for example, how well the provided model serves the needs of the user and various performance constraints associated with the model (e.g., amount of memory and power usage, speed, etc.). The automatic benchmarking component can report the data obtained from the benchmarking to the model management system 1100, which can store the data in the database 1106 of AI models and use it to train the model. In some implementations, model training can also be performed based on feedback provided by the user at query time.

[0171] Fig. 11B Schematic diagram of additional components and associated functions of the model management system 1100 for training an AI model pool in some exemplary embodiments. The training of a model may include improving the performance of an existing model, adding one or more new features to an existing model, and / or creating a new model. A user may provide training data at query time, or in other words, when a user defines their specifications using the model selection interface 1000. By way of example and not limitation, the training data may include a request from a user to perform object detection for an object that may not currently be included in the model selection interface 1000, but the user may have data corresponding to those objects. The user may then provide their data along with their desired specifications to the model management system 1100, which may use the data to refine its existing models related to object detection to include new object categories provided by the user. For another example, a user may provide performance feedback for the model at query time (e.g., the extent to which the previous model serves its needs in terms of speed, memory, and power usage), and the model management system 1100 may use the performance feedback to train the AI ​​model pool on the server (indicated by reference numeral 1122). Other possible ways of training the model are possible and within the scope of this disclosure.

[0172] The trained AI models 1124 may be sent to a ranker 1126 for ranking. The ranker 1126 may rank the models based on various factors. These factors may include, for example, performance metrics associated with each of the models, benchmark data, user feedback on the models, and the like. Based on the ranking, the highest ranking models 1128 may be determined. For example, the top five ranked models may be selected and sent to the model converter 1130 of the device. The model converter 1130 may convert the highest ranked models into device-level models 1132, as discussed above. Automatic benchmarking may be performed on these models in terms of speed, memory, and power constraints (indicated by reference numeral 1134), and finally these device-level models may be stored in a database of AI models 1106 along with the benchmark data for future access and / or retrieval. In a possible embodiment, Fig. 11B Steps 1122-1134 shown may be used with Fig.11A The steps shown are performed asynchronously.

[0173] like Fig.12 As shown, during the test process, a sample image input may be provided to the neural network module along with the operating parameters. The neural network module may provide sample output data by processing the sample input image using the operating parameters. The sample output data may be compared with the known data of the sample image to see if the data matches in the matching data.

[0174] If the sample output data matches the sample image known data, then the operating parameters are set. If the sample output data does not match the sample image known data (within a desired tolerance), then the training process can be fine-tuned. Fine-tuning the training process can include providing additional training images to the training process and / or other adjustments in the training process to optimize the operating parameters of the neural network module (or generate new operating parameters).

[0175] Once the operating parameters for the neural network module are set in setting the operating parameters, the operating parameters may be applied to the device by providing the operating parameters to the neural network module in the electronic device or server. In certain embodiments, the operating parameters of the neural network module used for training are in a different numerical representation mode than the operating parameters of the neural network module applied on the device. For example, the neural network module used for training may use floating point numbers, while the neural network module used on the device uses integers. Therefore, in such embodiments, the operating parameters of the neural network module used for training are converted from floating point operating parameters to integer operating parameters of the neural network module applied on the device.

[0176] After providing the operating parameters to the neural network module on the device, the neural network module can operate on the device to implement the face detection process on the device. Fig.13As shown, the image input may include an image captured using a camera on the device. The captured image may be a flood infrared illuminated image or a depth map. A face detection process may be used to detect whether a face is present in the image (e.g., placing a bounding box around the face), and if a face is detected, then evaluating values ​​of attributes of the face (e.g., position, pose, and / or distance).

[0177] The captured image from the image input can be provided to the encoder process. The encoder process can be performed by an encoder. In certain embodiments, the encoder module is a multi-scale convolutional neural network. In the encoder process, the encoder module can encode the image input to represent the features in the image as feature vectors in a feature space. The encoder process can output a feature vector. The feature vector can be, for example, an encoded image feature represented as a vector.

[0178] The feature vector may be provided to a decoder process. The decoder process may be performed by an encoder module. In certain embodiments, the decoder module may be a repetitive neural network. In the decoder process, the decoder module may decode the feature vector to evaluate one or more attributes of the image input, thereby determining (e.g., extracting) output data from the image input. Decoding the feature vector may include classifying the feature vector using classification parameters determined during the training process. Classifying the feature vector may include operating the feature vector using one or more classifiers or networks that support classification.

[0179] In some embodiments, the decoder process includes decoding the feature vectors for each region in the feature space. The feature vectors from each region of the feature space can be decoded into non-overlapping boxes in the output data. In some embodiments, decoding the feature vectors of a region (e.g., extracting information from the feature vectors) includes determining (e.g., detecting) whether there is a face in the region. Since the decoder process operates on each region in the feature space, the decoder module can provide a face detection score for each region in the feature space (e.g., based on a prediction of a confidence score as to whether a face or a part of a face is detected / existed in the region). In some embodiments, using an RNN, multiple predictions about whether there is a face (or a part of a face) can be provided for each region of the feature space, wherein the prediction includes predictions about both the face part within the region and the face part around the region (e.g., in an adjacent region). These predictions can be folded into a final decision on the presence of a face in the image input (e.g., detection of a face in the image input). In some embodiments, the prediction is used to form (e.g., place) a bounding box around a face detected in the image input. The output data may include a decision on the presence of a face in the image input (e.g., in a captured image) and a bounding box formed around the face.

[0180] In some embodiments, a face detection process detects the presence of a face in an image input regardless of the orientation of the face in the image input. For example, a neural network module implemented in an electronic device may operate the face detection process using operating parameters implemented by a training process that is developed to detect faces in any orientation in an image input, as described above. Thus, the face detection process can detect faces in an image input having any orientation in the image input without rotating the image and / or receiving any other sensor input data (e.g., accelerometer or gyroscope data) that can provide information about the orientation of the image. Detecting faces in any orientation also increases the range of pose estimates in a bounding box. For example, roll estimates in a bounding box are from -180° to +180° (all roll orientations).

[0181] In some embodiments, a face detection process detects the presence of partial faces in an image input. Partial faces detected in an image input may include any portion of a face present in the image input. The amount of faces that need to be present in the image input to be detected and / or the ability of the neural network module to detect partial faces in the image input may depend on the operating parameters used to generate the neural network module and / or the training of facial features detectable in the image input.

[0182] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0183] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0184] In an exemplary embodiment, Fig.14 As shown, an image processing device is provided, comprising:

[0185] The acquisition module 1402 is used to acquire an initial image, a background blur image and a target part segmentation map corresponding to the initial image; the initial image includes a target object to which the target part belongs;

[0186] An adjustment module 1404, configured to obtain image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map;

[0187] The fusion module 1406 is used to fuse the adjusted image data of the target part with the background blurred image to obtain a target image.

[0188] In one embodiment, the fusion module 1406 is used to:

[0189] Performing fuzzy processing based on the target part segmentation map to obtain the fusion weight of each point contained in the target part;

[0190] According to the fusion weight, the adjusted image data of the target part is fused with the background blurred image to obtain a target image.

[0191] In one embodiment, the fusion module 1406 is used to:

[0192] Performing external expansion processing on the target part segmentation map to obtain an externally expanded target part segmentation map;

[0193] The target part segmentation map after the expansion is fuzzy processed to obtain a fusion weight.

[0194] In one embodiment, the adjustment module 1404 is used to:

[0195] Merging channels of the initial image, the background blur image and the target part segmentation map to obtain combined data; wherein the channels of the combined data include the channels of the initial image, the channels of the background blur image and the channels of the target part segmentation map;

[0196] The target part in the combined data is adjusted to obtain image data of the adjusted target part.

[0197] In one of the embodiments, the channels of the initial image and the channels of the background blurred image are both multiple, and the channel of the target part segmentation map is only one.

[0198] In one embodiment, the adjustment module 1404 is used to:

[0199] According to the size corresponding to the target part segmentation map, cropping the area where the target part of the target object in the initial image is located to obtain a cropped initial image;

[0200] According to the size corresponding to the target part segmentation map, the region where the target part of the target object is located in the background blur image is cropped to obtain a cropped background blur image;

[0201] Adjustment is performed according to the cropped initial image, the cropped background blur image and the target part segmentation map to obtain adjusted image data of the target part.

[0202] In one embodiment, the adjustment module 1404 is used to:

[0203] Based on the area ratio of the target part in the initial image, a size corresponding to the target part segmentation map is determined.

[0204] In one embodiment, the adjustment module 1404 is used to:

[0205] Determining a size corresponding to the target part segmentation map based on the area ratio of the target part in the initial image and the background complexity of the initial image;

[0206] Among them, the background complexity is negatively correlated with the size corresponding to the target part segmentation map.

[0207] In one embodiment, the acquisition module 1402 is used to:

[0208] Get the initial image;

[0209] Blurring the background area based on the target object in the initial image to obtain a background blurred image;

[0210] A target part segmentation image for representing the target part is extracted from the initial image.

[0211] In one embodiment, the acquisition module 1402 is used to:

[0212] According to the focal length and aperture values, an image of the target object in a blurred background area is obtained; the image of the target object in a blurred background area includes an image after background blurring.

[0213] In one embodiment, the adjustment module 1404 is used to:

[0214] Combining the initial image, the background blur image and the target part segmentation map to obtain combined data;

[0215] Performing downsampling processing on the combined data to obtain downsampling features;

[0216] Perform feature fusion processing according to the down-sampled features to obtain fused features;

[0217] Performing upsampling processing on the fused features to obtain upsampled fused features;

[0218] The down-sampled features and the up-sampled fused features are fused to obtain adjusted image data of the target part.

[0219] In one embodiment, the adjustment module 1404 is used to:

[0220] Fully connect the downsampled features at the target scale to obtain fused features;

[0221] The number of channels of the fused features is less than the number of channels of the downsampled features.

[0222] In one embodiment, the target part segmentation map is a binary map, the first pixel value in the binary map represents the area where the target part is located, and the second pixel value in the binary map represents the area outside the target part.

[0223] In one of the embodiments, the target part segmentation map is confidence image data, and the pixel value of the target part segmentation map represents the confidence that each pixel is located in the area where the target part is located.

[0224] In one embodiment, the target part includes hair, and the target object includes a human face.

[0225] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0226] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.15As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0227] Those skilled in the art will understand that Fig.15 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0228] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0229] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0230] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0231] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0232] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0233] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0234] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an initial image, and a background blur image and a target part segmentation map corresponding to the initial image; the initial image includes a target object to which the target part belongs; Obtaining image data of the target part after adjustment according to the initial image, the background blur image and the target part segmentation map; The adjusted image data of the target part is fused with the background blurred image to obtain a target image.

2. The method according to claim 1, characterized in that: The step of fusing the adjusted image data of the target part with the background blurred image to obtain the target image comprises: Performing fuzzy processing based on the target part segmentation map to obtain the fusion weight of each point contained in the target part; According to the fusion weight, the adjusted image data of the target part is fused with the background blurred image to obtain a target image.

3. The method according to claim 2, characterized in that The fuzzy processing is performed based on the target part segmentation map to obtain the fusion weight of each point contained in the target part, including: Performing external expansion processing on the target part segmentation map to obtain an externally expanded target part segmentation map; The target part segmentation map after the expansion is fuzzy processed to obtain a fusion weight.

4. The method according to claim 1, characterized in that: The step of obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map comprises: Merging channels of the initial image, the background blur image and the target part segmentation map to obtain combined data; wherein the channels of the combined data include the channels of the initial image, the channels of the background blur image and the channels of the target part segmentation map; The target part in the combined data is adjusted to obtain image data of the adjusted target part.

5. The method according to claim 4, characterized in that The channels of the initial image and the channels of the background blurred image are both multiple, and the channel of the target part segmentation map is only one.

6. The method according to claim 1, characterized in that The step of obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map comprises: According to the size corresponding to the target part segmentation map, cropping the area where the target part of the target object in the initial image is located to obtain a cropped initial image; According to the size corresponding to the target part segmentation map, the region where the target part of the target object is located in the background blur image is cropped to obtain a cropped background blur image; Adjustment is performed according to the cropped initial image, the cropped background blur image and the target part segmentation map to obtain adjusted image data of the target part.

7. The method according to claim 6, characterized in that The method further comprises: Based on the area ratio of the target part in the initial image, a size corresponding to the target part segmentation map is determined.

8. The method according to claim 7, characterized in that The determining the size corresponding to the target part segmentation map based on the area ratio of the target part in the initial image includes: Determining a size corresponding to the target part segmentation map based on the area ratio of the target part in the initial image and the background complexity of the initial image; Among them, the background complexity is negatively correlated with the size corresponding to the target part segmentation map.

9. The method according to claim 1, characterized in that: The obtaining of the initial image, and the background blur image and the target part segmentation map corresponding to the initial image, comprises: Get the initial image; Blurring the background area based on the target object in the initial image to obtain a background blurred image; A target part segmentation image for representing the target part is extracted from the initial image.

10. The method according to claim 9, characterized in that The blurring of the background area based on the target object in the initial image to obtain a background blurred image includes: According to the focal length and aperture values, an image of the target object in a blurred background area is obtained; the image of the target object in a blurred background area includes an image after background blurring.

11. The method according to claim 1, characterized in that: The step of obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map comprises: Combining the initial image, the background blur image and the target part segmentation map to obtain combined data; Performing downsampling processing on the combined data to obtain downsampling features; Perform feature fusion processing according to the down-sampled features to obtain fused features; Performing upsampling processing on the fused features to obtain upsampled fused features; The down-sampled features and the up-sampled fused features are fused to obtain adjusted image data of the target part.

12. The method according to claim 11, characterized in that The performing feature fusion processing according to the down-sampled features to obtain fused features includes: Fully connect the downsampled features at the target scale to obtain fused features; The number of channels of the fused features is less than the number of channels of the downsampled features.

13. The method according to claim 1, characterized in that The target part segmentation map is a binary map, the first pixel value in the binary map represents the area where the target part is located, and the second pixel value in the binary map represents the area outside the target part.

14. The method according to claim 1, characterized in that The target part segmentation map is confidence image data, and the pixel value of the target part segmentation map represents the confidence that each pixel is located in the area where the target part is located.

15. The method according to claim 1, characterized in that The target part includes hair, and the target object includes a face.

16. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire an initial image, a background blur image corresponding to the initial image, and a target part segmentation map; the initial image includes a target object to which the target part belongs; An adjustment module, used for obtaining image data of the target part after adjustment according to the initial image, the background blurred image and the target part segmentation map; The fusion module is used to fuse the adjusted image data of the target part with the background blurred image to obtain a target image.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images