Image processing method, apparatus, device and computer program

Abstract

An image processing method for performing image matting on an original image includes first obtaining multiple segmented images containing different regions using a semantic region segmentation method, then drawing lines of different widths on the contours of the foreground regions based on these segmented images to obtain a target trimap, and finally generating a target image based on the target trimap. Because the target trimap draws lines of different widths on the contours of the foreground regions, it is possible to achieve directional image matting for each region, improving the image matting accuracy in regions that require finer image matting while ensuring the image matting accuracy in other regions, ultimately obtaining a matted image that is finely detailed and natural in effect.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to a Chinese patent application bearing application number 2021100625671 and title "Image processing method, apparatus, device and storage medium" filed with the China Patent Office on January 18, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of artificial intelligence, and in particular to an image processing method, apparatus, device, storage medium and computer program product. [Background technology]

[0003] With the development of computer technology, image processing has become more and more common. Among them, image matting is a very widely used image processing technique, which specifically refers to separating foreground areas in an image from background areas in an image.

[0004] In related technologies, image matting is usually achieved by using a segmentation method, specifically, by classifying each pixel point of an image and obtaining block segmentation results of different categories, thereby obtaining foreground regions of an image, such as human image regions and architectural regions.

[0005] However, when using the above method, each pixel point is assigned a fixed classification, which tends to result in rough edges in the foreground region and poor image matting. Summary of the Invention [Problem to be solved by the invention]

[0006] Embodiments of the present application provide an image processing method, an apparatus, a device, a storage medium and a computer program product. [Means for solving the problem]

[0007] In one aspect, there is provided a computing device implemented method of image processing, comprising the steps of: performing semantic image segmentation on an original image to obtain a first image, a second image, and a third image, wherein a foreground region in the first image is a region in the original image where a target object is located, the second image is a segmented image of a first target region of the target object, the third image is a segmented image of a second target region of the target object, and a sub-region of the foreground region includes the first target region and the second target region; generating a target trimap based on the first image, the second image and the third image, the target trimap including the foreground region and a line segment drawing region, the line segment drawing region being obtained by drawing a line segment on a contour line of the foreground region, different sub-regions of the foreground region corresponding to different line widths; performing an image matting process on the target object in the original image based on the target trimap to obtain a target image including the target object.

[0008] In another aspect, an apparatus for image processing, comprising: an image segmentation module for performing semantic image segmentation on an original image to obtain a first image, a second image, and a third image, where a foreground region in the first image is a region in the original image where a target object is located, the second image is a segmented image of a first target region of the target object, the third image is a segmented image of a second target region of the target object, and a sub-region of the foreground region includes the first target region and the second target region; a trimap generation module for generating a target trimap based on the first image, the second image and the third image, the target trimap including the foreground region and a line segment drawing region, the line segment drawing region being obtained by drawing a line segment on a contour line of the foreground region, and different sub-regions of the foreground region corresponding to different line widths; an image matting module for performing an image matting process on the target object in the original image based on the target trimap to obtain a target image including the target object.

[0009] In another aspect, a computing device including one or more processors and a memory, the memory storing at least one computer-readable instruction that is loaded and executed by the one or more processors to implement operations performed in an image processing method according to an embodiment of the present application.

[0010] In another aspect, one or more computer-readable storage media are provided having stored thereon at least one computer-readable instruction that, when loaded and executed by one or more processors, implements operations performed in an image processing method according to an embodiment of the present application.

[0011] In another aspect, a computer program product is provided, comprising computer readable instructions stored on a computer readable storage medium, such that one or more processors of a computing device read the computer readable instructions from the computer readable storage medium, and when the one or more processors execute the computer readable instructions, the computing device performs a method of image processing according to any of the above embodiments. [Brief description of the drawings]

[0012] In order to more clearly describe the technical solutions in the embodiments of the present application, the drawings necessary for the description of the embodiments will be briefly described below. However, the drawings in the following description are only some embodiments of the present application, and it is obvious that a person skilled in the art can obtain other drawings based on these drawings without any creative efforts.

[0013] [Figure 1] FIG. 2 illustrates the structure of a high-resolution network according to an embodiment of the present application. [Diagram 2] FIG. 2 illustrates the structure of an object context representation according to an embodiment of the present application. [Diagram 3] FIG. 1 illustrates an implementation environment for a method of image processing according to an embodiment of the present application. [Figure 4] 1 is a flow chart of a method of image processing according to an embodiment of the present application; [Diagram 5] 4 is a flowchart of another image processing method according to an embodiment of the present application. [Figure 6] FIG. 2 is a diagram showing a result of semantic region segmentation of an image according to an embodiment of the present application. [Figure 7] FIG. 2 illustrates a first trimap according to an embodiment of the present application. [Figure 8] FIG. 13 illustrates a second trimap according to an embodiment of the present application. [Figure 9] FIG. 13 illustrates a third trimap according to an embodiment of the present application. [Figure 10] FIG. 13 illustrates a target trimap according to an embodiment of the present application. [Figure 11] FIG. 2 illustrates an image matting model according to an embodiment of the present application. [Figure 12] FIG. 2 illustrates a target image according to an embodiment of the present application. [Figure 13] FIG. 2 illustrates a method of image processing according to an embodiment of the present application. [Figure 14] FIG. 1 shows the structure of an apparatus for image processing according to an embodiment of the present application; [Figure 15] FIG. 2 illustrates the structure of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in more detail below with reference to the drawings.

[0015] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as set forth in the appended claims.

[0016] It should be understood that the terms "first" and "second" in this application are used to distinguish between identical or similar items having essentially the same role and function, and there is no logical or chronological dependency between "first", "second", and "nth", and there is no restriction on the number and execution order. In the following description, various elements are described using terms such as first and second, but these elements should not be limited by the terms.

[0017] These terms are merely used to distinguish one element from another. For example, a first image can be called a second image, and a second image can be called a first image, without departing from the scope of various examples. The first and second images may both be images, and in some cases may be separate and distinct images.

[0018] Here, at least one refers to one or more, and for example, at least one image may be any integer number of images equal to or greater than one, such as one image, two images, three images, etc. Plural refers to two or more, and for example, multiple images may be any integer number of images equal to or greater than two, such as two images, three images, etc.

[0019] The image processing schemes according to the embodiments of the present application may use computer vision technology in artificial intelligence technology. For example, the semantic region segmentation process in the present application uses computer vision technology. Specifically, in the image processing schemes according to the embodiments of the present application, a high-resolution network can be used to extract image feature information, and an Object-Contextual Representations (OCR) technology can be used to calculate the semantic category of each pixel in the image.

[0020] High Resolution Network (HRNET) is a computational model for acquiring image feature information, which can maintain high-resolution representation throughout the entire computation process. HRNET starts with high-resolution convolution, gradually adds low-resolution convolution branches, and connects them in parallel. Referring to FIG. 1, FIG. 1 is a diagram showing the structure of the high-resolution network according to the present application. As shown in FIG. 1, the network connects feature maps of different resolutions in parallel, each resolution occupies one channel, and continuously exchanges information between the combinations of parallel computations through multi-resolution fusion throughout the entire process.

[0021] OCR is a computational model for indicating the semantic category of pixels in an image.

[0022] Referring to FIG. 2, FIG. 2 is a diagram showing the structure of the object context representation according to the present application. As shown in FIG. 2, the first step is to obtain a rough semantic region segmentation result, i.e., soft object regions, through the middle layer of the backbone. The second step is to calculate K sets (K>1) of vectors, i.e., object region representations, through the pixel representations and soft object regions output from the deep layer of the backbone, where each vector corresponds to a representation of one semantic category. The third step is to calculate a relationship matrix between the pixel representations and the object region representations. The fourth step is to weight and sum up each object region feature according to the numerical value in the relationship matrix of the pixel representation and the object region representation of each pixel to obtain an object context representation, i.e., OCR. Finally, an augmented representation is obtained as context information based on the OCR and the pixel representation, and the augmented representation can predict the semantic category of each pixel.

[0023] Below, key terms or abbreviations that may be used in the image processing scheme according to the embodiments of the present application will be briefly described.

[0024] Semantic Segmentation: For an input image, it is a process of dividing pixels with the same meaning into identical parts or regions based on the semantic understanding of each pixel, and obtaining several different semantic regions.

[0025] Foreground: The subject in an image, such as a person's image in a portrait shot.

[0026] Background: The environment in which the subject is placed in the image, such as a landscape, road, building, etc. in portrait photography.

[0027] Image Matting: An image processing technique that separates the foreground of an image from its background.

[0028] Trimap: An image that contains three markers: foreground, background, and mixed foreground-background regions. It is usually used as input to an image matting model along with the original image. In the following examples, the mixed foreground-background region is also called the line drawing region.

[0029] Identification value: A numerical value for identifying the color of a pixel point in an image. For example, if the identification value of a pixel point is 255, the RGB (Red-Green-Blue) value of this pixel point is (255, 255, 255), which indicates that it is expressed as white, and if the identification value of a pixel point is 0, the RGB value of the pixel point is (0, 0, 0), which indicates that it is expressed as black, and if the identification value of a pixel point is 128, the RGB value of the pixel point is (128, 128, 128), which indicates that it is expressed as gray.

[0030] Open Source Computer Vision Library (OpenCV): A cross-platform computer vision and machine learning software library that can run on a variety of operating systems. OpenCV can be used to develop real-time image processing, computer vision, and pattern recognition programs.

[0031] findContours: An OpenCV function for detecting contours in an image.

[0032] drawContours: An OpenCV function for drawing contours on an image.

[0033] Image matting model: A computational model for calculating the probability that each pixel point in an original image belongs to the foreground based on the original image and the trimap. For example, image matting includes the IndexNet model, the GCAMatting model, and the ContextNet model.

[0034] The implementation environment of the image processing method according to the embodiment of the present application will now be described.

[0035] 3 is a diagram showing an implementation environment of the image processing method according to the embodiment of the present application. The implementation environment includes a terminal 301 and a server 302.

[0036] The terminal 301 and the server 302 may be directly or indirectly connected by wired or wireless communication, but the present application is not limited thereto. In some embodiments, the terminal 301 may be, but is not limited to, a smartphone, a tablet, a laptop, a desktop computer, etc. An application program may be installed and executed on the terminal 301. In some embodiments, the application program may be, for example, a social application program, an image processing application program, or a photography application program. Exemplarily, the terminal 301 is a terminal used by a user, and when the social application program is executed on the terminal 301, the user may matte a person image in a photo through the social application program.

[0037] The server 302 may be an independent physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that can provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Networks (CDNs), and big data and artificial intelligence platforms. The server 302 is used to provide background services to application programs executed on the terminal 301.

[0038] In some embodiments, in the image processing process, the server 302 performs the main calculations and the terminal 301 performs the secondary calculations, or the server 302 performs the secondary calculations and the terminal 301 performs the main calculations, or the server 302 or the terminal 301 can perform the calculations independently.

[0039] In some embodiments, the terminal 301 generally refers to one of multiple terminals, and the present embodiment will be described using only the terminal 301 as an example. Those skilled in the art will appreciate that the number of the terminals 301 may be greater. For example, the number of the terminals 301 may be tens, hundreds, or more, and in this case, the implementation environment of the image processing method may also include other terminals. The embodiments of the present application do not limit the number of terminals and the type of devices.

[0040] In some embodiments, the wireless or wired network uses standard communication technologies and / or protocols. The network may typically be the Internet, but may be any network including, but not limited to, a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), any combination of mobile, wired or wireless networks, private networks or virtual private networks, etc. In some embodiments, technologies and / or formats including HTML, Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. Also, the links may be encrypted in whole or in part using conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In some other embodiments, custom and / or proprietary data communication technologies may be used in place of or in addition to the above data communication technologies.

[0041] In the embodiment of the present application, a method for image processing is provided that can meet the needs of matting target objects in various scenes. In addition, for an original image containing a target object, local regions such as a first target region and a second target region of the target object, which are relatively fine, can be automatically matted from the original image, and the matting effect is delicate and natural. FIG. 4 is a flow chart of the method for image processing according to the embodiment of the present application. As shown in FIG. 4, in the embodiment of the present application, the application to a computer device is taken as an example. The computer device may be a terminal or a server, and the above method includes the following steps 401, 402, and 403.

[0042] In 401, a semantic region segmentation of an original image is performed to obtain a first image, a second image, and a third image, in which a foreground region in the first image is a region in the original image where a target object is located, the second image is a segmented image of a first target region of the target object, and the third image is a segmented image of a second target region of the target object.

[0043] In the present embodiment, the source image is the image that needs image matting, and the target object is the object in the source image that should be separated to generate the target image.

[0044] Note that the first image, the second image, and the third image all substantially belong to segmented images, and since the first image is a segmented image obtained by segmenting the entire target object, the foreground region in the first image is all elements of the target object in the original image. Since the second image is a segmented image obtained by segmenting a local portion called the first target area of ​​the target object, the foreground region in the second image is all elements in the first target area of ​​the target object, and all areas other than the first target area belong to the background region in the second image. Similarly, since the second image is a segmented image obtained by segmenting a local portion called the second target area of ​​the target object, the foreground region in the third image is all elements in the second target area of ​​the target object, and all areas other than the first target area belong to the background region in the third image.

[0045] It should be understood that the target object may include at least one of a human portrait, an animal image, a plant image, and the like in the original image.

[0046] It should be understood that the foreground region of the target object in the first image is a region for showing the entire target object, and the first target region and the second target region are regions of local parts of the target object, and therefore the first target region and the second target region are both sub-regions of the foreground region of the target object in the first image.

[0047] In one embodiment, the first target area may be an area of ​​the target object that requires refined image matting. The second target area may be an area of ​​the target object associated with the first target area that has a different image matting refinement requirement than the first target area. Specifically, the image matting refinement requirement of the second target area may be lower than the image matting refinement requirement of the first target area. For example, the second target area has lower detailed information than the first target area, and therefore has a lower image matting refinement requirement than the first target area.

[0048] In one embodiment, when the target object is a portrait, the first target region and the second target region may be a hair region and a face region, respectively.

[0049] In one embodiment, when the target object is an animal image, the first target region and the second target region may be a hair region and a head region, respectively.

[0050] In one embodiment, when the target object is a plant image, the first target region and the second target region may be a leaf region and a branch region, respectively.

[0051] The target object may be a portrait of a real person, a portrait of a cartoon character, or a portrait of an anime or manga character.

[0052] At 402, a target trimap is generated based on the first image, the second image, and the third image, the target trimap including a foreground region and a line segment drawing region, the line segment drawing region being obtained by drawing lines on a contour of the foreground region, and different sub-regions of the foreground region corresponding to different line widths.

[0053] In the embodiment of the present application, the sub-region refers to a partial region in the foreground region, including a first target region and a second target region, and so on.

[0054] In 403, an image matting process is performed on the target object in the original image based on the target trimap to obtain a target image including the target object.

[0055] In the present embodiment, the image matting process is a process of obtaining a target image by separating a target object from a background region in an original image.

[0056] In the embodiment of the present application, when performing image matting on an original image, first, a method of semantic region division is used to obtain a number of divided images including different regions, and then, based on these divided images, lines of different widths are drawn on the contours of the foreground region to obtain a target trimap, and finally, a target image is generated based on the target trimap. Since the target trimap is drawn on the contours of the foreground region with lines of different widths, it is possible to realize directional image matting for each region, which can improve the image matting accuracy of the region that needs to be subjected to fine image matting, while ensuring the image matting accuracy of the other regions, and finally obtain a matting image with fine and natural effects. In addition, the image matting process is fully automated, which greatly improves the efficiency of image matting.

[0057] The above Fig. 4 only shows the basic flow of the present application, and the solution of the present application will be further described below based on specific embodiments.

[0058] Fig. 5 is a flow chart of another image processing method according to an embodiment of the present application, in which the first target area is a hair area and the second target area is a face area. As shown in Fig. 5, in the embodiment of the present application, the application to a terminal is taken as an example. The above method includes the following steps 501 to 508.

[0059] In 501, the original image is acquired.

[0060] In some embodiments, the terminal provides an image matting function, and the user can perform an image matting operation on the terminal, and the terminal responds to the image matting operation to obtain the original image. In some embodiments, the original image is a local image stored in the terminal or an online image, but the embodiments of the present application do not limit the source of the original image.

[0061] In some embodiments, an image processing interface of the original image is displayed on the terminal, the image processing interface includes an image matting option, a cropping option, etc., the user can perform a hit operation on the image matting option, and the terminal responds to the hit operation to obtain the original image.

[0062] In some embodiments, an image processing interface is displayed on the terminal, the image processing interface includes an image matting option, the user can perform a hit operation on the image matting option, the terminal responds to the hit operation by displaying an image selection interface, the user can perform a click operation on the image to be matted, and select an original image to perform image matting, and the terminal responds to the click operation by acquiring the original image.

[0063] It should be noted that the embodiment of the present application does not limit the method of acquiring the original image of the terminal.

[0064] At 502, the original image is input into an image segmentation model.

[0065] In an embodiment of the present application, an image segmentation model is used to calculate a semantic category of each pixel point in an input original image based on the input original image, and output at least one image of the original image.

[0066] In some embodiments, the image segmentation model is a calculation model combining the HRNET model and the OCR model as the HRNET-OCR model. The calculation process of the HRNET-OCR model is as follows: First, the feature extraction of the original image is performed by the HRNET model to obtain the feature information of the original image. Then, the obtained feature information is input into the backbone of the OCR model. Then, the semantic category of each pixel point in the original image is calculated based on the OCR model, for example, the semantic categories include hair, nose, eyes, torso, clothing, and buildings. Finally, at least one image of the original image is output based on the semantic category of each pixel point. The specific calculation process of the above HRNET-OCR model has been described in detail in combination with FIG. 1 and FIG. 2, so the description is omitted here.

[0067] In addition, in practical applications, at least one image of the original image can be output by adjusting some structures in the HRNET-OCR model, but the embodiments of the present application do not limit the configuration of the structures in the HRNET-OCR model. In some other embodiments, the image segmentation model can be further realized by other network models, but the embodiments of the present application do not limit the type of the image semantic region segmentation model.

[0068] At 503, a first image, a second image, and a third image output from the image segmentation model are obtained, where the first image includes a foreground region in which a target object is located in the original image, the second image is a segmented image of a hair region of the target object, and the third image is a segmented image of a face region of the target object.

[0069] In the embodiment of the present application, by classifying each pixel point in the original image with an image segmentation model, the terminal can obtain three divided images of the first image, the second image and the third image in this step. Here, each image includes two regions, and the two regions are marked with different identification values. For example, take the first image as an example, the first image includes a foreground region and a background region. Here, the identification value of each pixel point in the foreground region is 255, and the identification value of each pixel point in the background region is 0. In addition, in actual applications, developers can flexibly set the identification value as needed, but the embodiment of the present application is not limited thereto.

[0070] For example, refer to Fig. 6, which is a diagram showing a result of semantic region segmentation of an image according to an embodiment of the present application. As shown in Fig. 6, an original image is a portrait of a person, the image shown in Fig. 6(a) is a first image including a foreground region 1 and a background region 2, the foreground region 1 including all elements of the portrait of the person, the image shown in Fig. 6(b) is a second image including a hair region 3 and a background region 4, and the image shown in Fig. 6(c) is a third image including a face region 5 and a background region 6.

[0071] At 504, a first trimap is generated based on the first image and the second image, the first trimap including a foreground region, a first line drawing subregion, and a second line drawing subregion.

[0072] In an embodiment of the present application, the first line segment drawing sub-region covers a contour of a hair region adjacent to a background region in the first image, and the second line segment drawing sub-region covers a contour of a non-hair region other than the hair region in the foreground region. The first line width is larger than the second line width, and the first line width is used to draw the first line segment drawing sub-region, and the second line width is used to draw the second line segment drawing sub-region.

[0073] Here, the first trimap further includes a background region, and the identification values ​​of the first line segment drawing sub-region and the second line segment drawing sub-region are different from the identification value of the foreground region and different from the identification value of the background region, for example, the identification value of each pixel point in the foreground region is 255, the identification value of each pixel point in the background region is 0, and the identification value of each pixel point in the first line segment drawing sub-region and the second line segment drawing sub-region is 128.

[0074] In actual applications, developers can flexibly set the identification value of the line drawing area as needed, but the embodiments of the present application are not limited thereto.

[0075] Fig. 7 is a diagram showing a first trimap according to an embodiment of the present application. As shown in the right diagram in Fig. 7, the first trimap includes a foreground region 7, a background region 8, a first line segment drawing subregion 9, and a second line segment drawing subregion 10, where the first line segment drawing subregion 9 is drawn based on a first line width, and the second line segment drawing subregion 10 is drawn based on a second line width.

[0076] In the embodiment of the present application, there is a difference in area range between foreground regions in different images, and there is also a difference in area range between background regions in different images, for example, referring to background region 2 and background region 4 in Fig. 6, both are background regions, although there is a clear difference in their area ranges.Furthermore, referring to foreground region 1 in Fig. 6 and foreground region 7 in Fig. 7, both are foreground regions, although there is a subtle difference in their area ranges.

[0077] Hereinafter, a specific embodiment in which the terminal generates the first trimap in this step will be described in detail. This step includes the following steps 5041 to 5045.

[0078] In 5041, the complete contour of the foreground region is obtained in the first image.

[0079] Here, the complete contour of the foreground region is the boundary line between the foreground region and the background region. Based on the acquired first image, the terminal acquires the complete contour of the foreground region from the first image through a contour detection algorithm. In some embodiments, the contour detection algorithm can be realized by a findContours function, but the embodiments of the present application are not limited thereto.

[0080] In 5042, a line segment is drawn to the complete contour of the foreground region based on the second line width to obtain a second trimap.

[0081] Here, the second trimap includes a foreground region and a third line drawing subregion that covers a complete contour of the foreground region. The second line width is calculated based on the dimensions of the original image. In some embodiments, the second line width is calculated according to the following formula (1):

[0082] S=[min(width,height) / 1000]*N (1)

[0083] In the formula, S is the second line width, width and height are the width and height of the original image respectively, min() is the minimum value function, and min(width, height) indicates selecting the minimum value from the width and height of the original image, and N is the default line size, for example, N may be 17, and the embodiments of the present application are not limited thereto.

[0084] After obtaining the complete contour of the foreground region, the terminal uses a contour drawing algorithm to draw a line segment on the complete contour according to the second line width, and the identification value of the line segment is different from the identification values ​​of both the foreground region and the background region. In some embodiments, the contour drawing algorithm can be realized by a drawContours function, for example, the identification value of the foreground region is 255 and the identification value of the background region is 0, and the line segment is drawn on the complete contour of the foreground region according to the following formula (2):

[0085] cv::drawContours(segResult,contours,-1,Scalar(128,128,128),S) (2)

[0086] where segResult is the first image, contours is the complete contours of the foreground regions found by the findContours function, -1 indicates that all contours are to be operated on, Scalar is the discrimination value, Scalar(128,128,128) indicates that the color values ​​of the R, G and B channels of the RGB channels are all set to 128, and S is the second line width.

[0087] The line segment drawing method is to manipulate the acquired complete contour, that is, to cover the complete contour. For example, if pixel points A1 to A10 are included in the complete contour acquired by the findContours function, these pixel points are manipulated to realize line segment drawing. That is, the third line segment drawing sub-region obtained by line segment drawing not only covers the foreground region of the first image, but also covers the background region in the first image.

[0088] Figure 8 shows a second trimap according to an embodiment. The left diagram in Figure 8 shows a first image, and the right diagram in Figure 8 shows a second trimap, which includes a foreground region 11, a background region 12, and a third line segment drawing sub-region 13, which is drawn based on a second line width.

[0089] At 5043, the complete contour of the hair region is obtained in the second image.

[0090] Here, the complete contour of the hair region is the boundary between the hair region and the background region. The terminal obtains the complete contour of the hair region from the second image through a contour detection algorithm based on the acquired second image. In some embodiments, the contour detection algorithm can be realized by a findContours function, but the embodiments of the present application are not limited thereto.

[0091] In 5044, a line segment is drawn to the complete contour of the hair region based on the first line width to obtain a third trimap.

[0092] Here, the third trimap includes a hair region and a fourth line segment drawing sub-region covering the complete contour of the hair region. The first line width is M times the second line width (M>1). For example, the first line width is three times the second line width, i.e., if the second line width is S, the first line width is S*3, but the embodiment of the present application is not limited thereto.

[0093] Note that the method for drawing lines on the complete contour of the hair region in this step is similar to that in step 5042 above, so an example will be given using only equation (3) below, and a detailed description will be omitted here.

[0094] cv::drawContours(segResultHair,contours,-1,Scalar(128,128,128),S*3) (3)

[0095] where segResultHair is the second image, contours is the complete contour of the hair region detected by the findContours function, -1 indicates to operate on all contours, Scalar is the discrimination value, Scalar(128,128,128) indicates to set the color values ​​of R, G and B channels of the RGB channels to 128, and S*3 is the first line width.

[0096] Fig. 9 is a diagram showing a third trimap according to an embodiment. The left diagram in Fig. 9 shows the second image, and the right diagram in Fig. 9 shows the third trimap, which includes a foreground region 14, a background region 15 and a fourth line segment drawing subregion 16, and the fourth line segment drawing subregion 16 is drawn based on the first line width.

[0097] In 5045, a merge process is performed on the second trimap and the third trimap to obtain the first trimap.

[0098] Here, the merging process for the second trimap and the third trimap means that the maximum discrimination value at the same position in the two trimaps is set as the discrimination value at the corresponding position in the first trimap.

[0099] A specific embodiment of this step will be described in detail below. This step includes the following steps A to C.

[0100] In step A, a first identification value of each pixel point in the second trimap is obtained, and the first identification value is used to identify the color of the pixel point in the second trimap.

[0101] In step B, a second identification value of each pixel point in the third trimap is obtained, and the second identification value is used to identify the color of the pixel point in the third trimap.

[0102] In step C, a first trimap is generated based on the magnitude relationship between the first and second discrimination values.

[0103] Here, an embodiment of step C includes comparing a first identification value of a pixel point at a given location in the second trimap with a second identification value of a pixel point at the same location in the third trimap, and taking the maximum of the first and second identification values ​​as a third identification value for the pixel point at the same location in the first trimap, which third identification value is used to identify the color of the pixel point in the first trimap.

[0104] For example, still referring to FIG. 7, after the terminal obtains the second trimap and the third trimap, the terminal realizes a merging process of the two trimaps according to the following equation (4) to obtain the first trimap. Equation (4) is as follows:

[0105] Pixel result =Pixel leftUp >Pixel leftDown ?Pixel leftUp :Pixel leftDown (4)

[0106] In the ceremony, Pixel result is the first trimap, that is, the right diagram in Figure 7, and Pixel leftUp is the second trimap, i.e., the upper left diagram in Fig. 7, Pixel leftDown is the third trimap, that is, the lower left diagram in FIG.

[0107] As shown in the right diagram in FIG. 7, in the first trimap generated in step 5045, the line width for drawing the first line segment drawing sub-region 9 corresponding to the hair region is larger than the line width for drawing the second line segment drawing sub-region 10 corresponding to other regions other than the hair region. Through the method of obtaining different line segment drawing regions by drawing based on such different line widths, the image matting resolution of relatively complex image matting regions such as hair regions can be improved in the subsequent image matting process, and the image matting effect of this portion of the region can be improved.

[0108] At 505, a target trimap is generated based on the third image and the first trimap.

[0109] In some embodiments, the target trimap includes a foreground region and a line segment drawing region, the line segment drawing region is obtained by drawing lines on the contour of the foreground region, and different sub-regions of the foreground region correspond to different line widths. In some embodiments, the foreground region further includes a body region of the target object, where in the target trimap, the line width corresponding to the hair region is larger than the line width corresponding to the body region, and the line width corresponding to the body region is larger than the line width corresponding to the face region.

[0110] FIG. 10 is a diagram showing a target trimap according to an embodiment. The upper right diagram in FIG. 10 shows the target trimap, which includes a foreground region 17, a background region 18, and a line segment drawing region 19. In the line segment drawing region 19, the line width corresponding to the hair region is larger than the line width corresponding to the body region, and the line width corresponding to the body region is larger than the line width corresponding to the face region. For example, the relationship of line widths in the line segment drawing region 19 can be continued to refer to the lower right diagram in FIG. 10, which includes line segment drawing regions 19a, 19b, and 19c, in which 19a shows the line width corresponding to the hair region, 19b shows the line width corresponding to the face region, and 19c shows the line width corresponding to the body region, as shown in the diagram, the line width 19a corresponding to the hair region is larger than the line width 19c corresponding to the body region, and the line width 19c corresponding to the body region is larger than the line width 19b corresponding to the face region.

[0111] Hereinafter, a specific embodiment in which the terminal generates a target trimap in this step will be described in detail. This step includes the following steps 5051 to 5052.

[0112] In 5051, a target overlap region of the first trimap is determined based on the face region in the third image, the target overlap region being an overlap region between the face region and the second line segment drawing sub-region.

[0113] Here, after obtaining the first trimap, the terminal determines a target overlap region from the first trimap according to the positions of the pixel points of the face region in the third image. For example, still referring to Fig. 10, as shown in the lower right diagram of Fig. 10, the diagram includes a target overlap region 20, which is the overlap region between the face region in the third image and the second line drawing sub-region 19 in the first trimap.

[0114] At 5052, a target discrimination value is assigned to the pixel points of the target overlap region to generate a target trimap, the target discrimination value being used to discriminate the color of the pixel points in the face region.

[0115] Here, in the first trimap, the identification value of the target overlap region is the identification value of the second line segment drawing sub-region, and in this step, the identification value of the target overlap region in the first trimap is changed, and the target identification value is assigned to the pixel points of this region to generate a target trimap. For example, if the identification value of the face region is 255 and the identification value of the second line segment drawing sub-region is 128, in the first trimap, the identification value of the target overlap region was 128, but after this step 5052, the identification value of 255 is reassigned to the pixel points of the target overlap region to obtain a target trimap.

[0116] In some embodiments, the above steps 5051 and 5052 can be realized by the following step (5).

[0117] Pixel=Pixel∈Φ{Face}?255:Pixel trimp (5)

[0118] In the formula, Φ{Face} denotes the face region, 255 denotes the target discrimination value, and Pixel trimp is the target trimap. By assigning the target discrimination value to the pixel points belonging to the face region of the first trimap using the formula (5), the face region can be prevented from participating in the process of calculating the transparency using the image matting model in the following steps 506 to 507.

[0119] Through the above steps 501 to 505, after the terminal obtains the original image, it automatically generates a target trimap, in which the line widths corresponding to different sub-regions of the foreground region are different.

[0120] In addition, in practical applications, the marking of the foreground-background mixed area in the trimap will directly affect the resolution of the image matting effect, and marking the foreground area in the foreground-background mixed area in the trimap will cause inaccurate image matting results.

[0121] In the embodiment of the present application, the line drawing area in the target trimap is the foreground and background mixed area. In the process of automatically generating the target trimap by the terminal, line drawing is performed for the hair area and other areas other than the hair area according to different line widths to ensure the image matting range of complex areas such as the hair area, improve the image matting accuracy of this area, and assign the same target identification value as the foreground area to the pixel points belonging to the face area, so as to protect the key areas in the portrait and avoid the loss of details in the image matting.

[0122] At 506, the target trimap and the original image are input into an image matting model.

[0123] In some embodiments, the image matting model calculates the probability that each pixel point in the original image belongs to the target image based on the input target trimap and original image, and outputs the transparency. In some embodiments, the transparency is calculated according to the following formula (6):

[0124] I = α * F + (1 - α) * B (6)

[0125] In the formula, I denotes the original image, F denotes the foreground, i.e. the region containing all the elements of the target object, B denotes the background, and α is the transparency, which is used to indicate the proportion of the foreground color in the original image. Equation (6) shows that the original image is a superposition of the foreground and background with a certain transparency.

[0126] In some embodiments, the image matting model may be an IndexNet image matting model, or a GCAMatting image matting model, or a ContextNet model, etc., and the embodiments of the present application do not limit the specific type of the image matting model.

[0127] Hereinafter, this step will be described by taking the IndexNet image matting model as an example. For example, refer to FIG. 11, which is a diagram showing an image matting model according to the embodiment of the present invention. As shown in FIG. 11, the target trimap and the original image are input, and a rough Alpha (i.e., α) map and Alpha prediction loss are obtained, and the rough foreground and background are synthesized at the same time, and then compared with the original image to obtain the image synthesis loss. Finally, a fine Alpha map is obtained by optimizing using a convolution layer, and the fine result of image matting, that is, the Alpha value of each pixel point is output.

[0128] At 507, a transparency output from the image matting model is obtained, and the transparency is used to indicate the probability that a pixel point belongs to a target object.

[0129] In 508, an image matting process is performed on the original image based on the transparency to obtain a target image including the target object.

[0130] In the embodiment of the present application, the image matting process in step 508 is a process of separating the target object in the original image from the background based on the transparency of each pixel point to obtain the target image. For example, refer to FIG. 12, which is a diagram showing a target image according to the embodiment of the present application. The left side of FIG. 12 shows the original image, and the upper right side of FIG. 12 shows the target image obtained by the present method, in which the portrait of the person has been subjected to fine image matting with smooth hair tips, and the face is also perfect, and the lower right side of FIG. 12 shows the target image obtained according to the image segmentation method in the related art, in which the image is segmented accurately, but the hair tips of the portrait of the person are very coarse, and there is a loss of details in the face.

[0131] In the embodiment of the present application, when performing image matting on an original image, first, a method of semantic region division is used to obtain a number of divided images including different regions, and then, based on these divided images, lines of different widths are drawn on the contours of the foreground region to obtain a target trimap, and finally, a target image is generated based on the target trimap. Since the target trimap is drawn on the contours of the foreground region with lines of different widths, it is possible to realize directional image matting for each region, which can improve the image matting accuracy of the region that needs to be subjected to fine image matting, while ensuring the image matting accuracy of the other regions, and finally obtain a matting image with fine and natural effects. In addition, the image matting process is fully automated, which greatly improves the efficiency of image matting.

[0132] The image processing method according to the embodiment of the present application will be exemplarily briefly summarized below with reference to Fig. 13. As shown in Fig. 13, it includes the following six image processing stages:

[0133] First, an original image is obtained, the original image being an image including a portrait of a person.

[0134] Second, perform image semantic region segmentation on the original image based on the HRNET-OCR model, and obtain three kinds of segmented images based on the segmentation results. The first type of segmented image includes the foreground region, which includes all elements of the human portrait; the second type of segmented image includes the hair region, which has relatively clear edge lines between the human body and the background, but the hair often blends badly with the background due to its shape characteristics, so it is necessary to focus on image matting; the third type of segmented image includes the face region, which can be understood as the protection region, and the face is an important attention part of the human portrait, and if it is incorrectly matted or damaged, it will have a great impact on the appearance, so it is necessary to protect this part of the region from being damaged by image matting.

[0135] The third is to draw a line segment on the contour of the foreground region at the basic size for the first type of divided image to obtain a second trimap, and to draw a line segment on the contour of the hair region at three times the basic size for the second type of divided image to obtain a third trimap, where the discrimination value of the drawn line is 128.

[0136] The fourth performs a merge operation on the second trimap and the third trimap to obtain the merged first trimap.

[0137] Fifth, re-set the face region in the first trimap as a foreground marker, i.e. re-assign 255 to each pixel point of the face region in the first trimap to obtain the target trimap.

[0138] Sixth, based on the target trimap, finally obtain the target image, i.e., the person portrait in the original image.

[0139] The above target trimap uses lines of different widths to draw the contours of the foreground region, so that directional image matting can be realized for each region, the image matting accuracy of the region that needs fine image matting can be improved, and the image matting accuracy of the other regions can be ensured, so that a matting image with fine and natural effect can be finally obtained. In addition, the above image matting process can be fully automated, which greatly improves the efficiency of image matting.

[0140] Illustratively, application scenes of the image processing method according to the embodiments of the present application include, but are not limited to, the following:

[0141] Scene 1, Stamp Scene

[0142] With the popularity of sticker culture, many applications have also added sticker creation features, which makes it easier for users to express their emotions and moods by creating stickers. In some scenarios, users want to create their desired emoji by matting a portrait in a picture and adding stickers, text, background, etc. based on the portrait.

[0143] For example, a terminal provides a function of creating a stamp of a person's portrait through an application. A user performs an operation on the terminal and inputs an original image to which a person's portrait is to be matted. After obtaining the original image, the terminal automatically mattes the person's portrait in the original image using an image processing method according to an embodiment of the present application, and displays the image on the terminal. Thereby, the user can then perform other image processing operations based on the person's portrait to obtain a stamp desired by the user. Exemplarily, the process of the terminal matting a person's portrait includes the following steps 1 to 8.

[0144] 1. The terminal acquires the original image. 2. The terminal inputs the original image into an image segmentation model. 3. A terminal obtains a first image, a second image and a third image output from the image segmentation model, where the first image includes a foreground region where a portrait of a person is located in the original image, the second image includes a hair region of the portrait of the person, and the third image includes a face region of the portrait of the person. 4. The terminal generates a first trimap based on the first image and the second image, where the first trimap includes a foreground region, a first line segment drawing sub-region and a second line segment drawing sub-region. 5. The terminal generates a target trimap based on the third image and the first trimap. 6. The terminal inputs the target trimap and the original image into an image matting model. 7. The terminal obtains a transparency output from the image matting model, and the transparency is used to indicate the probability that a pixel point belongs to a human portrait. 8. The terminal performs image matting on the original image based on the target trimap to obtain a target image containing a portrait of a person, and then the user creates a stamp based on the target image.

[0145] According to the image processing method of the embodiment of the present application, automatic matting of portraits can be realized, and the matted portraits have a detailed and natural effect, which can meet the individual needs of users in creating stamps.

[0146] Scene 2, live streaming scene

[0147] In some live streaming scenes, in order to protect personal privacy, live streamers may choose to hide their real background environment, and only display the live streamer's portrait on the live streaming screen, or add other virtual backgrounds based on the live streamer's portrait.

[0148] For example, the terminal provides a portrait mode during live streaming. When the live streamer turns on the portrait mode, the terminal obtains the original image of each frame captured by the camera in real time, and then uses the image processing method according to the embodiment of the present application to matte the live streamer's portrait in the original image of each frame, and generates a live streaming screen in real time for live streaming. The specific process of matting the portrait by the terminal is similar to the above scene 1, so the description is omitted here.

[0149] Thus, the image processing method according to the embodiment of the present application realizes automatic matting of human images, and can be directly applied to scenes requiring real-time matting of human portraits.

[0150] In addition, each step in the flowchart of each embodiment is shown in sequence according to the direction of the arrow, but it should be understood that these steps are not necessarily performed in the order indicated by the arrow. Unless expressly described in this specification, there is no strict restriction on the order of execution of these steps, and these steps may be performed in other orders. In addition, at least some of the steps in the flowchart of each embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily performed simultaneously, but may be performed at different times, and the order of execution of these steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least some of the steps or stages in other steps.

[0151] 14 is a diagram showing the structure of an apparatus for image processing according to an embodiment of the present application, which is used to perform the execution steps of the above image processing method. Referring to FIG. 14, the apparatus includes an image segmentation module 1401, a trimap generation module 1402 and an image matting module 1403.

[0152] The image segmentation module 1401 performs semantic image segmentation on the original image to obtain a first image, a second image, and a third image, where the first image includes a foreground region in which a target object is located in the original image, the second image includes a hair region of the target object, and the third image includes a facial region of the target object.

[0153] A trimap generation module 1402 generates a target trimap based on the first image, the second image and the third image, where the target trimap includes the foreground region and a line segment drawing region, the line segment drawing region is obtained by drawing lines on the contour line of the foreground region, and different sub-regions of the foreground region correspond to different line widths.

[0154] The image matting module 1403 performs an image matting process on the original image based on the target trimap to obtain a target image including the target object.

[0155] In some embodiments, the foreground region further includes a body region of the target subject, where in the target trimap, a line width corresponding to the hair region is greater than a line width corresponding to the body region, and the line width corresponding to the body region is greater than a line width corresponding to the face region.

[0156] In some embodiments, the trimap generation module 1402 includes a first generation unit and a second generation unit.

[0157] A first generation unit generates a first trimap based on the first image and the second image, where the first trimap includes the foreground region, a first line segment drawing sub-region and a second line segment drawing sub-region.

[0158] Here, the first line segment drawing sub-region covers the contour of the hair region adjacent to the background region, and the second line segment drawing sub-region covers the contour of other regions in the foreground region other than the hair region, the first line width is larger than the second line width, the first line width is used to draw the first line segment drawing sub-region, and the second line width is used to draw the second line segment drawing sub-region.

[0159] A second generation unit generates the target trimap based on the third image and the first trimap.

[0160] In some embodiments, the first generation unit obtains a complete contour of the foreground region in the first image, draws lines on the complete contour of the foreground region based on the second line width, and obtains a second trimap including the foreground region and a third line segment drawing sub-region that covers the complete contour of the foreground region, obtains a complete contour of the hair region in the second image, draws lines on the complete contour of the hair region based on the first line width, and obtains a third trimap including the hair region and a fourth line segment drawing sub-region that covers the complete contour of the hair region, and performs a merging operation on the second trimap and the third trimap to obtain the first trimap.

[0161] In some embodiments, the first line width is M times the second line width (M>1).

[0162] In some embodiments, the first generation unit further obtains a first identification value for each pixel point in the second trimap, where the first identification value is used to identify a color of the pixel point in the second trimap, obtains a second identification value for each pixel point in the third trimap, where the second identification value is used to identify a color of the pixel point in the third trimap, and generates the first trimap based on a magnitude relationship between the first identification value and the second identification value.

[0163] In some embodiments, the first generation unit further compares a first identification value of a pixel point at an arbitrary position of the second trimap with a second identification value of a pixel point at the same position of the third trimap, and takes a maximum of the first identification value and the second identification value as a third identification value of the pixel point at the same position of the first trimap, which is used to identify a color of the pixel point in the first trimap.

[0164] In some embodiments, the second generation unit determines a target overlap region of the first trimap based on a face region in the third image, the target overlap region being an overlap region between the face region and a second line drawing sub-region, assigns target identification values ​​to pixel points of the target overlap region, and generates the target trimap, the target identification value being used to identify a color of the pixel points in the face region.

[0165] In some embodiments, the image matting module 1403 obtains a transparency for each pixel point in the original image based on the target trimap, which is used to indicate the probability that the pixel point belongs to the target object, and performs an image matting process on the original image based on the transparency to obtain the target image.

[0166] In some embodiments, the image segmentation module 1401 further acquires the original image, and inputs the original image into an image segmentation model for outputting at least one image of the original image by calculating a semantic category of each pixel point in the original image based on the input original image, and obtains the first image, the second image and the third image output from the image segmentation model.

[0167] In some embodiments, the image matting module 1403 further inputs the target trimap and the original image into an image matting model for outputting the transparency by calculating the probability that each pixel point in the original image belongs to the target image based on the input target trimap and the original image, and obtains the transparency output from the image matting model.

[0168] In the embodiment of the present application, when performing image matting on an original image, first, a method of semantic region division is used to obtain a number of divided images including different regions, and then, based on these divided images, lines of different widths are drawn on the contours of the foreground region to obtain a target trimap, and finally, a target image is generated based on the target trimap. Since the target trimap is drawn on the contours of the foreground region with lines of different widths, it is possible to realize directional image matting for each region, which can improve the image matting accuracy of the region that needs to be subjected to fine image matting, while ensuring the image matting accuracy of the other regions, and finally obtain a matting image with fine and natural effects. In addition, the image matting process is fully automated, which greatly improves the efficiency of image matting.

[0169] In addition, the image processing device according to the above embodiment is described as merely dividing the above functional modules when performing image processing. In actual application, the allocation of the above functions may be completed by different functional modules as necessary, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above-described functions. All or part of the modules in the above device may be realized by software, hardware, or a combination thereof. Each of the above modules may be built into a processor in a computer device in the form of hardware, or may be independent of the processor, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules. In addition, the image processing device according to the above embodiment belongs to the same idea as the embodiment of the image processing method, and the specific realization process thereof has been described in detail in the embodiment of the method, so that the description will be omitted here.

[0170] In an exemplary embodiment, a computer device is further provided. Taking the computer device as a terminal as an example, FIG. 15 illustrates a structure of a terminal 1500 according to an exemplary embodiment of the present application. The terminal 1500 may be a smartphone, a tablet, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop or a desktop computer. The terminal 1500 may also be called a user device, a mobile terminal, a laptop terminal, a desktop terminal, or other names.

[0171] Typically, the terminal 1500 includes one or more processors 1501 and a memory 1502 .

[0172] The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1501 may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1501 may include a main processor, also called a CPU (Central Processing Unit), which is a processor that processes data in a wake-up state, and a co-processor, which is a low-power processor that processes data in a standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 1501 may further include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

[0173] The memory 1502 may include one or more computer readable storage media, which may be non-transitory. The memory 1502 may further include non-volatile memory, such as high speed random memory, and one or more magnetic storage devices, flash memory, etc. In some embodiments, the non-transitory computer readable storage medium in the memory 1502 stores at least one computer readable instruction that, when executed by the one or more processors 1501, performs a method of image processing according to an embodiment of the method of the present application.

[0174] In some embodiments, the terminal 1500 may further include a peripheral device interface 1503 and at least one peripheral device. The one or more processors 1501, the memory 1502, and the peripheral device interface 1503 may be connected to each other by a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1503 by a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, a positioning assembly 1508, and a power supply 1509.

[0175] The peripheral device interface 1503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the one or more processors 1501 and memory 1502 .

[0176] The radio frequency circuitry 1504 transmits and receives radio frequency (RF) signals, also called electromagnetic signals, and communicates with communication networks and other communication devices via the electromagnetic signals.

[0177] The display screen 1505 displays a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. If the display screen 1505 is a touch display screen, the display screen 1505 also has a function of collecting touch signals on or above the surface of the display screen 1505. The touch signals may be input as control signals to the one or more processors 1501 for processing. In this case, the display screen 1505 may further provide virtual buttons and / or virtual keyboards, also called soft buttons and / or soft keyboards. In some embodiments, the display screen 1505 is one and may be installed on the front panel of the terminal 1500, and in other embodiments, the display screen 1505 is at least two and may be installed on different surfaces of the terminal 1500 or designed to be folded, and in other embodiments, the display screen 1505 is a flexible display screen and may be installed on a curved or folded surface of the terminal 1500. In addition, the display screen 1505 may also be configured to have a non-rectangular irregular shape, i.e., an irregularly shaped display screen. The display screen 1505 may be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and the like.

[0178] The camera assembly 1506 collects images or video.

[0179] The audio circuitry 1507 may include a microphone and a speaker. The microphone collects sound waves from the user and the environment and converts the sound waves into electrical signals to input to one or more processors 1501 or to the radio frequency circuitry 1504 for voice communication. The speaker converts electrical signals from one or more processors 1501 or the radio frequency circuitry 1504 into sound waves.

[0180] A positioning assembly 1508 determines the current geographic location of the terminal 1500 and provides navigation or Location Based Services (LBS).

[0181] A power supply 1509 provides power to each assembly within terminal 1500 .

[0182] In some embodiments, the terminal further includes one or more sensors 1510, including, but not limited to, an acceleration sensor 1511, a gyro sensor 1512, a pressure sensor 1513, a fingerprint sensor 1514, an optical sensor 1515, and a proximity sensor 1516.

[0183] The acceleration sensor 1511 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1500 .

[0184] The gyro sensor 1512 can detect the orientation and rotation angle of the body of the terminal 1500, and in cooperation with the acceleration sensor 1511 can collect the user's 3D movements relative to the terminal 1500.

[0185] The pressure sensor 1513 may be located on a side frame of the terminal 1500 and / or underneath the display screen 1505 .

[0186] The fingerprint sensor 1514 collects a fingerprint of the user, and the one or more processors 1501 identify the user's identity from the fingerprint collected by the fingerprint sensor 1514, or the fingerprint sensor 1514 identifies the user's identity from the fingerprint collected.

[0187] The optical sensor 1515 collects the ambient light intensity. In one embodiment, the one or more processors 1501 can control the display brightness of the display screen 1505 based on the ambient light intensity collected by the optical sensor 1515.

[0188] The proximity sensor 1516, also called a distance sensor, is typically mounted on the front panel of the terminal 1500. The proximity sensor 1516 collects the distance between the user and the front of the terminal 1500.

[0189] The structure shown in FIG. 15 does not constitute a limitation on terminal 1500, and one skilled in the art will recognize that it may include more or fewer components than those shown, or may combine certain components, or may use a different arrangement of components.

[0190] The embodiments of the present application further provide one or more computer-readable storage media applied to a computing device, the computer-readable storage media storing at least one computer-readable instruction that is loaded and executed by one or more processors to realize the operations performed by the computing device in the image processing method according to the embodiments.

[0191] An embodiment of the present application further provides a computer-readable instruction product or computer-readable instructions, the computer-readable instructions including computer-readable instruction codes stored in a computer-readable storage medium, one or more processors of a computing device read the computer-readable instruction codes from the computer-readable storage medium, and when the one or more processors execute the computer-readable instruction codes, the computing device performs the method of image processing according to the various alternative implementations described above.

[0192] Those skilled in the art can understand that all or part of the steps for realizing the above embodiments may be completed by hardware, or may be completed by instructing relevant hardware by a program, and the program may be stored in a computer-readable storage medium, and the above storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0193] The above description is merely a selectable embodiment of the present application, and does not limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included within the protection scope of the present application. [Explanation of symbols]

[0194] 1401 Image Segmentation Module 1402 Trimap Generation Module 1403 Image Matting Module 1501 Processor 1502 Memory 1503 Peripheral Device Interface 1504 Radio Frequency Circuits 1505 Display Screen 1506 Camera Assembly 1507 Audio Circuit 1508 Positioning Assembly 1509 Power supply 1510 Sensor 1511 Accelerometer 1512 Gyro Sensor 1513 Pressure Sensor 1514 Fingerprint Sensor 1515 Optical Sensor 1516 Proximity Sensor

Claims

1. 1. A method of image processing implemented by a computing device, comprising: performing semantic image segmentation of an original image to obtain a first image, a second image, and a third image, wherein a foreground region in the first image is a region in the original image where a target object is located, the second image is a segmented image of a first target region of the target object, the third image is a segmented image of a second target region of the target object, sub-regions of the foreground region include the first target region and the second target region, the first target region and the second target region are regions of the target object that require refined image matting, and a refinement requirement for image matting of the second target region is lower than a refinement requirement for image matting of the first target region; generating a target trimap based on the first image, the second image, and the third image, the target trimap including the foreground region and a line segment drawing region, the line segment drawing region being obtained by drawing a line segment on a contour line of the foreground region, and different sub-regions of the foreground region corresponding to different line widths; performing an image matting process on the target object in the original image based on the target trimap to obtain a target image including the target object.

2. the first target region is a hair region, the second target region is a face region, and the foreground region further includes a body region of the target object; 2. The method of claim 1, wherein in the target trimap, a line width corresponding to the hair region is greater than a line width corresponding to the body region, which is greater than a line width corresponding to the face region.

3. the first target area is a hair area and the second target area is a face area; The step of generating a target trimap based on the first image, the second image, and the third image includes: a step of generating a first trimap based on the first image and the second image, the first trimap including the foreground region, a first line segment drawing subregion and a second line segment drawing subregion, the first line segment drawing subregion covering a contour of the hair region adjacent to a background region side in the first image, the second line segment drawing subregion covering a contour of a non-hair region that is a region other than the hair region in the foreground region, the first line segment drawing subregion being drawn with a first line width and the second line segment drawing subregion being drawn with a second line width, the first line width being greater than the second line width; generating the target trimap based on the third image and the first trimap.

4. The step of generating a first trimap based on the first image and the second image includes: obtaining a complete contour of the foreground region in the first image; drawing a line segment on a complete contour of the foreground region based on the second line width to obtain a second trimap, the second trimap including the foreground region and a third line segment drawing sub-region, the third line segment drawing sub-region covering a complete contour of the foreground region; obtaining a complete contour of the hair region in the second image; drawing a line segment on a complete contour of the hair region based on the first line width to obtain a third trimap, the third trimap including the hair region and a fourth line segment drawing sub-region, the fourth line segment drawing sub-region covering a complete contour of the hair region; 4. The method of claim 3, further comprising: performing a merging process on the second trimap and the third trimap to obtain the first trimap.

5. 5. The method according to claim 3 or 4, characterized in that the first line width is M times the second line width (M>1).

6. The step of performing a merging process on the second trimap and the third trimap to obtain the first trimap includes: obtaining a first discrimination value for each pixel point in the second trimap, the first discrimination value being used to discriminate a color of the pixel point in the second trimap; obtaining a second discrimination value for each pixel point in the third trimap, the second discrimination value being used to discriminate a color of the pixel point in the third trimap; and generating a first trimap based on a magnitude relationship between the first and second discrimination values.

7. generating the first trimap based on a magnitude relationship between the first and second discrimination values, comparing a first discrimination value of a pixel point at a given location of the second trimap with a second discrimination value of a pixel point at the same location of the third trimap; 7. The method of claim 6, further comprising: determining a maximum of the first and second discrimination values ​​as a third discrimination value for a pixel point at the same location in the first trimap, the third discrimination value being used to identify a color of the pixel point in the first trimap.

8. The step of generating the target trimap based on the third image and the first trimap includes: determining a target overlap region of the first trimap based on the face region in the third image, the target overlap region being an overlap region between the face region and the second line segment drawing sub-region; 4. The method of claim 3, further comprising the steps of: assigning target discrimination values ​​to pixel points of the target overlap region to generate the target trimap, the target discrimination values ​​being used to discriminate colors of pixel points in the facial region.

9. The step of performing an image matting process on the target object in the original image based on the target trimap to obtain a target image including the target object includes: obtaining a transparency for each pixel point in the source image based on the target trimap, the transparency being used to indicate a probability that the pixel point belongs to the target object; and performing an image matting process on the original image based on the transparency to obtain the target image.

10. The step of obtaining the transparency of each pixel point in the original image based on the target trimap includes: inputting the target trimap and the original image into an image matting model, and using the image matting model to calculate the probability that each pixel point in the original image belongs to the target image based on the target trimap and the original image, and outputting a transparency; and obtaining the transparency output from the image matting model.

11. an image segmentation module that performs semantic image segmentation on an original image to obtain a first image, a second image, and a third image, wherein a foreground region in the first image is a region in the original image where a target object is located, the second image is a segmented image of a first target region of the target object, the third image is a segmented image of a second target region of the target object, sub-regions of the foreground region include the first target region and the second target region, the first target region and the second target region are regions of the target object that require refined image matting, and a refinement requirement for the image matting of the second target region is lower than a refinement requirement for the image matting of the first target region; a trimap generation module for generating a target trimap based on the first image, the second image, and the third image, the target trimap including the foreground region and a line segment drawing region, the line segment drawing region being obtained by drawing a line segment on a contour line of the foreground region, and different sub-regions of the foreground region corresponding to different line widths; and an image matting module for performing an image matting process on the target object in the original image based on the target trimap to obtain a target image including the target object.

12. the first target region is a hair region, the second target region is a face region, and the foreground region further includes a body region of the target object; 12. The apparatus of claim 11, wherein in the target trimap, a line width corresponding to the hair region is greater than a line width corresponding to the body region, which is greater than a line width corresponding to the face region.

13. the first target area is a hair area and the second target area is a face area; The trimap generation module includes: a first generation unit that generates a first trimap based on the first image and the second image, the first trimap including the foreground region, a first line segment drawing subregion and a second line segment drawing subregion, the first line segment drawing subregion covering a contour of the hair region adjacent to a background region side, the second line segment drawing subregion covering a contour of another region in the foreground region other than the hair region, a first line width being larger than a second line width, the first line width being used to draw the first line segment drawing subregion, and the second line width being used to draw the second line segment drawing subregion; and a second generation unit for generating the target trimap based on the third image and the first trimap.

14. The first generation unit is obtaining a complete contour of the foreground region in the first image; Draw a line segment on the complete contour of the foreground region based on the second line width to obtain a second trimap including the foreground region and a third line segment drawing sub-region covering the complete contour of the foreground region; obtaining a complete contour of the hair region in the second image; Draw a line segment on a complete contour of the hair region based on the first line width to obtain a third trimap including the hair region and a fourth line segment drawing sub-region covering the complete contour of the hair region; The apparatus of claim 13, further comprising: performing a merging process on the second trimap and the third trimap to obtain the first trimap.

15. A computing device including one or more processors and a memory, A computing device, characterized in that the memory stores at least one computer readable instruction, which is loaded by the one or more processors to perform the method of image processing according to any one of claims 1 to 10.

16. A computer program comprising computer readable instructions which, when executed by one or more processors, implements the method of image processing according to any one of claims 1 to 10.

Images