Image processing method, device, apparatus and storage medium

Abstract

The application provides an image processing method, device and equipment and a storage medium, and belongs to the technical field of artificial intelligence. When performing matting on an original image, a plurality of segmentation images containing different regions are obtained by using semantic segmentation, and further, different width lines are drawn on the contour line of the foreground region according to the segmentation images to obtain a target ternary graph. Finally, the target image is generated based on the target ternary graph. For the target ternary graph, different width lines are drawn on the contour line of the foreground region, so that targeted matting of different regions can be realized. For the region that needs to be finely matted, the matting precision of the region can be improved, and the matting precision of other regions can also be ensured, so that a matting image with fine and natural effect is finally obtained. In addition, the above matting process is fully automated, which greatly improves the matting efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an image processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of computer technology, image processing is becoming more and more common. Among them, image matting is a very widely used image processing technology, which specifically refers to separating the foreground region in the image from the background region in the image.

[0003] In the related art, the method of segmentation is usually used to realize image matting, that is, each pixel point in the image is classified to obtain block-shaped segmentation results of different categories, so as to obtain the foreground region in the image, such as a portrait region or a building region.

[0004] However, the above method gives a fixed classification for each pixel point, which can easily cause the edge of the foreground region to be rough, resulting in poor image matting effect. SUMMARY

[0005] The embodiments of the present application provide an image processing method, device, equipment and storage medium, which can obtain an image with fine and natural image matting effect and greatly improve the image matting efficiency. The technical scheme is as follows:

[0006] On the one hand, an image processing method is provided, which comprises:

[0007] performing image semantic segmentation on an original image to obtain a first image, a second image and a third image, the first image comprising a foreground region in which a target object in the original image is located, the second image comprising a hair region of the target object, and the third image comprising a face region of the target object;

[0008] generating a target ternary graph based on the first image, the second image and the third image, the target ternary graph comprising the foreground region and a line drawing region, the line drawing region being obtained by drawing lines on the contour line of the foreground region, and different sub-regions of the foreground region corresponding to different line widths;

[0009] performing image matting processing on the original image based on the target ternary graph to obtain a target image comprising the target object.

[0010] In an optional implementation manner, the image semantic segmentation on the original image to obtain the first image, the second image and the third image comprises:

[0011] obtaining the original image;

[0012] inputting the original image into an image segmentation model, where the image segmentation model is configured to calculate semantic categories of each pixel in the original image based on the inputted original image, to output at least one image of the original image;

[0013] obtaining the first image, the second image and the third image output by the image segmentation model.

[0014] In an alternative implementation, the transparency of each pixel in the original image is obtained based on the target ternary graph, including:

[0015] inputting the target ternary graph and the original image into a matting model, where the matting model is configured to calculate a probability of each pixel in the original image belonging to the target image based on the inputted target ternary graph and original image, to output the transparency;

[0016] obtaining the transparency output by the matting model.

[0017] In another aspect, an image processing apparatus is provided, which includes:

[0018] an image segmentation module configured to perform image semantic segmentation on an original image to obtain a first image, a second image and a third image, where the first image includes a foreground region in which a target object in the original image is located, the second image includes a hair region of the target object, and the third image includes a face region of the target object;

[0019] a ternary graph generation module configured to generate a target ternary graph based on the first image, the second image and the third image, where the target ternary graph includes the foreground region and a line drawing region, the line drawing region is obtained by drawing lines on an outline of the foreground region, and different sub-regions of the foreground region correspond to different line widths;

[0020] a matting module configured to perform matting processing on the original image based on the target ternary graph to obtain a target image including the target object.

[0021] In an alternative implementation, the ternary graph generation module includes:

[0022] a first generation unit configured to generate a first ternary graph based on the first image and the second image, where the first ternary graph includes the foreground region, a first line drawing sub-region and a second line drawing sub-region;

[0023] The first line drawing sub-region covers the contour line near the background region on one side of the hair region, and the second line drawing sub-region covers the contour line of other regions, which are regions in the foreground region except the hair region; a first line width is greater than a second line width, the first line width is used to draw the first line drawing sub-region, and the second line width is used to draw the second line drawing sub-region.

[0024] The second generation unit is configured to generate the target ternary image based on the third image and the first ternary image.

[0025] In an optional implementation, the first generation unit is configured to:

[0026] In the first image, a complete contour line of the foreground region is obtained;

[0027] In the foreground region, a line is drawn on the complete contour line according to the second line width, to obtain a second ternary image; the second ternary image includes the foreground region and a third line drawing sub-region, and the third line drawing sub-region covers the complete contour line of the foreground region.

[0028] In the second image, a complete contour line of the hair region is obtained;

[0029] In the hair region, a line is drawn on the complete contour line according to the first line width, to obtain a third ternary image; the third ternary image includes the hair region and a fourth line drawing sub-region, and the fourth line drawing sub-region covers the complete contour line of the hair region.

[0030] The second ternary image and the third ternary image are merged to obtain the first ternary image.

[0031] In an optional implementation, the first line width is M times of the second line width, and M is greater than 1.

[0032] In an optional implementation, the first generation unit is further configured to:

[0033] A first identification value of each pixel point in the second ternary image is obtained, and the first identification value is used to identify the color of the pixel point in the second ternary image.

[0034] A second identification value of each pixel point in the third ternary image is obtained, and the second identification value is used to identify the color of the pixel point in the third ternary image.

[0035] The first ternary image is generated based on the size relationship between the first identification value and the second identification value.

[0036] In an optional implementation, the first generation unit is further configured to:

[0037] comparing the first identification value of the pixel point at any position in the second ternary graph with the second identification value of the pixel point at the same position in the third ternary graph;

[0038] taking the maximum of the first identification value and the second identification value as a third identification value of the pixel point at the same position in the first ternary graph, the third identification value being used to identify the color of the pixel point in the first ternary graph.

[0039] In an optional implementation, the second generation unit is configured to:

[0040] determine a target overlapping region of the first ternary graph based on the face region in the third image, the target overlapping region being an overlapping region of the face region and the second line drawing sub-region;

[0041] value the pixel points in the target overlapping region with a target identification value to generate the target ternary graph, the target identification value being used to identify the color of the pixel point in the face region.

[0042] In an optional implementation, the matting module is configured to:

[0043] obtain the transparency of each pixel point in the original image based on the target ternary graph, the transparency being used to represent the probability that the pixel point belongs to the target object;

[0044] perform matting processing on the original image based on the transparency to obtain the target image.

[0045] In an optional implementation, the image segmentation module is further configured to:

[0046] obtain the original image;

[0047] input the original image into an image segmentation model, wherein the image segmentation model is configured to calculate the semantic category of each pixel point in the original image according to the input original image, to output at least one image of the original image;

[0048] obtain the first image, the second image, and the third image output by the image segmentation model.

[0049] In an optional implementation, the matting module is further configured to:

[0050] input the target ternary graph and the original image into a matting model, the matting model being configured to calculate the probability that each pixel point in the original image belongs to the target image according to the input target ternary graph and original image, to output the transparency;

[0051] obtain the transparency output by the matting model.

[0052] In another aspect, a computer device is provided, which includes a processor and a memory for storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the operations performed in the image processing method in the embodiments of the present application.

[0053] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, the at least one computer program being loaded and executed by a processor to implement the operations performed in the image processing method in the embodiments of the present application.

[0054] In another aspect, a computer program product or computer program is provided, which includes computer program code stored in a computer readable storage medium. A processor of a computer device reads the computer program code from the computer readable storage medium, and the processor executes the computer program code, so that the computer device performs the image processing method provided in the various optional implementation manners described above.

[0055] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0056] In the embodiments of the present application, when performing matting on an original image, first, a semantic segmentation method is used to obtain a plurality of segmentation images containing different regions, and further, according to the segmentation images, lines of different widths are drawn on the contour lines of the foreground region to obtain a target ternary graph, and finally, a target image is generated based on the target ternary graph. For the target ternary graph described above, since lines of different widths are drawn on the contour lines of the foreground region, targeted matting of different regions can be achieved, the matting accuracy of the region that needs to be finely matted can be improved, and at the same time, the matting accuracy of other regions can be ensured, so that a matting image with fine and natural effect is finally obtained. In addition, the above matting process is fully automated, which greatly improves the matting efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 is a structural schematic diagram of a high-resolution network according to the embodiments of the present application;

[0059] Figure 2is a structural schematic diagram of an object context feature representation according to an embodiment of the present application;

[0060] Figure 3 is an implementation environment schematic diagram of an image processing method according to an embodiment of the present application;

[0061] Figure 4 is a flowchart of an image processing method according to an embodiment of the present application;

[0062] Figure 5 is a flowchart of another image processing method according to an embodiment of the present application;

[0063] Figure 6 is a schematic diagram of an image semantic segmentation result according to an embodiment of the present application;

[0064] Figure 7 is a schematic diagram of a first ternary graph according to an embodiment of the present application;

[0065] Figure 8 is a schematic diagram of a second ternary graph according to an embodiment of the present application;

[0066] Figure 9 is a schematic diagram of a third ternary graph according to an embodiment of the present application;

[0067] Figure 10 is a schematic diagram of a target ternary graph according to an embodiment of the present application;

[0068] Figure 11 is a schematic diagram of a matting model according to an embodiment of the present application;

[0069] Figure 12 is a schematic diagram of a target image according to an embodiment of the present application;

[0070] Figure 13 is a schematic diagram of an image processing method according to an embodiment of the present application;

[0071] Figure 14 is a structural schematic diagram of an image processing apparatus according to an embodiment of the present application;

[0072] Figure 15 is a structural schematic diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0074] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description herein relates to the drawings, in which the same numbers represent the same or similar elements throughout several figures. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present disclosure. Rather, they are merely examples that are consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] The terms "first", "second", and so on are used herein to distinguish between the same or similar items or elements having substantially the same function, and it should be understood that there is no logical or chronological dependency between "first", "second", and "n", nor is there any limitation on the number and execution order. It should also be understood that although the following description uses the terms first, second, and so on to describe various elements, these elements should not be limited by the terms.

[0076] These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first image can be referred to as a second image, and similarly, a second image can also be referred to as a first image. The first image and the second image can both be images, and in some cases, can be separate and distinct images.

[0077] Among them, at least one refers to one or more, for example, at least one image can be one image, two images, three images, etc. any integer greater than or equal to one image. And multiple refers to two or more, for example, multiple images can be two images, three images, etc. any integer greater than or equal to two images.

[0078] The following briefly introduces the technology that can be used in the image processing scheme provided by the embodiments of the present application.

[0079] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0080] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields.

[0081] Computer vision (CV) Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, object-contextual representations (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies, as well as common face recognition, fingerprint recognition and other biometric identification technologies.

[0082] High Resolution Network (HRNET) is a computing model for obtaining image feature information, which can maintain high resolution representation throughout the operation process. HRNET starts with a set of high-resolution convolutions, then gradually adds low-resolution convolution branches, and connects them in parallel. Reference Figure 1 , Figure 1 is a structural diagram of a high-resolution network provided by the present application, as Figure 1 shown, the network connects feature maps of different resolutions in parallel, each resolution is connected in one way, and information is constantly exchanged between parallel operation combinations throughout the process.

[0083] OCR is a computing model for representing the semantic categories of pixels in an image. After obtaining the features of the image through HRNET, the semantic categories of each pixel in the image are calculated through the OCR method.

[0084] Reference Figure 2 , Figure 2 is a structural diagram of an object-contextual representation provided by the present application, as Figure 2As shown: first, a rough semantic segmentation result, i.e. Soft Object Regions, is obtained by the middle layer of the backbone network; second, K groups of vectors, K>1, i.e. Object Region Representations, are calculated by the pixel representation and the soft object regions output by the deep layer of the backbone network, wherein each vector corresponds to a feature representation of a semantic category; third, a relationship matrix between the pixel representation and the object region representation is calculated; fourth, each object region representation is weighted and summed to obtain the context feature representation of the object, i.e. OCR, according to the numerical value of the pixel representation and the object region representation of each pixel in the relationship matrix; finally, an Augmented Representation is obtained based on the OCR and the pixel representation as context information enhanced feature representation, and the enhanced feature representation can be used to predict the semantic category of each pixel.

[0085] The following briefly introduces the key terms or abbreviations that may be used in the image processing scheme provided by the embodiments of the present application.

[0086] Semantic Segmentation: For an input image, based on the semantic understanding of each pixel, the pixels of the same semantics are segmented into the same part or region, and the process of obtaining a plurality of different semantic regions.

[0087] Foreground: The subject in the image, such as the portrait in a portrait shot.

[0088] Background: The environment in which the subject is located in the image, such as the scenery, road, building, etc. in which the person is located in a portrait shot.

[0089] Image Matting: An image processing technique for separating the foreground of an image from the background.

[0090] Trimap: An image containing three labels of foreground, background and foreground-background mixed area, which is usually input together with the original image as a matting model. It should be noted that in the following embodiments, the foreground-background mixed area is also referred to as the line drawing area.

[0091] Label value: a numerical value used to identify the color of a pixel in an image. For example, a pixel with a label value of 255 indicates that the RGB (Red-Green-Blue) color value of the pixel is (255, 255, 255), representing white color. For another example, a pixel with a label value of 0 indicates that the RGB color value of the pixel is (0, 0, 0), representing black color. For yet another example, a pixel with a label value of 128 indicates that the RGB color value of the pixel is (128, 128, 128), representing gray color.

[0092] OpenCV (Open Source Computer Vision Library): a cross-platform computer vision and machine learning software library that can run on various operating systems. OpenCV can be used to develop real-time image processing, computer vision, and pattern recognition programs.

[0093] findContours: a function in OpenCV for detecting contours in an image.

[0094] drawContours: a function in OpenCV for drawing contours in an image.

[0095] Matting model: a computational model used to calculate the probability of each pixel in an original image belonging to the foreground based on the original image and a trimap. For example, the Matting model includes IndexNet model, GCAMatting model, and ContextNet model, etc.

[0096] The implementation environment of the image processing method provided by the embodiments of the present application is introduced as follows.

[0097] Figure 3 is a schematic diagram of the implementation environment of the image processing method provided by the embodiments of the present application. The implementation environment includes a terminal 301 and a server 302.

[0098] The terminal 301 and the server 302 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. Optionally, the terminal 301 is a smartphone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal 301 can install and run an application program. Optionally, the application program is a social application program, an image processing application program, or a shooting application program, etc. Illustratively, the terminal 301 is a terminal used by a user, and the social application program is run in the terminal 301, so that the user can extract the portrait in the picture through the social application program.

[0099] The server 302 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The server 302 is configured to provide background services for the application program running on the terminal 301.

[0100] Optionally, in the process of image processing, the server 302 undertakes the main computing work, and the terminal 301 undertakes the secondary computing work; or the server 302 undertakes the secondary computing work, and the terminal 301 undertakes the main computing work; or the server 302 or the terminal 301 can undertake the computing work independently.

[0101] Optionally, the terminal 301 refers to one of a plurality of terminals, and the embodiment is only exemplified by the terminal 301. Those skilled in the art can know that the number of the terminal 301 can be more. For example, the number of the terminal 301 is dozens or hundreds, or more, and the implementation environment of the image processing method further includes other terminals. The number and type of the terminal are not limited in the embodiment.

[0102] Optionally, the wireless network or the wired network uses standard communication technology and / or protocol. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of virtual private networks. In some embodiments, technologies and / or formats including HTML, Extensible Markup Language (XML) and the like are used to represent data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) and the like. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies.

[0103] In the embodiments of the present application, an image processing method is provided, which can meet the needs of users for portrait extraction in various social entertainment scenarios, and can automatically extract the human body and hair from the portrait picture input by the user, with fine and natural cutout effect. The image processing method provided in the embodiments of the present application is described in detail below by taking the application of the image processing method to portrait extraction as an example.

[0104] Figure 4 is a flowchart of an image processing method provided in the embodiments of the present application, as shown in Figure 4 The method includes the following steps:

[0105] 401. Perform image semantic segmentation on an original image to obtain a first image, a second image and a third image, wherein the first image includes a foreground region of a target object in the original image, the second image includes a hair region of the target object, and the third image includes a face region of the target object.

[0106] In the embodiments of the present application, the original image refers to an image input by a user and requiring image extraction. The target object refers to a human portrait in the original image.

[0107] It should be noted that the target object is taken as a human portrait for description in the present application, and in other embodiments, the target object is an animal image or a plant image in the original image, and the like. Correspondingly, when the target object is an animal image, the hair region in this step can be replaced by a fur region or other region requiring fine cutout, and the face region can be replaced by a head region or other region not requiring fine cutout; when the target object is a plant image, the hair region in this step can be replaced by a leaf region, and the face region can be replaced by a branch region, without limitation in the embodiments of the present application.

[0108] In addition, it should be noted that the target object can be a real human portrait, a cartoon character portrait or an animation character portrait, without limitation in the embodiments of the present application.

[0109] 402. Based on the first image, the second image and the third image, generate a target ternary graph, wherein the target ternary graph includes a foreground region and a line drawing region, the line drawing region is obtained by drawing lines on the contour line of the foreground region, and different sub-regions of the foreground region correspond to different line widths.

[0110] In the embodiments of the present application, the sub-region refers to a part of the foreground region, including the hair region and the face region, and the like.

[0111] 403. Perform cutout processing on the original image based on the target ternary graph to obtain a target image including the target object.

[0112] In the embodiments of the present application, the matting processing refers to a process of separating a target object in an original image from a background region to obtain a target image.

[0113] In the embodiments of the present application, when performing matting on the original image, first, a plurality of segmentation images containing different regions are obtained by using semantic segmentation, and then, according to the segmentation images, lines with different widths are drawn on the contour line of the foreground region to obtain a target ternary graph, and finally, the target image is generated based on the target ternary graph. For the target ternary graph, since lines with different widths are drawn on the contour line of the foreground region, targeted matting of different regions can be achieved, the matting precision of the region that needs to be finely matted can be improved, and the matting precision of other regions can also be ensured, so that a matted image with fine and natural effect is finally obtained. In addition, the above matting process is fully automated, which greatly improves the matting efficiency.

[0114] The above Figure 4 is only the basic flow of the present application, and the scheme provided by the present application will be further described based on a specific implementation manner.

[0115] Figure 5 is a flowchart of another image processing method provided by the embodiments of the present application, as shown in the figure, in the embodiments of the present application, the application in the terminal is taken as an example for description. The method comprises the following steps: Figure 5

[0116] 501, obtaining an original image.

[0117] In the embodiments of the present application, the terminal provides a matting function, the user can perform a matting operation on the terminal, and the terminal obtains the original image in response to the matting operation. Optionally, the original image is a local image stored on the terminal, or the original image is an online image, and the source of the original image is not limited in the embodiments of the present application.

[0118] Optionally, an image processing interface for the original image is displayed on the terminal, the image processing interface comprises a matting option, a clipping option and the like, the user can perform a check operation on the matting option, and the terminal obtains the original image in response to the check operation.

[0119] Optionally, an image processing interface is displayed on the terminal, the image processing interface comprises a matting option, the user can perform a check operation on the matting option, the terminal displays an image selection interface in response to the check operation, the user can select an original image for matting by performing a click operation on an image to be matted, and the terminal obtains the original image in response to the click operation.

[0120] ​It should be noted that the embodiments of the present application do not limit the manner in which the terminal obtains the original image.

[0121] 502, input the original image into the image segmentation model.

[0122] In the embodiments of the present application, the image segmentation model is used to calculate the semantic categories of each pixel point in the original image according to the input original image, so as to output at least one image of the original image.

[0123] Optionally, the image segmentation model is an HRNET-OCR model, which is a calculation model combining an HRNET model and an OCR model. The calculation process of the HRNET-OCR model is as follows: first, the HRNET model is used to extract features of the original image to obtain feature information of the original image; second, the obtained feature information is input into the backbone network of the OCR model; third, based on the OCR model, the semantic categories of each pixel point in the original image are calculated; for example, the semantic categories include hair, nose, eyes, torso, clothes, and buildings, etc.; finally, based on the semantic categories of each pixel point, at least one image of the original image is output. The specific calculation process of the HRNET-OCR model has been described in detail in combination with Figure 1 and Figure 2 Therefore, details are not repeated here.

[0124] It should be noted that in actual applications, at least one image of the original image can be output by adjusting part of the structure of the HRNET-OCR model described above, and the embodiments of the present application do not limit the structure composition of the HRNET-OCR model. In some other embodiments, the image segmentation model described above can also be implemented by other network models, and the embodiments of the present application do not limit the type of image semantic segmentation model.

[0125] 503, obtain the first image, the second image and the third image output by the image segmentation model. The first image includes a foreground region where the target object in the original image is located, the second image includes a hair region of the target object, and the third image includes a face region of the target object.

[0126] In the embodiment of the present application, the image segmentation model is used to classify each pixel point in the original image, and the terminal can obtain three segmented images, i.e., the first image, the second image and the third image in the present step. Each image includes two regions, which are marked by different identification values, for example, taking the first image as an example, the first image includes a foreground region and a background region, wherein the identification value of each pixel point in the foreground region is 255, and the identification value of each pixel point in the background region is 0. It should be noted that in actual application, the developer can flexibly set the identification value according to the needs, and the present application does not limit this.

[0127] Schematically, referring to Figure 6 , Figure 6 is a schematic diagram of an image semantic segmentation result provided by the embodiment of the present application. As shown in Figure 6 , the original image is a portrait of a person, Figure 6 the first image shown in (a) of the figure includes a foreground region 1 and a background region 2, and the foreground region 1 contains all elements of the portrait of the person; Figure 6 the second image shown in (b) of the figure includes a hair region 3 and a background region 4; Figure 6 the third image shown in (c) of the figure includes a face region 5 and a background region 6.

[0128] 504, based on the first image and the second image, a first ternary graph is generated, which includes a foreground region, a first drawing sub-region and a second drawing sub-region.

[0129] In the embodiment of the present application, the first drawing sub-region covers the contour line on the side of the hair region close to the background region, the second drawing sub-region covers the contour line on the other region, and the other region is the region in the foreground region except the hair region. The first line width is greater than the second line width, the first line width is used to draw the first drawing sub-region, and the second line width is used to draw the second drawing sub-region.

[0130] The first ternary graph further includes a background region, and the identification values of the first drawing sub-region and the second drawing sub-region are different from the identification value of the foreground region and also different from the identification value of the background region. For example, the identification value of each pixel point in the foreground region is 255, the identification value of each pixel point in the background region is 0, and the identification value of each pixel point in the first drawing sub-region and the second drawing sub-region is 128.

[0131] It should be noted that in actual application, the developer can flexibly set the identification value of the drawing region according to the needs, and the present application does not limit this.

[0132] Schematically, reference is made to Figure 7 , Figure 7 is a schematic diagram of a first ternary diagram provided by an embodiment of the present application. As shown in the right diagram of FIG. 7, the first ternary diagram includes a foreground region 7, a background region 8, a first drawing sub-region 9 and a second drawing sub-region 10, wherein the first drawing sub-region 9 is drawn according to a first line width, and the second drawing sub-region 10 is drawn according to a second line width. Figure 7

[0133] In addition, it should be noted that in the embodiments of the present application, there are differences in the region ranges between the foreground regions in different images, and there are also differences in the region ranges between the background regions in different images. For example, referring to the background region 2 and the background region 4 in FIG. 6, although there is a significant difference in the region ranges between the two, both are background regions. For another example, referring to the foreground region 1 in FIG. 5 and the foreground region 7 in FIG. 7, although there is a slight difference in the region ranges between the two, both are foreground regions. Figure 6 Figure 6 Figure 7

[0134] The specific implementation of the terminal generating the first ternary diagram in this step will be described in detail below, including the following steps 5041 to 5045:

[0135] 5041. In the first image, the complete contour line of the foreground region is obtained.

[0136] The complete contour line of the foreground region refers to the boundary line between the foreground region and the background region. Based on the obtained first image, the terminal obtains the complete contour line of the foreground region in the first image through a contour detection algorithm. Optionally, the contour detection algorithm can be implemented through a findContours function, which is not limited in the embodiments of the present application.

[0137] 5042. A line is drawn on the complete contour line of the foreground region according to a second line width, to obtain a second ternary diagram.

[0138] The second ternary diagram includes a foreground region and a third drawing sub-region, and the third drawing sub-region is overlaid on the complete contour line of the foreground region. The second line width is calculated according to the size of the original image. Optionally, the second line width can be calculated through the following formula (1):

[0139]

[0140] In the formula, S is the second line width; width and height are the width and height of the original image respectively; and N is a default line size, for example, N can be 17, which is not limited in the embodiments of the present application.

[0141] ​​​​After the terminal obtains the complete contour line of the foreground region, the terminal draws a line on the complete contour line according to the second line width by using a contour drawing algorithm, and the identification value of the line is different from the identification values of the foreground region and the background region. Optionally, the contour drawing algorithm can be implemented by using a drawContours function. For example, taking the identification value of the foreground region as 255 and the identification value of the background region as 0 as an example, the line is drawn on the complete contour line of the foreground region by using the following formula (2):

[0142] cv::drawContours(segResult, contours, -1, Scalar(128, 128, 128), S) (2)

[0143] In the formula, segResult is the first image, contours is the complete contour line of the foreground region detected by using the findContours function, -1 indicates that all contour lines are operated, Scalar is an identification value, Scalar(128, 128, 128) indicates that the color values of R, G, and B channels in the RGB channel are all set to 128, and S is the second line width.

[0144] It should be noted that the above method of drawing a line is to operate on the obtained complete contour line, that is, to cover the complete contour line. For example, the complete contour line obtained by using the findContours function includes pixel points A1 to A10, and the pixel points are operated to draw a line. That is, the third drawing sub-region obtained by drawing a line covers both the foreground region in the first image and the background region in the first image.

[0145] For example, refer to Figure 8 , Figure 8 FIG. 2 is a schematic diagram of a second ternary graph provided by an embodiment of the present application. Figure 8 The left image in FIG. 2 is a first image, Figure 8 The right image in FIG. 2 is a second ternary graph, and the second ternary graph includes a foreground region 11, a background region 12, and a third drawing sub-region 13. The third drawing sub-region 13 is drawn according to a second line width.

[0146] 5043、In the second image, a complete contour line of the hair region is obtained.

[0147] The complete contour line of the hair region is a boundary line between the hair region and the background region. Based on the obtained second image, the terminal obtains the complete contour line of the hair region in the second image by using a contour detection algorithm. Optionally, the contour detection algorithm can be implemented by using a findContours function, and the present application does not limit this.

[0148] 5044、Draw lines on the complete contour line of the hair region according to the first line width, to obtain a third ternary image.

[0149] The third ternary image includes the hair region and a fourth drawing sub-region, and the fourth drawing sub-region is drawn on the complete contour line of the hair region. The first line width is M times of the second line width, where M is greater than 1. For example, the first line width is 3 times of the second line width; that is, when the second line width is S, the first line width is S*3, which is not limited in the embodiments of the present application.

[0150] It should be noted that the way of drawing lines on the complete contour line of the hair region in this step is similar to that in the step 5042, and thus will not be described here again, and only an example is given by the following formula (3):

[0151] cv::drawContours(segResultHair, contours, -1, Scalar(128, 128, 128), S*3) (3)

[0152] In the formula, segResultHair is the second image; contours is the complete contour line of the hair region detected by the findContours function, -1 represents that all contour lines are operated, Scalar is an identification value, Scalar(128, 128, 128) represents that the color values of R, G and B channels in the RGB channel are all set to 128; and S*3 is the first line width.

[0153] For example, referring to Figure 9 , Figure 9 is a schematic diagram of the third ternary image provided by the embodiments of the present application. Figure 9 The left image in FIG. 4 shows the second image, Figure 9 The right image in FIG. 4 shows the third ternary image, which includes the foreground region 14, the background region 15 and the fourth drawing sub-region 16, and the fourth drawing sub-region 16 is drawn according to the first line width.

[0154] 5045、Merge the second ternary image and the third ternary image to obtain a first ternary image.

[0155] The merging of the second ternary image and the third ternary image means that the maximum identification value of the same position in the two ternary images is taken as the identification value of the corresponding position in the first ternary image.

[0156] The specific implementation of this step will be described in detail below, including the following steps A to C:

[0157] Step A: obtaining a first identification value of each pixel point in the second ternary graph, the first identification value being used to identify the color of the pixel point in the second ternary graph.

[0158] Step B: obtaining a second identification value of each pixel point in the third ternary graph, the second identification value being used to identify the color of the pixel point in the third ternary graph.

[0159] Step C: generating the first ternary graph based on the size relationship between the first identification value and the second identification value.

[0160] The implementation of Step C includes: comparing the first identification value of the pixel point at any position in the second ternary graph with the second identification value of the pixel point at the same position in the third ternary graph; taking the maximum of the first identification value and the second identification value as the third identification value of the pixel point at the same position in the first ternary graph, the third identification value being used to identify the color of the pixel point in the first ternary graph.

[0161] Illustratively, continuing to refer to Figure 7 , after obtaining the second ternary graph and the third ternary graph, the terminal implements the merging processing of the two ternary graphs by the following formula (4) to obtain the first ternary graph. The formula (4) is as follows:

[0162] Pixel result =Pixel leftUp >Pixel leftDown ?Pixel leftUp :Pixel leftDown (4)

[0163] In the formula, Pixel result is the first ternary graph, i.e., the right graph in Figure 7 ; Pixel leftUp is the second ternary graph, i.e., the upper left graph in Figure 7 ; and Pixel leftDown is the third ternary graph, i.e., the lower left graph in Figure 7 .

[0164] As shown in the right graph in Figure 7 , the first line drawing sub-region 9 corresponding to the hair region in the first ternary graph generated by the above-mentioned Step 5045 is larger than the second line drawing sub-region 10 corresponding to other regions, and in this way of drawing different line drawing regions according to different line widths, the fine degree of the matting can be improved for the matting region such as the hair region which is relatively complex, so as to improve the matting effect of this part of region.

[0165] 505. Generating a target ternary graph based on the third image and the first ternary graph.

[0166] In the embodiments of the present application, the target ternary graph includes a foreground region and a line drawing region, the line drawing region being obtained by drawing lines on the contour line of the foreground region, and different sub-regions of the foreground region correspond to different line widths. Optionally, the foreground region further includes a torso region of the target object, wherein in the target ternary graph, the line width corresponding to the hair region is greater than the line width corresponding to the torso region, and the line width corresponding to the torso region is greater than the line width corresponding to the face region.

[0167] Illustratively, reference is made to Figure 10 , Figure 10 is a schematic diagram of a target ternary graph provided by the embodiments of the present application. Figure 10 The target ternary graph is shown in the upper right graph in FIG. 17. The graph includes a foreground region 17, a background region 18, and a line drawing region 19. In the line drawing region 19, the line width corresponding to the hair region is greater than the line width corresponding to the torso region, and the line width corresponding to the torso region is greater than the line width corresponding to the face region. Illustratively, the relationship of the line widths in the line drawing region 19 can continue to refer to the lower right graph in FIG. 17, which includes line drawing regions 19a, 19b, and 19c. In the graph, 19a represents the line width corresponding to the hair region, 19b represents the line width corresponding to the face region, and 19c represents the line width corresponding to the torso region. As shown in the graph, the line width 19a corresponding to the hair region is greater than the line width 19c corresponding to the torso region, and the line width 19c corresponding to the torso region is greater than the line width 19b corresponding to the face region. Figure 10

[0168] The specific implementation of the terminal generating the target ternary graph in this step is described in detail below, including the following steps 5051 to 5052:

[0169] 5051. Based on the face region in the third image, determine a target overlapping region of the first ternary graph, the target overlapping region being an overlapping region of the face region and the second line drawing sub-region.

[0170] Wherein, after obtaining the first ternary graph, the terminal determines the target overlapping region in the first ternary graph based on the pixel position of the face region in the third image. Illustratively, reference is made to the lower right graph in FIG. 17, which includes a target overlapping region 20, which is the overlapping region between the face region in the third image and the second line drawing sub-region 19 in the first ternary graph. Figure 10 As shown in the lower right graph in FIG. 17, the target overlapping region 20 is the overlapping region between the face region in the third image and the second line drawing sub-region 19 in the first ternary graph. Figure 10

[0171] 5052. Assign a target identification value to the pixel points of the target overlapping region to generate a target ternary graph, the target identification value being used to identify the color of the pixel points in the face region.

[0172] ​​In the first ternary graph, the identification value of the target overlapping region is the identification value of the second line drawing sub-region. In this step, the identification value of the target overlapping region in the first ternary graph is changed, and the pixel points in this region are assigned with the target identification value to generate a target ternary graph. For example, taking the identification value of the face region as 255 and the identification value of the second line drawing sub-region as 128 as an example, the identification value of the target overlapping region in the first ternary graph is originally 128. After this step 5052, the pixel points in the target overlapping region are re-assigned with the identification value of 255 to obtain the target ternary graph.

[0173] Optionally, the above step 5051 and step 5052 can be implemented by the following formula (5):

[0174] Pixel = Pixel ∈ φ{Face}? 255 : Pixel trimp (5)

[0175] In the formula, φ{Face} represents the face region; 255 represents the target identification value, Pixel trimp is the target ternary graph. By the formula (5), the pixel points belonging to the face region in the first ternary graph are assigned with the target identification value, so that the face region does not participate in the process of calculating the transparency by using the matting model in the following steps 506 to 507.

[0176] After the above steps 501 to 505, the terminal automatically generates the target ternary graph after obtaining the original image, and in the target ternary graph, the line widths corresponding to different sub-regions of the foreground region are different.

[0177] It should be noted that in actual application, the marking of the foreground and background mixed region in the ternary graph will directly affect the fineness of the matting effect, and if the foreground region is marked as the foreground and background mixed region in the ternary graph, the matting result will be inaccurate.

[0178] In the embodiment of the present application, the line drawing region in the target ternary graph is the foreground and background mixed region. In the process of automatically generating the target ternary graph, the terminal draws lines on the hair region and other regions except the hair region according to different line widths, which ensures the matting range of the hair region and other complex regions and improves the matting precision of these regions. At the same time, the pixel points belonging to the face region are assigned with the target identification value which is the same as that of the foreground region, which considers the protection of the key region in the portrait of the person and avoids the loss of details in matting.

[0179] 506, inputting the target ternary graph and the original image into the matting model.

[0180] In the embodiments of the present application, the matting model is used to calculate the probability of each pixel point in the original image belonging to the target image according to the input target tri-graph and the original image, so as to output the transparency. Optionally, the transparency is calculated by the following formula (6):

[0181] I = a * F + (1-a) * B (6)

[0182] In the formula, I represents the original image; F represents the foreground, that is, the region including all elements of the target object; B represents the background; and a is the transparency, which is used to represent the proportion of the foreground color in the original image. Formula (6) shows that the original image is composed of the foreground and the background superimposed according to a certain transparency.

[0183] Optionally, the above matting model can be an IndexNet matting model, or the above matting model can also be a GCAMatting matting model, or the above matting model can also be a ContextNet model, and the like. The specific type of the matting model is not limited in the present application.

[0184] The following will take the IndexNet matting model as an example to illustrate this step. Illustratively, refer to Figure 11 , Figure 11 Fig. 1 is a schematic diagram of a matting model provided by the embodiments of the present application, as shown in the figure, the target tri-graph and the original image are taken as inputs to obtain a rough Alpha (that is, a) graph and an Alpha prediction loss; at the same time, the rough foreground and the background are synthesized and compared with the original image to obtain a combination loss of the image, and finally a convolution layer is used to optimize to obtain a fine Alpha graph, and output the fine result of the matting, that is, the Alpha value of each pixel point. Figure 11

[0185] 507、Obtain the transparency output by the matting model, which is used to represent the probability of the pixel point belonging to the target object.

[0186] 508, perform matting processing on the original image based on the transparency to obtain a target image including a target object.

[0187] In the embodiments of the present application, the matting processing in the step 508 is a process of separating the target object in the original image from the background based on the transparency of each pixel point to obtain a target image. Illustratively, refer to Figure 12 , Figure 12 Fig. 2 is a schematic diagram of a target image provided by the embodiments of the present application. Figure 12 The left image in Fig. 2 is an original image, Figure 12 ​The target image obtained according to the method is shown in the upper right of FIG. 1, in which the hair tips of the portrait are elegant, the matting is fine, and the face is complete. Figure 12 The target image obtained according to the image segmentation method in the related art is shown in the lower right of FIG. 1, in which the image segmentation is accurate, but the hair tips of the portrait are very rough, and the face has a detail loss.

[0188] In the embodiment of the present application, when performing matting on the original image, first, a plurality of segmentation images containing different regions are obtained by using semantic segmentation, and then, according to the segmentation images, lines with different widths are drawn on the contour line of the foreground region to obtain a target ternary graph, and finally, the target image is generated based on the target ternary graph. For the above-mentioned target ternary graph, since lines with different widths are drawn on the contour line of the foreground region, targeted matting of different regions can be achieved, the matting precision of the region that needs to be finely matted can be improved, and at the same time, the matting precision of other regions can be ensured, so that a finely and naturally matted image is finally obtained. In addition, the above-mentioned matting process is fully automated, which greatly improves the matting efficiency.

[0189] The image processing method provided in the embodiment of the present application will be described in detail below. Figure 13 The image processing method provided in the embodiment of the present application will be described in detail below. Figure 13 As shown in FIG. 1, the following six image processing stages are included:

[0190] First, an original image is obtained, which is an image including a portrait.

[0191] Second, the original image is subjected to image semantic segmentation based on the HRNET-OCR model, and three segmentation images are obtained according to the segmentation results. The first segmentation image includes a foreground region, which contains all elements of the portrait. The second segmentation image includes a hair region. Since the edge line between the human torso and the background is relatively clear, the hair is often seriously blended with the background and needs to be carefully matted. The third segmentation image includes a face region, which can also be understood as a protected region. The face is an important part of the portrait, and if it is mistakenly matted or damaged, it will greatly affect the visual effect and needs to be protected from being matted.

[0192] Third, for the first segmentation image, lines are drawn on the contour line of the foreground region with a basic size to obtain a second ternary graph. For the second segmentation image, lines are drawn on the contour line of the hair region with a basic size of three times to obtain a third ternary graph. The identification value of the drawn lines is 128.

[0193] Fourth, the second ternary graph and the third ternary graph are merged to obtain a merged first ternary graph.

[0194] Fifthly, the face region in the first ternary graph is re-set as a foreground mark, that is, each pixel point of the face region in the first ternary graph is re-assigned as 255, to obtain a target ternary graph.

[0195] Sixthly, based on the target ternary graph, a target image is finally obtained, that is, a portrait of a person in the original image.

[0196] For the above target ternary graph, since lines of different widths are used to draw on the contour line of the foreground region, targeted matting can be achieved for different regions. For regions that need to be finely matted, the matting accuracy of these regions can be improved, while the matting accuracy of other regions can also be ensured, so that a finally obtained matting image is fine and natural. In addition, the above matting process is fully automated, which greatly improves the matting efficiency.

[0197] Illustratively, the application scenarios of the image processing method provided by the embodiments of the present application include but are not limited to:

[0198] Scenario one, expression package scenario

[0199] With the popularity of expression package culture, many applications have added the function of making expression packages, so that users can express their emotions and moods through making expression packages. In some scenarios, users want to cut out the portrait of a person in a picture, and then make the expression package they want by adding stickers, text or backgrounds, etc. on the basis of the portrait of the person.

[0200] For example, a terminal provides an expression package making function of a portrait by an application. A user inputs an original image in which a portrait of a person is to be cut out by performing an operation on the terminal. After obtaining the original image, the terminal automatically cuts out the portrait of the person in the original image by using the image processing method provided by the embodiments of the present application, and displays the portrait of the person on the terminal, so as to provide the user with subsequent image processing operations on the basis of the portrait of the person, and then obtain the expression package the user wants. Illustratively, the process of cutting out the portrait of the person by the terminal includes the following steps 1 to 8:

[0201] 1. The terminal obtains an original image.

[0202] 2. The terminal inputs the original image into an image segmentation model.

[0203] 3. The terminal obtains a first image, a second image and a third image output by the image segmentation model. The first image includes a foreground region in which a portrait of a person in the original image is located, the second image includes a hair region of the portrait of the person, and the third image includes a face region of the portrait of the person.

[0204] 4. The terminal generates a first trimap based on the first image and the second image, the first trimap including a foreground region, a first scribble sub-region, and a second scribble sub-region.

[0205] 5. The terminal generates a target trimap based on a third image and the first trimap.

[0206] 6. The terminal inputs the target trimap and the original image into a matting model.

[0207] 7. The terminal acquires transparency output by the matting model, the transparency being used to represent a probability that a pixel belongs to a portrait.

[0208] 8. The terminal performs matting processing on the original image based on the transparency, to obtain a target image including a portrait. Subsequently, a user produces a sticker on the basis of the target image.

[0209] The image processing method provided in the embodiments of the present application can automatically extract a portrait, and the extracted portrait is fine and natural, thereby meeting the personalized requirement of a user for sticker production.

[0210] Scenario two: live broadcast scenario

[0211] In some live broadcast scenarios, a host may want to hide a real background environment in which the host is located, and display only a portrait of the host in a live broadcast picture, or add other virtual backgrounds to the portrait of the host, in order to protect personal privacy.

[0212] For example, a terminal provides a portrait mode in a live broadcast process. The host starts the portrait mode, so that the terminal acquires each frame of an original image captured by a camera in real time, and then extracts a portrait of the host from each frame of the original image by using the image processing method provided in the embodiments of the present application, and generates a live broadcast picture in real time for live broadcast. The terminal specifically extracts the portrait in a manner similar to that in the scenario one, and thus details are not described herein again.

[0213] It can be seen that the image processing method provided in the embodiments of the present application can be directly applied to the scenario in which a portrait needs to be extracted in real time, because the image processing method implements automatic portrait extraction.

[0214] Figure 14 is a structural schematic diagram of an image processing apparatus provided in the embodiments of the present application. The apparatus is used to execute the steps of the image processing method, and details are described in the image processing method. Figure 14 The apparatus includes an image segmentation module 1401, a trimap generation module 1402, and a matting module 1403.

[0215] The image segmentation module 1401 is configured to perform image semantic segmentation on the original image to obtain a first image, a second image, and a third image. The first image includes a foreground region in which a target object in the original image is located. The second image includes a hair region of the target object. The third image includes a face region of the target object.

[0216] The ternary graph generation module 1402 is configured to generate a target ternary graph based on the first image, the second image, and the third image. The target ternary graph includes the foreground region and a line drawing region. The line drawing region is obtained by drawing lines on an outline of the foreground region. Different sub-regions of the foreground region correspond to different line widths.

[0217] The matting module 1403 is configured to perform matting processing on the original image based on the target ternary graph to obtain a target image including the target object.

[0218] In an optional implementation, the foreground region further includes a torso region of the target object. In the target ternary graph, the line width corresponding to the hair region is greater than the line width corresponding to the torso region, and the line width corresponding to the torso region is greater than the line width corresponding to the face region.

[0219] In an optional implementation, the ternary graph generation module 1402 includes:

[0220] The first generation unit is configured to generate a first ternary graph based on the first image and the second image. The first ternary graph includes the foreground region, a first line drawing sub-region, and a second line drawing sub-region.

[0221] The first line drawing sub-region is overlaid on the outline of the hair region close to the background region, and the second line drawing sub-region is overlaid on the outline of other regions in the foreground region. The first line width is greater than the second line width. The first line width is used to draw the first line drawing sub-region, and the second line width is used to draw the second line drawing sub-region.

[0222] The second generation unit is configured to generate the target ternary graph based on the third image and the first ternary graph.

[0223] In an optional implementation, the first generation unit is configured to:

[0224] In the first image, the complete outline of the foreground region is obtained.

[0225] According to the second line width, draw lines on the complete contour line of the foreground region to obtain a second ternary graph; wherein the second ternary graph comprises the foreground region and a third drawing sub-region, and the third drawing sub-region covers the complete contour line of the foreground region;

[0226] In the second image, obtain a complete contour line of the hair region;

[0227] According to the first line width, draw lines on the complete contour line of the hair region to obtain a third ternary graph; wherein the third ternary graph comprises the hair region and a fourth drawing sub-region, and the fourth drawing sub-region covers the complete contour line of the hair region;

[0228] Merge the second ternary graph and the third ternary graph to obtain the first ternary graph.

[0229] In an optional implementation manner, the first line width is M times of the second line width, and M is greater than 1.

[0230] In an optional implementation manner, the first generating unit is further configured to:

[0231] Obtain a first identification value of each pixel point in the second ternary graph, and the first identification value is used to identify the color of the pixel point in the second ternary graph;

[0232] Obtain a second identification value of each pixel point in the third ternary graph, and the second identification value is used to identify the color of the pixel point in the third ternary graph;

[0233] Generate the first ternary graph based on the size relationship between the first identification value and the second identification value.

[0234] In an optional implementation manner, the first generating unit is further configured to:

[0235] Compare the first identification value of the pixel point at any position in the second ternary graph with the second identification value of the pixel point at the same position in the third ternary graph;

[0236] Take the maximum of the first identification value and the second identification value as a third identification value of the pixel point at the same position in the first ternary graph, and the third identification value is used to identify the color of the pixel point in the first ternary graph.

[0237] In an optional implementation manner, the second generating unit is configured to:

[0238] Determine a target overlapping region of the first ternary graph based on the face region in the third image, and the target overlapping region is an overlapping region of the face region and the second drawing sub-region;

[0239] The target identification value is used for identifying the color of the pixel point in the face region.

[0240] In an optional implementation, the matting module 1403 is configured to:

[0241] Based on the target ternary graph, the transparency of each pixel point in the original image is obtained, and the transparency is used to represent the probability that the pixel point belongs to the target object.

[0242] Based on the transparency, the original image is subjected to matting processing to obtain the target image.

[0243] In an optional implementation, the image segmentation module 1401 is further configured to:

[0244] The original image is obtained.

[0245] The original image is input into an image segmentation model, where the image segmentation model is configured to calculate the semantic category of each pixel point in the original image according to the input original image, to output at least one image of the original image.

[0246] The first image, the second image and the third image output by the image segmentation model are obtained.

[0247] In an optional implementation, the matting module 1403 is further configured to:

[0248] The target ternary graph and the original image are input into a matting model, where the matting model is configured to calculate the probability that each pixel point in the original image belongs to the target image according to the input target ternary graph and the original image, to output the transparency.

[0249] The transparency output by the matting model is obtained.

[0250] In the embodiments of the present application, when performing matting on the original image, first, the semantic segmentation method is used to obtain a plurality of segmentation images containing different regions, and further, according to the segmentation images, a line with different widths is drawn on the contour line of the foreground region to obtain a target ternary graph, and finally, the target image is generated based on the target ternary graph. For the above target ternary graph, since a line with different widths is drawn on the contour line of the foreground region, targeted matting of different regions can be achieved, the matting precision of the region that needs to be finely matted can be improved, and at the same time, the matting precision of other regions can be ensured, so that a matting image with fine and natural effect is finally obtained. In addition, the above matting process also realizes full automation, greatly improving the matting efficiency.

[0251] It should be noted that the image processing apparatus provided in the above embodiments is only taken as an example to illustrate the division of the above functional modules when performing image processing. In actual applications, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the image processing apparatus and the image processing method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0252] In an example embodiment, a computer device is also provided. Taking the computer device as a terminal for example, Figure 15 A structural diagram of a terminal 1500 provided in an example embodiment of the present application is shown. The terminal 1500 can be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer or a desktop computer. The terminal 1500 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0253] Generally, the terminal 1500 includes a processor 1501 and a memory 1502.

[0254] The processor 1501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1501 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1501 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also known as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1501 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by the display screen. In some embodiments, the processor 1501 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0255] The memory 1502 can include one or more computer-readable storage media that can be non-transitory. The memory 1502 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one program code for being executed by the processor 1501 to implement the image processing method provided by the method embodiments in the present application.

[0256] In some embodiments, the terminal 1500 can also optionally include a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502, and the peripheral device interface 1503 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1503 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, and a power supply 1509.

[0257] The peripheral interface 1503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502 and the peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502 and the peripheral interface 1503 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0258] The radio frequency circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1504 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area networks and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 1504 can also include NFC (Near Field Communication) related circuitry, which is not limited by the present application.

[0259] The display screen 1505 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 is further configured to capture touch signals on or above the surface of the display screen 1505. The touch signals can be input to the processor 1501 as control signals for processing. In this case, the display screen 1505 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1505 can be one, disposed on the front panel of the terminal 1500; in other embodiments, the display screen 1505 can be at least two, respectively disposed on different surfaces of the terminal 1500 or in a folding design; in other embodiments, the display screen 1505 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 1500. Even, the display screen 1505 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 1505 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0260] The camera assembly 1506 is configured to capture images or videos. Optionally, the camera assembly 1506 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1506 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0261] The audio circuit 1507 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1501 for processing, or input to the radio frequency circuit 1504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 1500. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can the electrical signal be converted into a sound wave that humans can hear, but also can be converted into a sound wave that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 1507 can also include a headphone jack.

[0262] The power supply 1509 is used to supply power to each component in the terminal 1500. The power supply 1509 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1509 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0263] In some embodiments, the terminal 1500 also includes one or more sensors 1510. The one or more sensors 1510 include, but are not limited to, an acceleration sensor 1511, a gyroscope sensor 1512, a pressure sensor 1513, an optical sensor 1515, and a proximity sensor 1516.

[0264] The acceleration sensor 1511 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal 1500. For example, the acceleration sensor 1511 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1501 can control the display screen 1505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1511. The acceleration sensor 1511 can also be used for game or user motion data collection.

[0265] The gyroscope sensor 1512 can detect the body direction and rotation angle of the terminal 1500, and the gyroscope sensor 1512 can collect 3D actions of the user on the terminal 1500 in cooperation with the acceleration sensor 1511. The processor 1501 can realize the following functions according to the data collected by the gyroscope sensor 1512: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0266] The pressure sensor 1513 can be disposed on the side bezel of the terminal 1500 and / or on the lower layer of the display screen 1505. When the pressure sensor 1513 is disposed on the side bezel of the terminal 1500, it can detect the user's grip signal on the terminal 1500, and the processor 1501 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1513. When the pressure sensor 1513 is disposed on the lower layer of the display screen 1505, the processor 1501 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0267] Optical sensor 1515 is used to collect ambient light intensity. In one embodiment, processor 1501 can control the display brightness of display screen 1505 based on the ambient light intensity collected by optical sensor 1515. Specifically, when the ambient light intensity is high, the display brightness of display screen 1505 is increased; when the ambient light intensity is low, the display brightness of display screen 1505 is decreased. In another embodiment, processor 1501 can also dynamically adjust the shooting parameters of camera assembly 1506 based on the ambient light intensity collected by optical sensor 1515.

[0268] The proximity sensor 1516, also known as a distance sensor, is typically located on the front panel of the terminal 1500. The proximity sensor 1516 is used to detect the distance between the user and the front of the terminal 1500. In one embodiment, when the proximity sensor 1516 detects that the distance between the user and the front of the terminal 1500 is gradually decreasing, the processor 1501 controls the display screen 1505 to switch from a screen-on state to a screen-off state; when the proximity sensor 1516 detects that the distance between the user and the front of the terminal 1500 is gradually increasing, the processor 1501 controls the display screen 1505 to switch from a screen-off state to a screen-on state.

[0269] Those skilled in the art will understand that Figure 15 The structure shown does not constitute a limitation on terminal 1500 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0270] This application also provides a computer-readable storage medium applied to a computer device. The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the computer device in the image processing method of the above embodiments.

[0271] The embodiment of the present application further provides a computer program product or computer program, which comprises computer program code stored in a computer readable storage medium. A processor of a computer device reads the computer program code from the computer readable storage medium, and the processor executes the computer program code, so that the computer device executes the image processing method provided in the various optional implementation manners.

[0272] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by a program to complete the related hardware, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0273] The above only describes optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: performing image semantic segmentation on an original image to obtain a first image, a second image and a third image, the first image comprising a foreground region and a background region in which a target object in the original image is located, the second image comprising a hair region and a background region of the target object, and the third image comprising a face region and a background region of the target object, different regions in each image being marked by different identification values, the identification values being used to identify the color of a pixel point in the image; drawing a line on a complete contour line of the foreground region in the first image according to a second line width to obtain a second ternary graph comprising the foreground region and a third line-drawing sub-region, the third line-drawing sub-region covering the complete contour line of the foreground region; drawing a line on a complete contour line of the hair region in the second image according to a first line width to obtain a third ternary graph comprising the hair region and a fourth line-drawing sub-region, the fourth line-drawing sub-region covering the complete contour line of the hair region; taking a maximum value of the identification values in the second ternary graph and the third ternary graph at any position as an identification value of the first ternary graph at the position to obtain the first ternary graph, the first ternary graph comprising the foreground region, a first line-drawing sub-region and a second line-drawing sub-region, the first line-drawing sub-region covering the contour line of the hair region close to the background region, the second line-drawing sub-region covering the contour line of other regions of the foreground region except the hair region, the identification values of the first line-drawing sub-region and the second line-drawing sub-region being different from the identification value of the foreground region and the identification value of the background region; the first line width for drawing the first line-drawing sub-region is greater than the second line width for drawing the second line-drawing sub-region; the region range of the foreground region is different in different images, and the region range of the background region is different in different images; determining an overlapping region of the face region and the second line-drawing sub-region in the first ternary graph based on the position of a pixel point of the face region in the third image, the identification value of the overlapping region in the first ternary graph being the identification value of the second line-drawing sub-region; reassigning the pixel points of the overlapping region by replacing the identification value of the overlapping region with the identification value of the face region to generate a target ternary graph; performing cutout processing on the original image based on the target ternary graph to obtain a target image comprising the target object.

2. The method of claim 1, wherein, The foreground region further comprises a torso region of the target object, wherein in the target ternary graph, the line width corresponding to the hair region is greater than the line width corresponding to the torso region, and the line width corresponding to the torso region is greater than the line width corresponding to the face region.

3. The method of claim 1, wherein, Before the step of drawing a line on a complete contour line of the foreground region in the first image according to a second line width to obtain a second ternary graph comprising the foreground region and a third line-drawing sub-region, the method further comprises: obtaining the complete contour line of the foreground region in the first image; Before the drawing of the line according to the first line width on the complete contour line of the hair region in the second image to obtain the third ternary graph including the hair region and the fourth drawing sub-region, the method further comprises: Obtaining the complete contour line of the hair region in the second image.

4. The method according to claim 1 or 3, characterized in that, The first line width is M times of the second line width, and M is greater than 1.

5. The method of claim 4, wherein, The maximum value of the identification values of any position in the second ternary graph and the third ternary graph is taken as the identification value of the position in the first ternary graph to obtain the first ternary graph, comprising: Obtaining a first identification value of each pixel point in the second ternary graph, the first identification value being used to identify the color of the pixel point in the second ternary graph; Obtaining a second identification value of each pixel point in the third ternary graph, the second identification value being used to identify the color of the pixel point in the third ternary graph; Generating the first ternary graph based on the size relationship between the first identification value and the second identification value.

6. The method of claim 5, wherein, The generating of the first ternary graph based on the size relationship between the first identification value and the second identification value comprises: Comparing the first identification value of the pixel point at any position in the second ternary graph with the second identification value of the pixel point at the same position in the third ternary graph; Taking the maximum of the first identification value and the second identification value as a third identification value of the pixel point at the same position in the first ternary graph, the third identification value being used to identify the color of the pixel point in the first ternary graph.

7. The method of claim 1, wherein, The image matting processing of the original image based on the target ternary graph to obtain a target image including the target object, comprising: Obtaining a transparency of each pixel point in the original image based on the target ternary graph, the transparency being used to represent the probability that the pixel point belongs to the target object; Performing the image matting processing on the original image based on the transparency to obtain the target image.

8. An image processing apparatus characterized by comprising: The device comprises: An image segmentation module, configured to perform image semantic segmentation on an original image to obtain a first image, a second image and a third image, the first image including a foreground region where a target object is located and a background region, the second image including a hair region of the target object and a background region, and the third image including a face region of the target object and a background region, different regions in each image being marked by different identification values, the identification values being used to identify the color of the pixel point in the image; A ternary graph generation module, comprising: The first generating unit is configured to draw a line on a complete contour line of the foreground region in the first image according to a second line width, to obtain a second ternary graph including the foreground region and a third line-drawing sub-region, the third line-drawing sub-region covering the complete contour line of the foreground region; draw a line on a complete contour line of the hair region in the second image according to a first line width, to obtain a third ternary graph including the hair region and a fourth line-drawing sub-region, the fourth line-drawing sub-region covering the complete contour line of the hair region; take a maximum value of identification values in the second ternary graph and the third ternary graph at any position as an identification value of the first ternary graph at the position, to obtain the first ternary graph, the first ternary graph including the foreground region, a first line-drawing sub-region and a second line-drawing sub-region, the first line-drawing sub-region covering the contour line of the hair region close to the background region, the second line-drawing sub-region covering the contour line of the region of the foreground region other than the hair region, the identification values of the first line-drawing sub-region and the second line-drawing sub-region being different from the identification value of the foreground region and different from the identification value of the background region; the first line width for drawing the first line-drawing sub-region is greater than the second line width for drawing the second line-drawing sub-region; the region range of the foreground region in different images is different, and the region range of the background region in different images is different; The second generating unit is configured to determine an overlapping region of the face region and the second line-drawing sub-region in the first ternary graph based on a pixel position of the face region in the third image, the identification value of the overlapping region in the first ternary graph being the identification value of the second line-drawing sub-region; replace the identification value of the overlapping region with the identification value of the face region to revalue the pixel points of the overlapping region, to generate a target ternary graph. The matting module is configured to perform matting processing on the original image based on the target ternary graph, to obtain a target image including the target object.

9. The apparatus of claim 8, wherein, The foreground region further includes a torso region of the target object, and in the target ternary graph, the line width corresponding to the hair region is greater than the line width corresponding to the torso region, and the line width corresponding to the torso region is greater than the line width corresponding to the face region.

10. The apparatus of claim 8, wherein, The first generating unit is further configured to: obtain a complete contour line of the foreground region in the first image; obtain a complete contour line of the hair region in the second image.

11. The apparatus of claim 8 or 10, wherein, The first line width is M times the second line width, and M is greater than 1.

12. The apparatus of claim 11, wherein, The first generating unit is configured to: obtain a first identification value of each pixel point in the second ternary graph, the first identification value being used to identify the color of the pixel point in the second ternary graph; obtain a second identification value of each pixel point in the third ternary graph, the second identification value being used to identify the color of the pixel point in the third ternary graph; generate the first ternary graph based on a size relationship between the first identification value and the second identification value.

13. The apparatus of claim 12, wherein, The first generating unit is configured to: The first identification value of a pixel point at any position in the second ternary graph is compared with the second identification value of a pixel point at the same position in the third ternary graph; The maximum of the first identification value and the second identification value is taken as a third identification value of a pixel point at the same position in the first ternary graph, and the third identification value is used to identify the color of the pixel point in the first ternary graph.

14. The apparatus of claim 8, wherein, The image processing method comprises the following steps: Based on the target ternary graph, the transparency of each pixel point in the original image is obtained, and the transparency is used to represent the probability that the pixel point belongs to the target object; The original image is processed based on the transparency to obtain the target image.

15. A computer device, comprising: The computer device comprises a processor and a memory, the memory is used to store at least one computer program, the at least one computer program is loaded and executed by the processor, and the image processing method in any one of claims 1 to 7 is executed.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, the at least one computer program is loaded and executed by the processor to realize the image processing method in any one of claims 1 to 7.

17. A computer program product, the computer program product comprises computer program code, the computer program code is stored in a computer readable storage medium, a processor of a computer device reads the computer program code from the computer readable storage medium, and the processor executes the computer program code, so that the computer device is executed to realize the image processing method in any one of claims 1 to 7.

Images