A portrait segmentation method and device

Abstract

The present application is applicable to the field of digital image processing and provides a portrait segmentation method and device, the method comprising: obtaining an original portrait image, wherein the original portrait image contains a portrait; inputting the original portrait image into a portrait segmentation network for processing, and outputting a portrait mask image; the portrait segmentation network comprises an encoding module, a pyramid pooling module, and a decoding module; inputting the portrait mask image and the original portrait image into an image restoration module for processing, and outputting an image restoration mask image; and dividing the original portrait image into a portrait image and a background image according to the image restoration mask image. In the present application, when constructing the portrait segmentation network, an encoding module, a pyramid pooling module, and a decoding module are provided to reduce the amount of computation required by the portrait segmentation network to obtain the portrait mask image, thereby reducing the computing power requirement of the terminal device equipped with the portrait segmentation network, and improving the accuracy of portrait segmentation by providing the image restoration module.

Description

Technical Field

[0001] The present application relates to the field of digital image processing, and in particular to a method and device for segmenting a human figure. Background Art

[0002] With the development of technology, more and more people rely on mobile devices such as mobile phones to take photos and videos. However, due to factors such as the hardware performance of mobile devices, they cannot meet users' needs for real-time background blur and replacement in application scenarios such as video chats and conference calls.

[0003] In the prior art, when using a mobile device to blur the background of an image or video containing a portrait, the user is required to manually mark the portrait to achieve the background blur. However, this makes it impossible to achieve fully automatic processing of the captured content and meet user needs in application scenarios such as video chat. Summary of the Invention

[0004] The embodiments of the present application provide a portrait segmentation method and device, which can segment portraits from images or videos based on a portrait segmentation network, solving the problem in the prior art that it is impossible to achieve fully automatic real-time portrait segmentation to blur and replace the background in mobile devices with limited computing power.

[0005] In a first aspect, an embodiment of the present application provides a portrait segmentation method, comprising:

[0006] Acquire an original portrait image, where the original portrait image contains a portrait;

[0007] The original portrait image is input into a portrait segmentation network for processing, and a portrait mask image is output; wherein the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module;

[0008] Inputting the portrait mask image and the original portrait image into an image restoration module for processing, and outputting an image restoration mask image;

[0009] The original portrait image is divided into a portrait image and a background image according to the image restoration mask image.

[0010] In a possible implementation of the first aspect, after dividing the original portrait image into a portrait image and a background image according to the image restoration mask image, the portrait segmentation method also includes: performing special processing on the background image according to user operation to generate a new background image; combining the portrait image and the new background image to generate a new portrait image, and displaying the new portrait image.

[0011] Exemplarily, the above-mentioned special processing can be blurring, replacement, adding filters or other image processing methods. The specific method still needs to be determined according to the user's needs (learned by receiving user operations). The image generated after the background image is specially processed is the new background image.

[0012] It should be understood that the aforementioned acquisition of the user's original portrait image includes: acquiring the user's original portrait video; and identifying each frame image within the original portrait video as the original portrait image. That is, the method provided in this application can not only perform portrait segmentation on an image, but also perform portrait segmentation on a video composed of multiple frame images, thereby achieving real-time portrait segmentation. Subsequently, after dividing the original portrait image into a portrait image and a background image based on the image restoration mask image, the portrait segmentation method further includes: generating a portrait video based on the portrait image corresponding to all frame images; and generating a background video based on the background image corresponding to all frame images.

[0013] In a second aspect, an embodiment of the present application provides a portrait segmentation device, comprising:

[0014] The original portrait image acquisition unit is used to acquire the original portrait image, wherein the original portrait image contains a portrait;

[0015] A portrait segmentation network unit is used to input the original portrait image into the portrait segmentation network for processing and output a portrait mask image; wherein the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module;

[0016] An image restoration unit, configured to input the portrait mask image and the original portrait image into an image restoration module for processing, and output an image restoration mask image;

[0017] The portrait segmentation unit is used to divide the original portrait image into a portrait image and a background image according to the image restoration mask image.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the computer program, when executed by the processor, implements the method described in any one of the first aspects above.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the method described in any one of the first aspects above.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the first aspect above.

[0021] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0022] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0023] Compared with the existing technology, the method provided by this application sets an encoding module, a pyramid pooling module and a decoding module when constructing a portrait segmentation network, thereby reducing the amount of computation when the portrait segmentation network obtains a portrait mask image; when segmenting a portrait from an image or video based on the portrait segmentation network, the accuracy of the portrait segmentation is ensured while reducing the amount of computation, thereby reducing the computing power requirements of a terminal device equipped with the portrait segmentation network; and by setting an image restoration module, the computing power requirements of the terminal device are reduced while improving the accuracy of subsequent portrait segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a flowchart of the implementation of the method provided in the first embodiment of the present application;

[0026] Figure 2 This is a flowchart of the implementation of the method provided in the second embodiment of the present application;

[0027] Figure 3 This is a structural diagram of a portrait segmentation network provided by an embodiment of the present application;

[0028] Figure 4 This is a flowchart of the implementation of the method provided in the third embodiment of the present application;

[0029] Figure 5 Schematic diagram of the pyramid pooling module structure provided by an embodiment of the present application;

[0030] Figure 6 This is a flowchart of the implementation of the method provided in the fourth embodiment of the present application;

[0031] Figure 7 This is a schematic diagram of the decoding unit structure provided by an embodiment of the present application;

[0032] Figure 8 This is a schematic diagram of the structure of a conversion unit provided in one embodiment of the present application;

[0033] Figure 9 is a flowchart of the implementation of the method provided in the fifth embodiment of the present application;

[0034] Figure 10 This is a flowchart of the implementation of the method provided in the sixth embodiment of the present application;

[0035] Figure 11 is a flowchart of the implementation of the method provided in the seventh embodiment of the present application;

[0036] Figure 12 is a visualization of the mask truth image and the boundary truth image provided by an embodiment of the present application;

[0037] Figure 13 This is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0038] Figure 14 It is a structural diagram of a device provided in one embodiment of the present application;

[0039] Figure 15 It is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0041] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0042] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0043] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0044] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0046] The terminology of this application is briefly explained here. "Feature information" includes the correlation between each pixel in the feature image. "Size" refers to the pixels (height and width) of the feature image. "Channel" refers to the number of layers of the feature image. "Feature fusion" means: feature fusion of feature image A and feature image B to extract the feature information contained in both. Specifically, if the feature image A and the feature image B have the same size, the two are directly channel-stitched to obtain a feature fusion image with the same size as the two and the sum of the number of channels; if the feature image A and the feature image B have different sizes, either feature image is processed to make it the same size as the other feature image. Specifically, a bilinear interpolation operation can be performed on the smaller feature image to obtain a feature image with the same size as the larger feature image. "Channel adjustment convolution processing" refers to: performing point convolution processing with a convolution kernel of 1*1 on the target feature image, which is mainly used to adjust the number of channels of the target feature image. Specifically, by adjusting the convolution parameters in the point convolution processing with a convolution kernel of 1*1 (the convolution parameter is specifically depth_multiplier, which is used to control the number of depth-direction convolution output channels of each input channel, and the value of the convolution parameter is the ratio of the total number of output channels to the total number of input channels), the number of channels of the target feature image can be adjusted, including increasing, decreasing, or keeping the number of channels unchanged. "Depth-separable convolution processing" refers to performing depth-separable convolution processing on the target feature image, and then performing point convolution processing on the result of the depth-separable convolution processing. Specifically, by adjusting the convolution parameters of the depth-separable convolution processing, the size and number of channels of the target feature image can be adjusted.

[0047] In the embodiments of the present application, the process is performed by a terminal device. This terminal device includes, but is not limited to, a server, a computer, a smartphone, a tablet computer, and other devices capable of executing the portrait segmentation method provided herein. Preferably, the terminal device is a mobile terminal capable of acquiring the original portrait image of the user. Figure 1 The following is a flowchart of the method according to the first embodiment of the present invention, which is described in detail as follows:

[0048] In S101 , an original portrait image is acquired, wherein the original portrait image includes a portrait.

[0049] In this embodiment, the original portrait image includes a portrait. The above-mentioned obtaining of the original portrait image may specifically be: obtaining the original portrait image through a camera of the terminal device.

[0050] It should be understood that obtaining the original portrait image mentioned above includes: obtaining an original portrait video; and identifying each frame image within the original portrait video as the original portrait image. In other words, the method provided in this embodiment can not only perform portrait segmentation on images, but also on each frame image within a video, thereby achieving real-time portrait segmentation on videos.

[0051] In S102 , the original portrait image is input into the portrait segmentation network for processing, and a portrait mask image is output.

[0052] In this embodiment, the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module. The encoding module can be a preset convolutional neural network. Compared with a general convolutional neural network, the encoding module uses a depthwise separable convolutional layer instead of a standard convolutional layer to reduce the amount of computation when extracting feature information of the original portrait image; the pyramid pooling module is used to obtain global information about the original portrait image based on the feature information to improve the accuracy of the subsequent output of the portrait mask image; and the decoding module is used to generate the portrait mask image based on the feature information and the global information.

[0053] In one possible implementation, the above-mentioned inputting the original portrait image into the portrait segmentation network for processing and outputting the portrait mask image can be specifically: inputting the original portrait image into the encoding module for processing and outputting a feature information image; inputting the feature information image into the pyramid pooling module for processing and outputting a global information image; and inputting the feature information and the global information image into the decoding module for processing and outputting the portrait mask image.

[0054] In S103 , the portrait mask image and the original portrait image are input into an image restoration module for processing, and an image restoration mask image is output.

[0055] In this embodiment, the image restoration module is used to integrate the feature information of the original portrait image with the feature information of the portrait mask image to obtain an image restoration mask image containing more detailed feature information.

[0056] In one possible implementation, the portrait mask image and the original portrait image are input into the image restoration module for processing, and the image restoration mask image is output. Specifically, the portrait mask image and the original portrait image are subjected to feature fusion to obtain the image restoration mask image.

[0057] In S104 , the original portrait image is divided into a portrait image and a background image according to the image restoration mask image.

[0058] In this embodiment, the image restoration mask image contains classification information for each pixel in the original portrait image. The above-described division of the original portrait image into a portrait image and a background image based on the image restoration mask image specifically involves classifying each pixel in the original portrait image as either a portrait pixel or a background pixel based on the image restoration mask image; generating a portrait image based on all the portrait pixels; and generating a background image based on all the background pixels. It should be understood that the division of the original portrait image into a portrait image and a background image achieves portrait segmentation.

[0059] In one possible implementation, the image restoration mask image includes the probability of each pixel in the original portrait image being a portrait pixel. Classifying each pixel in the original portrait image as a portrait pixel or a background pixel based on the image restoration mask image may be performed as follows: if the probability of the pixel being a portrait pixel is greater than or equal to a preset threshold (exemplarily, the preset threshold is 0.5), the pixel is identified as a portrait pixel; if the probability of the pixel being a portrait pixel is less than the preset threshold (exemplarily, the preset threshold is 0.5), the pixel is identified as a background pixel.

[0060] In this embodiment, when constructing the portrait segmentation network, by setting an encoding module, a pyramid pooling module and a decoding module, the structure of the portrait segmentation network is optimized compared to the neural network in the prior art. The encoding module uses a depthwise separable convolution layer instead of a standard convolution layer. Since the convolution kernel of the depthwise separable convolution layer is often smaller than the convolution kernel of the standard convolution layer, the computational complexity of the encoding module is reduced, thereby reducing the computational complexity of the portrait segmentation network. When segmenting the portrait in the image or video based on the portrait segmentation network, the accuracy of the portrait segmentation is guaranteed while reducing the computational complexity, thereby reducing the computing power requirements of the device equipped with the portrait segmentation network. Setting an image restoration module can further improve the accuracy of portrait segmentation.

[0061] Figure 2 The flowchart of the method provided in the second embodiment of the present application is shown. Figure 2 , relative to the above Figure 1 In the described embodiment, the method S102 provided in this embodiment includes S201 to S204, which are described in detail as follows:

[0062] Furthermore, the above-mentioned inputting the original portrait image into the portrait segmentation network for processing and outputting the portrait mask image includes:

[0063] In S201 , the original portrait image is input into the encoding module, and the depthwise separable convolution algorithm corresponding to the encoding module is used to perform multiple convolution processes on the original portrait image to output N downsampled feature images.

[0064] In this embodiment, N is an integer greater than 2. Generally, the value range of N is 4 to 6, and the value of N is preferably 5. This value is used to balance the computational complexity and the recognition accuracy of the portrait segmentation network. Compared with general convolutional neural networks (such as mobilenetV2), the encoding module uses a depthwise separable convolutional layer instead of a standard convolutional layer. Since the convolution kernel of the depthwise separable convolutional layer is often smaller than the convolution kernel of the standard convolutional layer, the computational complexity when extracting the feature information from the original portrait image is reduced compared to general convolutional neural networks. The depthwise separable convolutional layer includes at least one depthwise convolutional layer (depthwiseconv) and at least one pointwise convolutional layer (pointwiseconv).

[0065] In this embodiment, the original portrait image is input into the encoding module, and the original portrait image is subjected to multiple convolution processes using the depthwise separable convolution algorithm corresponding to the encoding module, outputting N downsampled feature images. Specifically, the original portrait image is subjected to a channel adjustment convolution process to obtain a channel-expanded image; the channel-expanded image is input into a series network composed of multiple encoding units of the encoding module to obtain multiple encoded result images, wherein the encoding unit includes at least one of the depthwise separable convolution layers. Optionally, the encoding unit may also include a downsampling layer, wherein the ratio of the input image size to the output image size of the downsampling layer conforms to a preset downsampling ratio; wherein the input image size is generally larger than the output image size, and the downsampling ratio of the two may be 2; and N of the encoded result images are selected and identified as the above-mentioned N downsampled feature images. The above-mentioned channel adjustment convolution process is performed once on the original portrait image to obtain the channel-expanded image. Specifically, by presetting the convolution parameters of the channel adjustment convolution process, the number of channels of the original portrait image can be increased, thereby obtaining the channel-expanded image with the increased number of channels.

[0066] Figure 3 FIG. 1 shows a schematic diagram of the structure of a portrait segmentation network provided by an embodiment of the present application. In one possible implementation, Figure 3 As shown, the encoding module of the portrait segmentation network includes 5 encoding units, that is, N=5 as mentioned above, where encoding units 1, 2, 3, and 4 all include downsampling layers, and encoding unit 5 does not include a downsampling layer. Each encoding unit in the figure outputs a corresponding downsampled feature image. The original portrait image is input into the encoding module, and the original portrait image is sequentially passed through the channel adjustment convolution, encoding unit 1, encoding unit 2, encoding unit 3, encoding unit 4, and encoding unit 5 in the figure, and the first downsampled feature image, the second downsampled feature image, the third downsampled feature image, the fourth downsampled feature image, and the fifth downsampled feature image are output.

[0067] It should be understood that the encoding unit 5 shown in the figure, that is, the fifth encoding unit, can be referred to here when all the units followed by numbers in the explanation figures in this application are explained; the above-mentioned first downsampled feature image, that is, the first downsampled feature image, can be referred to here when all the explanations of image names in this application are preceded by serial numbers, for example, the Nth downsampled feature image or the Nth downsampled feature image are the same thing.

[0068] In S202 , the Nth down-sampled feature image is input into a pyramid pooling module for processing, and an Nth up-sampled feature image is output.

[0069] In this embodiment, compared with the pooling layer of the existing neural network, the pyramid pooling module uses a dilated convolution layer instead of the pooling layer. Specifically, the Nth down-sampled feature image is input into the pyramid pooling module for processing, and the Nth up-sampled feature image is output. The steps can be as follows: performing dilated convolution on the Nth down-sampled feature image to obtain a dilated convolution feature image; adjusting the size of the dilated convolution image so that the size of the dilated convolution feature image is the same as that of the Nth down-sampled feature image; and performing channel splicing on the dilated convolution feature image and the Nth down-sampled feature image to obtain the Nth up-sampled feature image.

[0070] In one possible implementation, Figure 3 As shown, N is 5, the fifth down-sampled feature image is input into the pyramid pooling module for processing, and the fifth up-sampled feature image is output.

[0071] In S203 , the Nth up-sampled feature image and N−1 down-sampled feature images of the N down-sampled feature images except the Nth down-sampled feature image are input to a decoding module for processing, and a first up-sampled feature image is output.

[0072] In this embodiment, the decoding module includes N-1 decoding units, and the M-1th decoding unit is used to determine the M-1th upsampled feature image based on the M-1th upsampled feature image and the M-1th downsampled feature image, where M is an integer less than or equal to N and is greater than 0. The M-1th downsampled feature image has the same size as the M-1th upsampled feature image.

[0073] In this embodiment, the N-th up-sampled feature image and the N-1 down-sampled feature images of the N down-sampled feature images other than the N-th down-sampled feature image are input into the decoding module for processing, and the first up-sampled feature image is output. Specifically, it may include: at the beginning, that is, when the value of the above-mentioned M is N, the N-th up-sampled feature image and the N-1-th down-sampled feature image are input into the N-1-th decoding unit for processing, and the N-1-th up-sampled feature image is output; and so on. Finally, that is, when the value of the above-mentioned M is 2, the second up-sampled feature image and the first down-sampled feature image are input into the first decoding unit for processing, and the first up-sampled feature image is output.

[0074] In one possible implementation, Figure 3 As shown, N is 5, the fifth up-sampled feature image and the fourth down-sampled feature image are input into the graphic decoding unit 4 (i.e., the fourth decoding unit) for processing, and the fourth up-sampled feature image is output... and so on, and finally the graphic decoding unit 1 (i.e., the first decoding unit) outputs the first up-sampled feature image.

[0075] In S204 , image mask processing is performed on the first up-sampled feature image to obtain a portrait mask image.

[0076] In this embodiment, the first up-sampled feature image is subjected to image mask processing to obtain a portrait mask image, which can be specifically: Figure 3 , performing a channel adjustment convolution process on the first up-sampled feature image. By pre-setting the convolution parameters of the channel adjustment convolution process, the number of channels of the first up-sampled feature image can be reduced to obtain a channel-reduced image with reduced channels; performing a bilinear interpolation operation on the channel-reduced image to keep the size of the channel-reduced image consistent with the above-mentioned original portrait image, that is, to enlarge the size of the channel-reduced image, to obtain the portrait mask image.

[0077] In this embodiment, the encoding module uses a depth-wise separable convolutional layer instead of a standard convolutional layer, which helps to reduce the computational complexity of the portrait segmentation network and improve output efficiency, thereby reducing the computing power requirements of a terminal device (preferably a mobile device) executing the embodiment of the present application; the pyramid pooling module uses a hole convolutional layer instead of a pooling layer, so that the portrait segmentation network can obtain rich global information of the original portrait image, thereby improving the accuracy of subsequent portrait segmentation; the decoding module includes multiple decoding units corresponding to the encoding module, so as to obtain the portrait mask image based on the original portrait image, so as to facilitate subsequent portrait segmentation.

[0078] Figure 4 The flowchart of the method provided in the third embodiment of the present application is shown. Figure 4 , relative to the above Figure 2 In the described embodiment, the method S202 provided in this embodiment includes S2021 to S2023, which are described in detail as follows:

[0079] Furthermore, the Nth down-sampled feature image is input into the pyramid pooling module for processing, and the Nth up-sampled feature image is output, including:

[0080] In S2021, a plurality of preset convolution kernels of different scales are respectively used to perform dilated convolution processing on the Nth down-sampled feature image to obtain a plurality of dilated convolution feature images.

[0081] In this embodiment, the multiple atrous convolution feature images correspond one-to-one to the multiple preset convolution kernels of different scales. The atrous convolution process is used to increase the receptive field of the Nth down-sampled feature image, thereby improving the ability of the portrait segmentation network to obtain more feature information in the Nth down-sampled feature image. When the atrous convolution process is performed on the Nth down-sampled feature image, the multiple convolution kernels of different scales are used to obtain different atrous convolution feature images, so as to enrich the feature information of the subsequently obtained Nth up-sampled feature image. The number of the multiple preset convolution kernels of different scales is preferably 4, so as to balance the beneficial effects of enriching the feature information of the Nth up-sampled feature image and reducing the amount of computation.

[0082] Figure 5 FIG. 1 shows a schematic diagram of the pyramid pooling module structure provided by an embodiment of the present application. In one possible implementation, Figure 5 As shown in the figure, four convolution kernels of 1*1, 2*2, 3*3 and 6*6 are used to perform dilated convolution processing on the Nth downsampled feature image, and four dilated convolution feature images are obtained.

[0083] In S2022, convolution processing is performed on each dilated convolution feature image to obtain a plurality of feature images to be fused that correspond one-to-one to the plurality of dilated convolution feature images.

[0084] In this embodiment, the above-mentioned convolution processing is performed on each dilated convolution feature image to obtain multiple feature images to be fused that correspond one-to-one to the multiple dilated convolution feature images. Specifically, the convolution processing includes: performing a channel adjustment convolution processing on the dilated convolution feature image, and by presetting the convolution parameters of the channel adjustment convolution processing, the channels of the dilated convolution feature image can be reduced to obtain multiple feature images to be fused that correspond one-to-one to the multiple dilated convolution feature images after the number of channels is reduced. Preferably, the total number of channels of all the multiple feature images to be fused is the same as the Nth downsampled feature image, and the number of channels of each feature image to be fused is the same.

[0085] In one possible implementation, Figure 5 As shown in FIG, four hole convolution feature images are convolved to obtain four feature images to be fused. The number of channels of each feature image to be fused is one quarter of the Nth down-sampled feature image.

[0086] In S2023 , multiple feature images to be fused are subjected to feature fusion to obtain a pooled feature image.

[0087] In this embodiment, the feature fusion of the multiple feature images to be fused is performed to obtain a pooled feature image. Specifically, the multiple feature images to be fused are subjected to bilinear interpolation operations respectively to obtain multiple feature images to be spliced ​​that correspond one-to-one to the multiple feature images to be fused, and the feature images to be spliced ​​have the same size as the Nth down-sampled feature image; all the multiple feature images to be spliced ​​that have the same size are performed channel splicing to obtain a pooled feature image with the same number of channels as the Nth down-sampled feature image.

[0088] In one possible implementation, Figure 5 As shown, the four feature images to be fused are subjected to feature fusion to obtain a pooled feature image. Specifically, the four feature images to be fused are subjected to bilinear interpolation operations respectively to obtain four feature images to be spliced ​​whose sizes are the same as those of the Nth down-sampled feature image; the four feature images to be spliced ​​are subjected to channel splicing to obtain a pooled feature image whose size and number of channels are the same as those of the Nth down-sampled feature image.

[0089] In S2024, the pooled feature image and the Nth down-sampled feature image are channel-joined to obtain the Nth up-sampled feature image.

[0090] In this embodiment, the number of channels of the Nth up-sampled feature image is twice the number of channels of the Nth down-sampled feature image.

[0091] In this embodiment, in the pyramid pooling module, multiple convolution kernels of different scales are used to perform convolution processing on the Nth downsampled feature image, which can extract richer feature information; the pooled feature image is subsequently channel-joined with the Nth downsampled feature image, and the resulting Nth upsampled feature image contains richer global information and aggregates contextual information from convolution layers of different scales, so as to subsequently improve the accuracy of portrait segmentation.

[0092] Figure 6 FIG4 shows a flowchart of the method provided in the fourth embodiment of the present application. Figure 6 , relative to the above Figure 2 In the described embodiment, the method S203 provided in this embodiment includes S2031 to S2033, which are described in detail as follows:

[0093] Furthermore, the determining of the M-1th up-sampled feature image according to the Mth up-sampled feature image and the M-1th down-sampled feature image includes:

[0094] In this embodiment, the decoding unit includes an upsampling layer, a feature fusion layer, and a conversion unit.

[0095] In S2031 , the Mth up-sampled feature image is input into the up-sampling layer of the M-1th decoding unit, up-sampled feature mapping is performed on the Mth up-sampled feature image, and the M-1th initial feature image is output.

[0096] In this embodiment, the ratio of the input image size to the output image size of the upsampling layer conforms to a preset upsampling ratio; wherein, the input image size is generally smaller than the output image size, and the downsampling ratio of the two can be 0.5; the upsampling ratio and the above-mentioned downsampling ratio are reciprocals of each other; the above-mentioned input of the Mth upsampled feature image into the upsampling layer of the M-1th decoding unit, performing upsampled feature mapping on the Mth upsampled feature image, and outputting the M-1th initial feature image can be specifically: performing a bilinear interpolation operation on the Mth upsampled feature image to obtain the M-1th initial feature image with an enlarged size.

[0097] In one possible implementation, Figure 3 As shown, it should be noted that the fourth decoding unit does not include the above-mentioned upsampling layer. There are a total of 4 downsampling layers in the encoding module shown in the figure. The size of the original portrait image and the source portrait image should be the same, so the decoding module should have 3 upsampling layers to offset the size changes of the 3 downsampling layers (the bilinear interpolation in the figure offsets the size change of the remaining 1 downsampling layer); the encoding unit 5 shown in the figure does not include a downsampling layer, that is, the size of the fourth downsampling feature image is the same as the size of the fifth downsampling feature image, that is, the size of the fourth downsampling feature image is the same as the size of the fifth upsampling feature image, so the fourth decoding unit does not include the above-mentioned upsampling layer. It should be understood that whether the fourth decoding unit shown in the figure includes an upsampling layer depends on the total number of downsampling layers in the encoding module.

[0098] At this time, if M is 5, the steps of inputting the fifth upsampled feature image into the upsampled layer of the fourth decoding unit, performing upsampled feature mapping on the fifth upsampled feature image, and outputting the fourth initial feature image are not performed. Instead, the fifth upsampled feature image is identified as the fourth initial feature image for subsequent steps.

[0099] In S2032, the M-1th initial feature image and the M-1th downsampled feature image are input into the feature fusion layer of the M-1th decoding unit, and feature fusion is performed on the M-1th initial feature image and the M-1th downsampled feature image to obtain the M-1th upsampled feature fusion image.

[0100] In this embodiment, the M-1th initial feature image and the M-1th down-sampled feature image have the same size. Therefore, the above-mentioned feature fusion of the M-1th initial feature image and the M-1th down-sampled feature image is performed to obtain the M-1th up-sampled feature fusion image. Specifically, the M-1th initial feature image and the M-1th down-sampled feature image are channel-stitched to obtain the M-1th up-sampled feature fusion image with the same size as the sum of the number of channels of the two.

[0101] Figure 7 FIG1 shows a schematic diagram of the structure of a decoding unit provided by an embodiment of the present application. Figure 3 , Figure 7 The schematic diagram of the decoding unit structure shown is only used to describe Figure 3 The decoding unit of the upsampling layer is included in Figure 3 The decoding unit that does not include the upsampling layer can refer to Figure 7 , the only difference is that no upsampling layer is included.

[0102] In one possible implementation, Figure 7 As shown, the Mth up-sampled feature image is input into the up-sampling layer in the illustrated decoding unit for processing, and the M-1th down-sampled feature image is input into the feature fusion layer in the illustrated decoding unit for processing, and finally the decoding unit outputs the M-1th up-sampled feature image.

[0103] In S2033 , the M-1th up-sampled feature fusion image is input to the conversion unit of the M-1th decoding unit for processing, and the M-1th up-sampled feature image is output.

[0104] In this embodiment, the conversion unit may include at least one depthwise separable convolution layer and at least one channel adjustment convolution layer. The conversion unit may also include at least one feature fusion block.

[0105] Figure 8 FIG. 1 shows a schematic diagram of the structure of a conversion unit provided by an embodiment of the present application. In a possible implementation, as shown in FIG. Figure 8As shown in (a), the conversion unit includes two depth-wise separable convolution layers, a channel adjustment convolution layer and a feature fusion block; the implementation of the above S2033 can be specifically as follows: the M-1th up-sampled feature fusion image is passed through two depth-wise separable convolution layers connected in series to obtain the M-1th image to be converted; the depth-wise separable convolution layer includes a depth-wise convolution layer and a point-wise convolution layer; the M-1th up-sampled feature fusion image is passed through a channel adjustment convolution layer to obtain the M-1th channel-adjusted image with adjusted channels; the M-1th image to be converted and the M-1th channel adjustment image are input into the feature fusion block for feature fusion to obtain the M-1th up-sampled feature image. That is, in this conversion unit, the M-1th upsampled feature fusion image is input and the M-1th upsampled feature image is output, wherein an adjustment channel convolution layer is connected in parallel with two depth-separable convolution layers in series, and the outputs of the above-mentioned one adjustment channel convolution layer and the above-mentioned two depth-separable convolution layers in series are both inputs of the feature fusion block.

[0106] In another possible implementation, Figure 8 As shown in (b), the conversion unit includes four depth-wise separable convolutional layers, two adjustment channel convolutional layers and two feature fusion blocks; the implementation of the above S2033 can be specifically referred to the steps of one of the possible implementation methods mentioned above, except that the implementation of S2033 here is specifically to repeat the steps of one of the possible implementation methods twice, and the output of the first execution of the steps of one of the possible implementation methods is used as the input of the second execution of the steps of one of the possible implementation methods, and the output of the second execution of the steps of one of the possible implementation methods is used as the output of the conversion unit.

[0107] Specifically, each Figure 8 The convolution kernel of the depth convolution layer shown in is 3*3, each Figure 8 The convolution kernel of the point convolution layer shown in is 1*1, and each Figure 8 The convolutional layers shown in the figure all contain a RELU activation function. It should be understood that each Figure 8 After the convolutional layer in , there is a data batch normalization (BN) step to avoid the problem of internal covariate shift in the portrait segmentation network. For details, please refer to the relevant description of the prior art and will not be repeated here.

[0108] In this embodiment, in particular, when outputting the upsampled feature image, by performing feature fusion with the downsampled feature image, the feature information of the output upsampled feature image can be enriched, including the correlation with the downsampled feature information of each layer, so as to improve the accuracy of portrait segmentation.

[0109] Figure 9 FIG1 shows a flowchart of the implementation of the method provided in the fifth embodiment of the present application. Figure 9 , relative to the above Figure 1 In the described embodiment, the method S103 provided in this embodiment includes S1031 to S1033, which are described in detail as follows:

[0110] Furthermore, the portrait mask image and the original portrait image are input into the image restoration module for processing to obtain the image restoration mask image, including:

[0111] In this embodiment, the image restoration module includes at least two depth-wise separable convolutional layers connected in series.

[0112] In S1031 , the portrait mask image and the original portrait image are subjected to feature fusion to obtain a feature fused image.

[0113] In this embodiment, the portrait mask image has the same size as the original portrait image. The above-mentioned feature fusion of the portrait mask image and the original portrait image to obtain a feature fusion image is specifically: channel splicing of the portrait mask image and the original portrait image is performed to obtain a feature fusion image with the sum of the channels of the two.

[0114] In S1032, the feature fusion image is input into at least two depth-wise separable convolutional layers connected in series for processing, and an image restoration feature image is output.

[0115] In this embodiment, the two serially connected depthwise separable convolutional layers are used to extract feature information of the feature fused image.

[0116] Preferably, the feature fusion image is input into at least two depth-separable convolution layers connected in series for processing, and an image restoration feature image is output. Specifically, the feature fusion image is input into two depth-separable convolution layers connected in series for processing, and an image restoration feature image is output. Specifically, the feature fusion image is subjected to two depth-separable convolution processes in sequence, that is, the feature fusion image is subjected to a convolution process of depth convolution-point convolution-depth convolution-point convolution, and the obtained image restoration feature image has the same size as the feature fusion image.

[0117] It should be understood that the reason why two serially connected depth-wise separable convolutional layers are preferred is to consider the balance between the subsequent portrait segmentation accuracy and the consumption of computing resources.

[0118] In S1033 , the image restoration feature image and the portrait mask image are feature fused to obtain an image restoration mask image.

[0119] In this embodiment, to avoid losing some feature information of the portrait mask image in step S1032, the inpainted feature image needs to be fused with the portrait mask image. The inpainted feature image and the portrait mask image have the same size. The inpainted feature image and the portrait mask image are fused to obtain the inpainted mask image. Specifically, the inpainted feature image and the portrait mask image are channel-joined to obtain the inpainted mask image with the sum of the channels of the two.

[0120] In this embodiment, the image restoration module can be used to obtain the image restoration mask image that has more feature information (especially feature information related to the original portrait image) than the portrait mask image, so as to subsequently improve the accuracy of portrait segmentation.

[0121] Figure 10 FIG1 shows a flowchart of the implementation of the method provided in the sixth embodiment of the present application. Figure 10 , relative to any of the above embodiments, the method provided in this embodiment includes S10-a to S10-e, which are described in detail as follows:

[0122] Furthermore, before inputting the original portrait image into the portrait segmentation network for processing and outputting the portrait mask image, the portrait segmentation method further includes:

[0123] In this embodiment, generally, steps S10-a to S10-e of this embodiment can be performed on a training device other than the terminal device. The portrait segmentation network is trained through steps S10-a to S10-e on the training device, and then the trained portrait segmentation network is sent to the terminal device, so that the terminal device is equipped with the portrait segmentation network and operates normally. The portrait segmentation network of this embodiment has low requirements on the computing power of the carrying device.

[0124] In S10 - a , a training image set is obtained.

[0125] In this embodiment, the training image set includes a plurality of training images, and the training images contain portraits. Generally, the training images contain the portrait of the target object and background other than the portrait.

[0126] In one possible implementation, in order to improve the generalization of the portrait segmentation network, data augmentation is performed when obtaining a training image set. Specifically, a training original image is obtained; a deformation enhancement process is performed on the training original image to obtain a training deformed image; a texture enhancement process is performed on the training original image to obtain a training texture image; and the training original image, the training deformed image, and the training texture image are all identified as the training images in the training image set. The deformation enhancement process includes: random flipping of the image up and down, front and back, random scaling of the image, random cropping of the image, and random rotation of the image. It should be understood that the training deformed image after the deformation enhancement process has the same size as the training original image. After operations that affect the image size, such as random scaling or random cropping of the image, the image size needs to be adjusted to keep the training deformed image consistent with the size of the training original image. The texture enhancement processing includes: randomly adding weather effects to the image (such as rain, fog, sunny, and cloudy), randomly adjusting the image color, and randomly adjusting the image contrast, etc.; it should be understood that the texture enhancement processing should not affect the portrait area of ​​the original training image, that is, the above-mentioned texture enhancement processing is only applied to the background area of ​​the original training image, and the texture enhancement processing is achieved by changing the pixel values ​​of the background area in the original training image.

[0127] In S10-b, the training images are configured with associated mask ground truth images.

[0128] In this embodiment, in order to train the portrait segmentation network, it is necessary to perform data annotation on each training image in the training image set, that is, by marking the portrait area and the background area on each training image.

[0129] In one possible implementation, preferably, the above-mentioned training image is configured with an associated mask true value image, and the specific implementation includes: in the training image, the value of each pixel in the marked portrait area is set to 255, and the value of each pixel in the marked background area is set to 0, to obtain the mask true value image, and the mask true value image is associated with the training image.

[0130] In another possible implementation, optionally, a mask true value image is configured for each training image in the training image set to obtain multiple mask true value images corresponding one-to-one to the multiple training images. Specifically, one of the training images is taken as an example. In the training image, the value of each pixel in the marked portrait area is set to 0, and the value of each pixel in the marked background area is set to 255 to obtain the mask true value image, which is associated with the training image.

[0131] In S10 - c , a plurality of training images are input into a portrait segmentation network for processing, and prediction mask images corresponding one-to-one to the plurality of training images are output.

[0132] In this embodiment, since the implementation of S10-c is the same as the above Figure 1 The implementation method of S102 in the described embodiments is exactly the same. For detailed explanation, please refer to the relevant description of S102, which will not be repeated here.

[0133] In S10-d, the network loss value of the portrait segmentation network is calculated based on the predicted mask image and the mask ground truth image.

[0134] In this embodiment, the predicted mask image is any one of the aforementioned predicted mask images, and the true mask image is any one of the aforementioned true mask images.

[0135] In this embodiment, one of the training images is used as an example for illustration, and the training image corresponds to the predicted mask image and the mask true value image. The training image is used as input, the predicted mask image A is used as the predicted value (the output of the portrait segmentation network), and the mask true value image is used as the true value label (the ideal output of the portrait segmentation network). The portrait segmentation network is trained so that when the portrait segmentation network takes the training image as input, the output is as close as possible to the mask true value image. Specifically, based on the difference between the predicted mask image and the mask true value image, the network loss value of the portrait segmentation network is calculated, so that the weight parameters of the portrait segmentation network are subsequently adjusted according to the network loss value, so that when the adjusted portrait segmentation network takes the training image as input, the output is as close as possible to the mask true value image.

[0136] In one possible implementation, the network loss value may be calculated by calculating the cross entropy loss value of the portrait segmentation network according to a cross entropy loss function, and identifying the cross entropy loss value as the network loss value; the cross entropy loss function is as follows:

[0137] y ij =labels(t ij );

[0138] p ij =sigmoid(logout ij );

[0139] loss ij =-[y ij ×lnp ij +(1-y ij )×ln(1-p ij )];

[0140]

[0141] Among them, t ijis the pixel value of the pixel in the i-th row and j-th column of the mask true value image; y ij is the probability value of the pixel in the i-th row and j-th column of the mask truth image being a portrait pixel; out ij is the pixel value of the pixel in the i-th row and j-th column of the predicted mask image; p ij is the probability value of the pixel in the i-th row and j-th column of the predicted mask image A being a portrait pixel; loss ij is the deviation value corresponding to the pixel in the i-th row and j-th column of the predicted mask image; H and W are the height and width of the predicted mask image (also representing the image size); loss _ c is the cross entropy loss value.

[0142] In another possible implementation, the above calculation of the network loss value may be specifically: calculating the alpha loss value of the portrait segmentation network according to the alpha loss function, and identifying the alpha loss value as the network loss value; the alpha loss function is as follows:

[0143] y ij =labels(t ij )

[0144] p ij =sigmoid(logout ij )

[0145]

[0146] Among them, t ij is the pixel value of the pixel in the i-th row and j-th column of the mask true value image; y ij is the probability value of the pixel in the i-th row and j-th column of the mask truth image being a portrait pixel; out ij is the pixel value of the pixel in the i-th row and j-th column of the predicted mask image; p ij is the probability value of the pixel in the i-th row and j-th column of the predicted mask image being a portrait pixel; H and W are the height and width of the predicted mask image (also representing the image size); loss _ α is the alpha loss value.

[0147] It should be understood that the ij And the p ij The value range of is [0,1]. Specifically, if the mask true value image is configured as above, the value of each pixel in the portrait area is 255, and the value of each pixel in the background area is 0, then the labels function is specifically y ij =t ij / 255; if the mask true value image is configured as above, the pixel values ​​in the portrait area are set to 0, and the pixel values ​​in the background area are set to 255, then the labels function is specifically y ij =(255-t ij ) / 255. The sigmoid function can be referred to the related description in the prior art, which will not be repeated here.

[0148] In S10-e, the weight parameters of the portrait segmentation network are adjusted according to the network loss value.

[0149] In this embodiment, during the training of the portrait segmentation network, each training image corresponds to a network loss value. Taking one of the training images as an example, the training image corresponds to the network loss value, and the network loss value contains output correction information of the portrait segmentation network. The weight parameters of the portrait segmentation network are adjusted multiple times according to the network loss values ​​corresponding to multiple training images, so that the portrait segmentation network after adjusting the weight parameters can be used in subsequent portrait segmentation, thereby improving the accuracy of portrait segmentation.

[0150] In this embodiment, the portrait segmentation network is trained using the training image set. Specifically, the network loss value is used to adjust the weight parameters of the portrait segmentation network to improve the output accuracy of the portrait segmentation network, so as to improve the accuracy of subsequent portrait segmentation.

[0151] Figure 11 FIG1 shows a flowchart of the implementation of the method provided in the seventh embodiment of the present application. Figure 11 , relative to the above Figure 10 In the embodiment described, the method 10-d provided in this embodiment includes S10-d1 to S10-d4, which are described in detail as follows:

[0152] Furthermore, the network loss value of the portrait segmentation network is calculated based on the predicted mask image and the mask true value image, including:

[0153] In this embodiment, the network loss value includes an edge loss value, which is used to describe the deviation between edge information of the portrait area in the predicted mask image and the true mask image.

[0154] In S10-d1, a predicted probability image corresponding to the predicted mask image is obtained according to the softmax function.

[0155] In this embodiment, the softmax function is specifically: y_p ij =softmax(out ij ); where y_p ijis the pixel value of the pixel in the i-th row and j-th column of the predicted probability image, and is also the probability value of the pixel in the i-th row and j-th column of the predicted mask image being a portrait pixel; out ij is the pixel value of the pixel in the i-th row and j-th column of the predicted mask image; the softmax function can be referred to the relevant description in the prior art and will not be repeated here. It should be understood that y_p ij The value range is [0,1].

[0156] In S10 - d2 , the mask true value image is dilated and eroded to obtain a dilated true value image and an eroded true value image corresponding to the mask true value image.

[0157] In this embodiment, in order to calculate the edge loss value, it is necessary to extract the edge information within the masked true value image. The above-mentioned dilation and erosion processing of the masked true value image to obtain the dilated true value image and the eroded true value image can be preferably specifically: using the standard dilation and erosion operations of TensorFlow with kernel parameters of [5,5,2], performing standard dilation processing on the masked true value image to obtain the dilated true value image, and performing erosion processing on the masked true value image to obtain the eroded true value image. The specific implementation can be referred to the relevant description in the prior art and will not be repeated here.

[0158] In S10 - d3 , a boundary truth image is generated according to the dilated truth image and the eroded truth image.

[0159] In this embodiment, the boundary true value image is generated according to the dilated true value image and the eroded true value image. Specifically, the dilated true value image and the eroded true value image are subtracted to obtain the boundary true value image. That is, the boundary true value image is the result of morphological gradient processing (dilation result and erosion result subtraction) of the mask true value image. For details, see Figure 12 , Figure 12 Schematic diagram showing the mask true value image and the boundary true value image provided by this embodiment, Figure 12 The white part in the mask true value image shown is the portrait area. Figure 12 The white part in the boundary truth image shown is the edge of the portrait area.

[0160] In S10-d4, each pixel value contained in the predicted probability image, each pixel value contained in the mask true value image, and each pixel value contained in the boundary true value image are input into a preset edge loss function for calculation to obtain the edge loss value of the portrait segmentation network.

[0161] In this embodiment, the edge loss function is as follows:

[0162] loss_edge=∑{-edge_t ij *[y_tij *ln(y_p ij )+(1-y_t ij )*ln(1-y_p ij )]}

[0163] Among them, the loss_edge is the edge loss value; the edge_t ij is the pixel value of the i-th row and j-th column in the boundary truth image, that is, the probability value of the pixel of the i-th row and j-th column in the mask truth image being the edge of the portrait area; ij is the probability value of the pixel in the i-th row and j-th column of the mask true value image being a portrait pixel; ij is the probability value of the pixel in the i-th row and j-th column in the predicted probability image being a portrait pixel. ij ,y_t ij and edge_t ij The value range of y_t is [0,1]; ij The calculation of y can refer to the y described in S10-d above. ij The calculation of is not described here.

[0164] It should be understood that in this embodiment, the network loss value includes the edge loss value: exemplarily, the network loss value is the sum of the edge loss value and the cross entropy loss value; exemplarily, the network loss value is the sum of the edge loss value and the alpha loss value. Preferably, the network loss value is the sum of the edge loss value, the cross entropy loss value, and the edge loss values ​​before each decoding unit; refer to Figure 7 , the edge loss value before each of the above decoding units is also the loss value of the Mth upsampled feature image. The edge loss value of the Mth upsampled feature image is recorded as the Mth edge loss value. The Mth edge loss value is used to describe the edge loss of the portion of the portrait segmentation network before the M-1th decoding unit. The calculation of the above-mentioned Mth edge loss can refer to the calculation of the above-mentioned edge loss value in this embodiment. It should be noted that when calculating the Mth edge loss, the training image is used as input, and the Mth upsampled feature image corresponding to the training image is subjected to a channel adjustment convolution process and a point convolution process to adjust the number of channels and size of the Mth upsampled feature image corresponding to the training image, so as to obtain the Mth loss image with the same number of channels and size as the portrait mask image; the predicted mask image is replaced by the Mth loss image corresponding to the training image, and the Mth edge loss is calculated with reference to the calculation steps of the edge loss value in this embodiment.

[0165] In this embodiment, by designing the edge loss function, when adjusting the weight parameters of the portrait segmentation network according to the network loss value, the network loss value includes the loss of the edge of the portrait area in the predicted mask image (the edge loss value), which can improve the effect of adjusting the weight parameters of the portrait segmentation network and further improve the output accuracy of the portrait segmentation network.

[0166] Figure 13 A schematic diagram of an application scenario provided by an embodiment of the present application is shown, and details are as follows:

[0167] See also Figure 13 ,contrast Figure 3 , Figure 13 The input size of the portrait segmentation network shown in is 304*304 and the channel is 3, that is, the size of the original portrait image is 304*304 and the channel is 3, representing the three channels of RGB; the size of the portrait mask image is 304*304 and the channel is 2, representing the black and white channels. In one channel, the black is the portrait area and the white is the background area, and in the other channel, the black is the background area and the white is the portrait area. The network loss value is the sum of the above edge loss value, the above cross entropy loss value and the edge loss value of each loss image shown, that is Figure 13 The final output of the portrait segmentation network, the edge loss values ​​of the first loss image, the second loss image, the third loss image, and the fourth loss image, and the sum of the cross entropy loss values ​​of the final output of the portrait segmentation network are shown. After training the portrait segmentation network according to the network loss, a test image is collected to test the portrait segmentation network. During the test, the average intersection-and-union ratio (miou) of the predicted mask image and the true mask image is specifically 0.94946, which can represent the effect of the portrait segmentation network. The formula for calculating the average intersection-and-union ratio miou of the predicted mask image and the true mask image is:

[0168]

[0169] Among them, TP is the true positive, that is, the number of pixels whose portrait pixels in the predicted mask image are at the same position as the portrait pixels in the true mask image; FP is the false positive, that is, the number of pixels whose portrait pixels in the predicted mask image are at the same position as the background pixels in the true mask image; FN is the false negative, that is, the number of pixels whose background pixels in the predicted mask image are at the same position as the portrait pixels in the true mask image; n is the number of test images; MIOU is the average of the intersection-over-union (IoU) of each test image, which is also the above-mentioned average IoU.

[0170] Corresponding to the method described in the above embodiment, Figure 14 A schematic structural diagram of a device provided in one embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0171] Reference Figure 14 , the portrait segmentation device includes:

[0172] The original portrait image acquisition unit is used to acquire the original portrait image, wherein the original portrait image contains a portrait;

[0173] A portrait segmentation network unit is used to input the original portrait image into the portrait segmentation network for processing and output a portrait mask image; wherein the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module;

[0174] An image restoration unit is configured to input the portrait mask image and the original portrait image into the image restoration module for processing, and output an image restoration mask image;

[0175] Portrait segmentation ternary is used to divide the original portrait image into a portrait image and a background image according to the image restoration mask image.

[0176] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0178] Figure 15 FIG. 1 shows a schematic diagram of the structure of a terminal device provided by an embodiment of the present application. Figure 15 As shown, the terminal device 15 of this embodiment includes: at least one processor 150 ( Figure 15 Only one is shown in the figure) a processor, a memory 151, and a computer program 152 stored in the memory 151 and executable on the at least one processor 150. When the processor 150 executes the computer program 152, the steps of any of the above-mentioned method embodiments are implemented.

[0179] The terminal device 15 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device 15 is preferably a mobile terminal. The terminal device may include, but is not limited to, a processor 150 and a memory 151. It will be understood by those skilled in the art that Figure 15 It is merely an example of the terminal device 15 and does not constitute a limitation on the terminal device 15. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0180] The processor 150 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0181] In some embodiments, the memory 151 may be an internal storage unit of the terminal device 15, such as a hard disk or memory of the terminal device 15. In other embodiments, the memory 151 may also be an external storage device of the terminal device 15, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 15. Furthermore, the memory 151 may also include both an internal storage unit of the terminal device 15 and an external storage device. The memory 151 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 151 may also be used to temporarily store data that has been output or is about to be output.

[0182] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in the above-mentioned various method embodiments.

[0183] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0184] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process of the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, mobile hard drive, magnetic disk, or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0185] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0186] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0187] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0188] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A portrait segmentation method, characterized in that: include: Acquire an original portrait image, wherein the original portrait image contains a portrait; Input the original portrait image into a portrait segmentation network for processing, and output a portrait mask image; wherein the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module; the encoding module uses a depthwise separable convolution layer to extract feature information of the original portrait image; the decoding module includes a plurality of decoding units, each decoding unit includes an upsampling layer, a feature fusion layer, and a conversion unit, the conversion unit includes at least one depthwise separable convolution layer, at least one adjustment channel convolution layer, and at least one feature fusion block; the depthwise separable convolution layer and the adjustment channel convolution layer in the conversion unit are arranged in parallel, and the outputs of the two are fused through the feature fusion block to obtain the M-1th upsampled feature image; M is an integer and M is greater than 0; Inputting the portrait mask image and the original portrait image into an image restoration module for processing, and outputting an image restoration mask image; The original portrait image is divided into a portrait image and a background image according to the image restoration mask image.

2. The method according to claim 1, wherein The step of inputting the original portrait image into a portrait segmentation network for processing and outputting a portrait mask image comprises: Inputting the original portrait image into the encoding module, performing multiple convolution processes on the original portrait image using a depthwise separable convolution algorithm corresponding to the encoding module, and outputting N downsampled feature images, where N is an integer greater than 2; Input the Nth down-sampled feature image into the pyramid pooling module for processing, and output the Nth up-sampled feature image; Inputting the Nth up-sampled feature image and N-1 down-sampled feature images of the N down-sampled feature images other than the Nth down-sampled feature image into the decoding module for processing, and outputting a first up-sampled feature image; wherein the decoding module includes N-1 decoding units, and the M-1th decoding unit is used to determine the M-1th up-sampled feature image based on the Mth up-sampled feature image and the M-1th down-sampled feature image, where M is less than or equal to N; Perform image mask processing on the first up-sampled feature image to obtain a portrait mask image.

3. The method according to claim 2, wherein Inputting the Nth down-sampled feature image into the pyramid pooling module for processing and outputting the Nth up-sampled feature image includes: Performing dilated convolution processing on the Nth downsampled feature image using a plurality of preset convolution kernels of different scales to obtain a plurality of dilated convolution feature images, wherein the plurality of dilated convolution feature images correspond one-to-one to the plurality of convolution kernels of different scales; Performing convolution processing on each of the dilated convolution feature images respectively to obtain a plurality of feature images to be fused that correspond one-to-one to the plurality of dilated convolution feature images; Performing feature fusion on the multiple feature images to be fused to obtain a pooled feature image; Channel-joining is performed on the pooled feature image and the Nth down-sampled feature image to obtain an Nth up-sampled feature image.

4. The method according to claim 3, wherein The determining the M-1th up-sampled feature image according to the Mth up-sampled feature image and the M-1th down-sampled feature image includes: Inputting the Mth upsampled feature image into the upsampling layer of the M-1th decoding unit, performing upsampled feature mapping on the Mth upsampled feature image, and outputting the M-1th initial feature image; Inputting the M-1th initial feature image and the M-1th downsampled feature image into the feature fusion layer of the M-1th decoding unit, performing feature fusion on the M-1th initial feature image and the M-1th downsampled feature image to obtain an M-1th upsampled feature fusion image; The M-1th up-sampled feature fusion image is input into the conversion unit of the M-1th decoding unit for processing, and the M-1th up-sampled feature image is output.

5. The method according to claim 4, wherein The image restoration module includes at least two depth-wise separable convolutional layers connected in series. The portrait mask image and the original portrait image are input into the image restoration module for processing to obtain an image restoration mask image, including: Performing feature fusion on the portrait mask image and the portrait original image to obtain a feature fused image; Inputting the feature fusion image into the at least two serially connected depthwise separable convolutional layers for processing, and outputting an image restoration feature image; The image restoration feature image is fused with the portrait mask image to obtain an image restoration mask image.

6. The method according to any one of claims 1 to 5, wherein: Before inputting the original portrait image into the portrait segmentation network for processing and outputting the portrait mask image, the method further includes: Acquire a training image set, the training image set comprising a plurality of training images, wherein the training images contain human portraits; the training images are configured with associated masked true value images; Inputting the plurality of training images into the portrait segmentation network for processing, and outputting prediction mask images corresponding one-to-one to the plurality of training images; Calculating a network loss value of the portrait segmentation network according to the predicted mask image and the mask true value image, wherein the predicted mask image is any one of the plurality of predicted mask images, and the mask true value image is any one of the plurality of mask true value images; The weight parameters of the portrait segmentation network are adjusted according to the network loss value.

7. The method according to claim 6, wherein The network loss value includes an edge loss value; and the step of calculating the network loss value of the portrait segmentation network according to the predicted mask image and the mask true value image includes: Obtaining a predicted probability image corresponding to the predicted mask image according to a softmax function; Performing dilation and erosion processing on the mask true value image to obtain a dilated true value image and an eroded true value image corresponding to the mask true value image; Generate a boundary truth image according to the dilated truth image and the eroded truth image; Inputting each pixel value included in the predicted probability image, each pixel value included in the mask true value image, and each pixel value included in the boundary true value image into a preset edge loss function for calculation to obtain an edge loss value of the portrait segmentation network; Wherein, the edge loss function is: loss _ edge=∑{-edge_t ij *[y_t ij *ln(y_p ij )+(1-y_t ij )*ln(1-y_p ij )]} The loss_edge is the edge loss value; the edge_t ij is the pixel value of the i-th row and j-th column in the boundary truth image; ij is the pixel value of the i-th row and j-th column in the mask true value image; ij is the pixel value of the i-th row and j-th column in the predicted probability image.

8. A portrait segmentation device, characterized in that: include: A portrait original image acquisition unit, configured to acquire a portrait original image, wherein the portrait original image contains a portrait; A portrait segmentation network unit is configured to input the original portrait image into a portrait segmentation network for processing and output a portrait mask image; wherein the portrait segmentation network includes an encoding module, a pyramid pooling module, and a decoding module; the encoding module uses a depthwise separable convolutional layer to extract feature information of the original portrait image; the decoding module includes a plurality of decoding units, each decoding unit includes an upsampling layer, a feature fusion layer, and a conversion unit, the conversion unit includes at least one depthwise separable convolutional layer, at least one adjustment channel convolutional layer, and at least one feature fusion block; the depthwise separable convolutional layer and the adjustment channel convolutional layer in the conversion unit are arranged in parallel, and the outputs of the two are fused through the feature fusion block to obtain the M-1th upsampled feature image; where M is an integer and M is greater than 0; An image restoration unit, configured to input the portrait mask image and the original portrait image into an image restoration module for processing, and output an image restoration mask image; The portrait segmentation unit is used to divide the original portrait image into a portrait image and a background image according to the image restoration mask image.

9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images