Portrait segmentation model generation method, portrait segmentation model, and portrait segmentation method

Abstract

The application discloses a portrait segmentation model generation method, a portrait segmentation model and a portrait segmentation method. The portrait segmentation model generation method comprises the following steps: inputting a training image in a preset training sample into a preset network model, determining a predicted portrait mask graph corresponding to the training image through the preset network model; obtaining a portrait edge in the predicted portrait mask graph, and determining a weight graph corresponding to the predicted portrait mask graph based on the portrait edge; and training the preset network model based on the weight graph, the predicted portrait mask graph and a target portrait mask graph corresponding to the training image, so as to obtain the portrait segmentation model. The weight value of each pixel point in the predicted portrait mask graph in the training process is determined by the portrait edge in the predicted portrait mask, and the preset network model is trained based on the weight value, so that the recognition effect of the portrait edge corresponding to the portrait segmentation model obtained by training can be improved, and the accuracy of the portrait segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for generating a portrait segmentation model, a portrait segmentation model, and a portrait segmentation method. Background Technology

[0002] Dual cameras are increasingly being used in mobile devices, and most existing mobile devices with dual cameras now feature portrait mode. Current portrait mode functions generally rely on segmenting the subject into a portrait region using a portrait segmentation model before previewing and blurring. However, the portrait segmentation models currently in use generally suffer from low accuracy. Summary of the Invention

[0003] The technical problem to be solved by this application is to provide a method for generating a human image segmentation model, a human image segmentation model, and a human image segmentation method, in order to address the shortcomings of the existing technology.

[0004] To address the aforementioned technical problems, the first aspect of this application provides a method for generating a human portrait segmentation model, the method comprising:

[0005] The training images from the preset training samples are input into the preset network model, and the preset network model determines the predicted human face mask corresponding to the training images, wherein the training images include human face regions;

[0006] Obtain the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges;

[0007] The preset network model is trained based on the weight map, the predicted human face mask map, and the target human face mask map corresponding to the training image to obtain a human face segmentation model.

[0008] The method for generating the portrait segmentation model includes a weight map whose image size is the same as the image size of the predicted portrait mask map, and the weight map includes the weight values ​​corresponding to each mask pixel in the predicted portrait mask map.

[0009] The method for generating the portrait segmentation model, wherein determining the weight map corresponding to the predicted portrait mask map based on the portrait edges specifically includes:

[0010] For each mask pixel in the predicted human face mask image, the distance of the mask pixel to the edge of the human face is determined, and the distance is used as the weight value of the mask pixel to obtain the weight value corresponding to each mask pixel in the predicted human face mask image.

[0011] The weight map corresponding to the predicted human face mask is obtained by replacing the pixel value of each mask pixel with the weight value of each mask pixel in the predicted human face mask map.

[0012] The method for generating the portrait segmentation model, wherein training the preset network model based on the weight map, the predicted portrait mask map, and the target portrait mask map corresponding to the training image to obtain the portrait segmentation model specifically includes:

[0013] For each mask pixel in the predicted portrait mask image, a candidate loss value corresponding to the mask pixel is determined based on the predicted portrait mask image and the target portrait mask image corresponding to the training image, so as to obtain the candidate loss value corresponding to each mask pixel in the predicted portrait mask image.

[0014] Based on the weight map, the candidate loss values ​​corresponding to each mask pixel are weighted to obtain the target loss value corresponding to the training image.

[0015] The preset network model is trained based on the target loss value to obtain a human image segmentation model.

[0016] The method for generating the human image segmentation model includes a preset network model comprising several factorized convolutional modules. Each factorized convolutional module comprises several first convolutional layers, several second convolutional layers, and a fusion layer. The ratio of the width to the height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to the height of the second convolutional kernel. The input of the fusion layer includes the output of the first or second convolutional layer connected to it and the input of the factorized convolutional modules.

[0017] The method for generating the human portrait segmentation model, wherein the width of the first convolutional layer or the width of the second convolutional layer is equal to 1.

[0018] The method for generating the human image segmentation model includes a factorized convolutional module that further comprises two third convolutional layers, several first convolutional layers and several second convolutional layers that are alternately cascaded between the two third convolutional layers, and the input of the fusion layer includes the output of the last third convolutional layer and the input of the first third convolutional layer.

[0019] The method for generating the human image segmentation model includes a first convolutional module and a second convolutional module in the preset network model. The first convolutional module and the second convolutional module are set in series. A portion of the factorized convolutional modules are cascaded between the first convolutional module and the second convolutional module, and another portion of the factorized convolutional modules are cascaded after the second convolutional module.

[0020] The second aspect of this embodiment provides a portrait segmentation method, applied to a portrait segmentation model generated by the portrait segmentation model generation method described above. The portrait segmentation method specifically includes:

[0021] Input the image to be segmented into the portrait segmentation model;

[0022] The portrait segmentation model is used to determine the portrait mask image corresponding to the image to be segmented.

[0023] The portrait segmentation method, wherein after determining the portrait mask image corresponding to the image to be segmented using the portrait segmentation model, the method further includes:

[0024] Based on the portrait mask, the portrait region and background region of the image to be segmented are determined, wherein the background region is the image region in the image to be segmented other than the portrait region.

[0025] The portrait segmentation method, after determining the portrait region and background region of the image to be segmented based on the portrait mask, further includes:

[0026] Based on the portrait region and the background region, the image to be segmented is segmented to obtain the portrait image and / or background image corresponding to the image to be segmented.

[0027] The portrait segmentation method, wherein before inputting the image to be segmented into the portrait segmentation model, the method further includes:

[0028] The scene image is acquired by the imaging module, and the image size of the scene image is adjusted to obtain the image to be segmented, wherein the image size of the image to be segmented is smaller than the image size of the scene image.

[0029] The third aspect of this embodiment provides a portrait segmentation model, which is trained using the portrait segmentation model generation method described above. The portrait segmentation model includes a first convolutional module, a first feature extraction module, a second convolutional module, and a second feature extraction module cascaded in sequence. The first feature extraction module includes a first preset number of factorized convolutional modules, and the second feature extraction module includes a second preset number of factorized convolutional modules. The factorized convolutional module includes a plurality of first convolutional layers, a plurality of second convolutional layers, and a fusion layer. The ratio of the width to the height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to the height of the second convolutional kernel. The input of the fusion layer includes the output of the first or second convolutional layer connected to it and the input of the factorized convolutional modules.

[0030] The human portrait segmentation model, wherein the width of the first convolutional layer or the width of the second convolutional layer is equal to 1.

[0031] The human portrait segmentation model includes a factorized convolutional module that further comprises two third convolutional layers, several first convolutional layers and several second convolutional layers that are alternately cascaded between the two third convolutional layers, and the input of the fusion layer includes the output of the last third convolutional layer and the input of the first third convolutional layer.

[0032] The fourth aspect of this embodiment provides an apparatus for generating a human portrait segmentation model, the apparatus comprising:

[0033] The determination module is used to input training images from preset training samples into a preset network model, and determine the predicted human face mask corresponding to the training images through the preset network model, wherein the training images include human face regions;

[0034] The acquisition module is used to acquire the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges;

[0035] The training module is used to train the preset network model based on the weight map, the predicted portrait mask map, and the target portrait mask map corresponding to the training image, so as to obtain a portrait segmentation model.

[0036] A fifth aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the method for generating a portrait segmentation model as described above, or to implement the steps in the portrait segmentation method as described above.

[0037] A sixth aspect of this application provides a terminal device, which includes: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0038] The communication bus enables communication between the processor and the memory;

[0039] When the processor executes the computer-readable program, it implements the steps in the method for generating a human image segmentation model as described above, or implements the steps in the human image segmentation method as described above.

[0040] Beneficial Effects: Compared with existing technologies, this application provides a method for generating a portrait segmentation model, a portrait segmentation model, and a portrait segmentation method. The method for generating the portrait segmentation model includes inputting training images from preset training samples into a preset network model, determining a predicted portrait mask corresponding to the training images through the preset network model; obtaining portrait edges in the predicted portrait mask, and determining a weight map corresponding to the predicted portrait mask based on the portrait edges; training the preset network model based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training images to obtain a portrait segmentation model. This application determines the weight values ​​of each pixel in the predicted portrait mask during the training process by using the portrait edges in the predicted portrait mask. By training the preset network model based on these weight values, the recognition effect of the portrait edges corresponding to the trained portrait segmentation model can be improved, thereby improving the accuracy of portrait segmentation. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the method for generating the human image segmentation model provided in this application.

[0043] Figure 2 A schematic diagram of the pre-defined network model for the image segmentation model generation method provided in this application.

[0044] Figure 3 The structural principle diagram of the factorized convolution module in the preset network model of the image segmentation model generation method provided in this application.

[0045] Figure 4 This is a schematic diagram illustrating the structural principle of a factorized convolution module in a preset network model of the image segmentation model generation method provided in this application.

[0046] Figure 5 Example image of the human image edge in the method for generating the human image segmentation model provided in this application.

[0047] Figure 6 An example diagram of the weight map for the generation method of the portrait segmentation model provided in this application.

[0048] Figure 7 A flowchart of the portrait segmentation method provided for this application.

[0049] Figure 8 A schematic diagram of the structural principle of the device for generating the human portrait segmentation model provided in this application.

[0050] Figure 9 A schematic diagram of the portrait mask in the image blurring method provided in this application.

[0051] Figure 10 A schematic diagram of the disparity map in the image blurring method provided in this application.

[0052] Figure 11 A schematic diagram of a preview image in the image blurring method provided in this application.

[0053] Figure 12 A schematic diagram of the terminal device provided in this application. Detailed Implementation

[0054] This application provides a method for generating a portrait segmentation model, a portrait segmentation model, and a portrait segmentation method. To make the purpose, technical solution, and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0057] In specific implementations, the terminal devices described in the embodiments of this application include, but are not limited to, other portable devices such as mobile phones, laptops, or tablets with touch-sensitive surfaces (e.g., touch displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touch displays and / or touchpads).

[0058] The following discussion describes terminal devices that include displays and touch-sensitive surfaces. However, it should be understood that terminal devices may also include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0059] The terminal device supports a variety of applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, video conferencing applications, disc burning applications, spreadsheet applications, game applications, telephone applications, video conferencing applications, email applications, instant messaging applications, exercise support applications, photo management applications, digital camera applications, digital camcorder applications, web browsing applications, digital music player applications, and / or digital video playback applications, etc.

[0060] Various applications that can run on a terminal device can use at least one common physical user interface device, such as a touch-sensitive surface. First or more functions of the touch-sensitive surface and the corresponding information displayed on the terminal can be adjusted and / or changed between and / or within applications. In this way, the terminal's common physical framework (e.g., the touch-sensitive surface) can support various applications with user interfaces that are intuitive and transparent to the user.

[0061] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0062] The inventors discovered through research that dual cameras are increasingly being used in mobile devices, and existing mobile devices equipped with dual cameras generally include portrait blurring functionality. Current portrait blurring functions are generally based on segmenting the image into a portrait region using a portrait segmentation model before previewing and blurring; however, the portrait segmentation models currently in use generally suffer from low accuracy.

[0063] To address the aforementioned issues, in this embodiment, training images from a preset training sample are input into a preset network model. The preset network model determines a predicted portrait mask corresponding to the training image. Portrait edges are obtained from the predicted portrait mask, and a weight map corresponding to the predicted portrait mask is determined based on these edges. The preset network model is then trained based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training image to obtain a portrait segmentation model. This application determines the weight values ​​of each pixel in the predicted portrait mask during training by using the portrait edges in the predicted portrait mask. By training the preset network model based on these weight values, the recognition effect of the trained portrait segmentation model corresponding to the portrait edges can be improved, thereby increasing the accuracy of portrait segmentation.

[0064] The application content will be further explained below with reference to the accompanying drawings and through the description of the embodiments.

[0065] This embodiment provides a method for generating a human face segmentation model, such as... Figure 1 As shown, the method includes:

[0066] S10. Input the training images from the preset training samples into the preset network model, and determine the predicted human face mask corresponding to the training images through the preset network model.

[0067] Specifically, the preset training samples include several training image groups. Each training image group includes a training image and a corresponding target portrait mask. The training image includes a portrait region, and the target portrait mask is the mask image corresponding to the portrait region in the training image. In the target portrait mask, the pixel value of the pixel corresponding to the pixel in the portrait region of the training image is a first preset value, and the pixel value of the pixel corresponding to the pixel outside the portrait region of the training image is a second preset value. Furthermore, the target portrait mask is used as the basis for judging the predicted portrait mask output by the preset network model, so as to determine the accuracy of the predicted portrait mask output by the preset network model. For example, the preset training sample data may include a preset number (e.g., 100,000) of training image groups, each including a training image and a corresponding target portrait mask. When training the segmentation network model based on the training samples, data augmentation operations such as 90-degree rotation, contrast stretching, brightness adjustment, noise addition, saturation adjustment, and random cropping can be performed on the training samples to improve the diversity of the training samples.

[0068] The input to the preset network model is the training image, and the output is the predicted human face mask. For example... Figure 2As shown, the preset network model includes a first convolutional module 100, a first feature extraction module 200, a second convolutional module 300, and a second feature extraction module 400, all cascaded together. Correspondingly, the step of inputting training images from preset training samples into the preset network model and determining the predicted portrait mask corresponding to the training images through the preset network model specifically includes:

[0069] The training images in the preset training samples are input into the first convolution module, and the first feature map corresponding to the image to be processed is output through the first convolution module.

[0070] The first feature map is input into the first feature extraction module, and the second feature map corresponding to the image to be processed is output by the first feature extraction module.

[0071] The second feature map is input into the second convolution module, and the third feature map corresponding to the image to be processed is output through the second convolution module;

[0072] The third feature map is input into the second feature extraction module, and the second feature extraction module outputs the target feature map corresponding to the image to be processed. Based on the target feature map, the predicted human face mask map corresponding to the training image is determined.

[0073] Specifically, the first convolutional module 100 includes a convolutional layer A and a convolutional unit a, which are cascaded. The convolutional kernel of convolutional layer A can be 3*3 with a stride of 2. Convolutional unit a includes a convolutional layer, where the convolutional kernel can be 1*1 with a stride of 1. The number of channels in the output of convolutional unit a is half the number of input and output channels of the convolutional kernel. In one specific implementation, the image to be processed is the input of convolutional layer A, with an image scale of 240*160*3. The image scale of the output of convolutional layer A is 120*80*128. The image scale of the input of convolutional unit a is 120*80*128, and the image scale of the output of convolutional unit a is 120*80*64.

[0074] The second convolutional module 300 includes a convolutional layer B and a convolutional unit b, which are cascaded. The convolutional kernel of convolutional layer B can be 3*3 with a stride of 2. Each convolutional unit includes a convolutional layer, and the convolutional kernel of the convolutional layer in convolutional unit b can be 1*1 with a stride of 1. The number of channels in the output of convolutional unit b is the same as the number of input and output channels of the convolutional kernel. In one specific implementation, the image scale of the input of convolutional layer B is 120*80*64, and the image scale of the output of convolutional layer B is 60*40*256; the image scale of the input of convolutional unit a is 60*40*256, and the image scale of the output of convolutional unit a is 60*40*128.

[0075] The first feature extraction module 200 includes a first preset number of factorized convolutional modules 500, and the second feature extraction module 400 includes a second preset number of factorized convolutional modules 500. The first preset number of factorized convolutional modules in the first feature extraction module are cascaded sequentially, and adjacent factorized convolutional modules are connected by a convolutional layer C. In other words, the first preset number of factorized convolutional modules are arranged sequentially, and adjacent factorized convolutional modules are connected by a convolutional layer C to form the first feature extraction module. Correspondingly, the second preset number of factorized convolutional modules in the second feature extraction module are cascaded sequentially, and adjacent factorized convolutional modules are connected by a convolutional layer C. In other words, the second preset number of factorized convolutional modules are arranged sequentially, and adjacent factorized convolutional modules are connected by a convolutional layer C to form the second feature extraction module. In one implementation of this embodiment, the kernel size of the convolutional layer C can be 3*3, and the stride can be 1.

[0076] The factorized convolution module includes several first convolutional layers, several second convolutional layers 502, and a fusion layer 503. The ratio of the width to height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to height of the second convolutional kernel. The input of the fusion layer includes the output of the first or second convolutional layer connected to it and the input of the factorized convolution module. This embodiment achieves the convolution operation effect by using two convolutional layers with reciprocal kernel sizes, thus reducing the computational load of convolution. This improves the computational speed of the portrait segmentation module, thereby increasing the speed of obtaining the portrait mask and improving the real-time performance of the preview image blurring. Simultaneously, by reducing the computational load of the portrait segmentation model, the hardware requirements for the model are reduced, thus expanding its applicability. In one specific implementation, the width of the convolutional kernel of the first convolutional layer or the width of the convolutional kernel of the second convolutional layer is equal to 1, minimizing the computational load of the portrait segmentation model and improving the speed of determining the portrait mask.

[0077] In one implementation of this embodiment, the number of the plurality of first convolutional layers and the number of the plurality of second convolutional layers in the factorized convolutional module are the same, and as follows: Figure 3 As shown, several first convolutional layers 501 and several second convolutional layers 502 are arranged alternately and cascaded in sequence. The fusion layer 503 is connected to the first target convolutional layer, which is located at the end of the cascaded sequence of the first and second convolutional layers. The input of the second target convolutional layer, which is located at the beginning of the cascaded sequence of the first and second convolutional layers, is the input of the fusion layer. The fusion layer is used to fuse the output of the first target convolutional layer with the input of the second target convolutional layer to obtain the output of the factorized convolutional module.

[0078] Furthermore, in one specific implementation, since the ratio of the width to height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to height of the second convolutional kernel, when the size of the convolutional kernel of the first convolutional layer is denoted as M*K, the size of the convolutional kernel of the second convolutional layer is K*M, where M represents the width of the convolutional kernel of the first convolutional layer, K represents the height of the convolutional kernel of the first convolutional layer, and the width M of the convolutional kernel of the first convolutional layer or the width K of the convolutional kernel of the second convolutional layer is equal to 1. Therefore, the first and second convolutional layers can be equivalent to a reference convolutional layer, wherein the kernel size of the reference convolutional layer is K*K. For example, if the width of the kernel of the first convolutional layer is 1, then the height of the kernel of the second convolutional layer is 1. Accordingly, the size of the kernel of the first convolutional layer is 1*K, and the size of the kernel of the second convolutional layer is K*1. Thus, the convolutional layer with a kernel size of K*K is split into the product of the first convolutional layer with a kernel size of 1*K and the second convolutional layer with a kernel size of K*1.

[0079] Based on this, assuming the kernel of the first convolutional layer is 1*K and the kernel of the second convolutional layer is K*1, then the convolution calculations corresponding to the first and second convolutional layers can be represented as:

[0080]

[0081] Where K0 represents the first convolutional layer with a kernel of 1*K, K1 represents the second convolutional layer with a kernel of K*1, K represents the reference convolutional layer with a kernel of K*K, V represents the output term of the second convolutional layer, and U represents the input term of the first convolutional layer.

[0082] In one implementation of this embodiment, the factorized convolution module further includes two third convolutional layers, with a plurality of first convolutional layers and a plurality of second convolutional layers alternately arranged and cascaded between the two third convolutional layers. The input to the fusion layer includes the output of the last of the two third convolutional layers and the input of the first of the two third convolutional layers. In other words, the input to the factorized convolution module is the input of the first of the two third convolutional layers, and the input of the first of the two third convolutional layers is the input to the fusion layer.

[0083] For example: Figure 4 As shown, the factorized convolution module includes a first convolutional layer A, a first convolutional layer B, a second convolutional layer C, a second convolutional layer D, a third convolutional layer E, a third convolutional layer F, and an adder. The third convolutional layer E, the first convolutional layer A, the second convolutional layer C, the first convolutional layer B, the second convolutional layer D, and the third convolutional layer F are cascaded sequentially. The third convolutional layer F is connected to the adder, and the input of the third convolutional layer E is input to the adder. The convolution of the first convolutional layer A and the first convolutional layer B... The kernel size can be 1*3 for all convolutional layers. The kernel size of the second convolutional layer C and the second convolutional layer D can be 3*1 for all convolutional layers. The kernel size of the third convolutional layer E and the third convolutional layer F can be 3*3 for all convolutional layers. The number of channels for the output items of the first convolutional layer A, the first convolutional layer B, the second convolutional layer C, the second convolutional layer D and the third convolutional layer E is c / 2. The number of channels for the output items of the third convolutional layer F is c. The number of channels for the input items of the third convolutional layer E is c, where c is a positive integer and a multiple of 2.

[0084] S20. Obtain the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges.

[0085] Specifically, the image edge is the region edge of the image area in the predicted image mask image, for example, such as... Figure 5 As shown, the edges of the portrait are Figure 5The edges of the white areas in the image are identified. Therefore, after obtaining the predicted portrait mask, edge recognition can be performed on the mask image to obtain the edges of the portrait regions within the mask, for example, using the Sobel operator, Laplacian operator, or Canny operator. After obtaining the portrait edges, a weight map corresponding to the predicted portrait mask can be determined based on these edges. The image size of the weight map is the same as the image size of the predicted portrait mask; for example, the image size of the predicted portrait mask is 240*240. The weight map includes the weight values ​​corresponding to each mask pixel in the predicted portrait mask. The image size of the weight map is 240*240, and the pixel value of each weight pixel in the weight map is the weight value of the corresponding pixel in the predicted portrait mask. The pixel position of the weight pixel in the weight map is the same as the pixel position of its corresponding pixel in the predicted portrait mask. For example, if the pixel position of the weight pixel in the weight map is (10,25), then the pixel position of its corresponding pixel in the predicted portrait mask is also (10,25).

[0086] In one implementation of this embodiment, determining the weight map corresponding to the predicted portrait mask map based on the portrait edges specifically includes:

[0087] For each mask pixel in the predicted human face mask image, the distance between the mask pixel and the edge of the human face is determined, and the distance is used as the weight value of the mask pixel to obtain the weight value corresponding to each mask pixel in the predicted human face mask image.

[0088] Based on the weight values ​​corresponding to each mask pixel, the weight map corresponding to the predicted human face mask is determined.

[0089] Specifically, the distance between the mask pixel and the image edge is the Euclidean distance between the mask pixel and an edge point on the image edge. This edge point can be a specific pixel on the image edge, or it can be the edge point on the image edge with the smallest Euclidean distance to the mask pixel. In one specific implementation, this edge point is the edge point on the image edge with the smallest Euclidean distance to the mask pixel; that is, such as... Figure 5 As shown, the distance between a mask pixel and the edge of the portrait is the minimum Euclidean distance between the mask pixel and the portrait edge. Furthermore, after obtaining the minimum Euclidean distance between the mask pixel and the portrait edge, this minimum Euclidean distance can be normalized and used as the distance between the mask pixel and the portrait edge, so that the weight value corresponding to each mask pixel is a value between 0 and 1.

[0090] Furthermore, after obtaining the distance between the mask pixel and the edge of the portrait, this distance is used as the weight value corresponding to that mask pixel, thereby obtaining the weight value corresponding to each mask pixel in the predicted portrait mask image. After obtaining the weight value corresponding to each mask pixel, a blank image with the same size as the predicted portrait mask image can be created. Then, for each mask pixel in the predicted portrait mask, a candidate pixel corresponding to that mask pixel is selected in the blank image, and the pixel value of the candidate pixel is set as the weight value of the mask pixel, thereby obtaining the weight map corresponding to the predicted mask image, for example, as shown below. Figure 6 The weight map shown has a greater weight value for edge points than for non-edge points in the image edge. This increases the importance of the image edge in the target loss value, allowing the preset network model to learn the feature information of the image edge. This improves the accuracy of the trained image segmentation model in recognizing the image edge, and enhances the sharpness of the image edge in the image mask obtained by the image segmentation model.

[0091] S30. The preset network model is trained based on the weight map, the predicted human face mask map, and the target human face mask map corresponding to the training image to obtain a human face segmentation model.

[0092] Specifically, training the preset network model based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training image refers to determining the target loss value corresponding to the training image based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training image, and training the preset network model based on the target loss value. When the preset network model training meets preset conditions, the trained preset network model is used as a portrait segmentation model. The preset conditions are pre-set and may include a training iteration threshold and a loss threshold. The preset network model training meets the preset conditions when the number of training iterations reaches the threshold or the target loss value is less than the loss threshold. For example, the number of epochs for training the preset network model can be 300, the batch size can be 16, and when training the preset network model based on the target loss value, the stochastic gradient descent (SGD) method can be used to optimize the network gradient. The initial learning rate is adjusted to 0.05, and the learning rate is multiplied by 0.1 at iterations 100 and 200, with a momentum of 0.85.

[0093] In one implementation of this embodiment, training the preset network model based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training image to obtain a portrait segmentation model specifically includes:

[0094] For each mask pixel in the predicted portrait mask image, a candidate loss value corresponding to that mask pixel is determined based on the target portrait mask image corresponding to the training image, so as to obtain the candidate loss value corresponding to each mask pixel in the predicted portrait mask image.

[0095] Based on the weight map, the candidate loss values ​​corresponding to each mask pixel are weighted to obtain the target loss value corresponding to the training image.

[0096] The preset network model is trained based on the target loss value to obtain a human image segmentation model.

[0097] Specifically, the model structure of the preset network model is the same as that of the portrait segmentation model. The difference between the preset network model and the portrait segmentation model is that the model parameters of the preset network model are preset initial model parameters, while the model parameters of the portrait segmentation model are model parameters trained based on preset training samples. Therefore, the model structure of the preset network model will not be described in detail here, but can be referred to the model structure of the portrait segmentation model described above.

[0098] The candidate loss value is used to reflect the difference between the pixel value corresponding to the mask pixel and the pixel value of the target mask pixel in the target portrait mask image. The target mask pixel is the pixel in the template mask image whose pixel position is the same as that of the mask pixel. For example, if the pixel position of the mask pixel is (100, 100), then the pixel position of the target mask pixel in the target portrait mask is (100, 100). In one implementation of this embodiment, the formula for calculating the candidate loss value can be:

[0099]

[0100] Among them, P i Let represent the candidate loss value for predicting the i-th mask pixel in the portrait mask image, C be the number of channels in the predicted portrait mask image, and C be the total number of classifications. In this method, C = 2. This represents the pixel value of the i-th pixel in the c-th channel of the predicted portrait mask. This represents the pixel value of the i-th pixel in the target image mask.

[0101] Furthermore, after obtaining the candidate loss values, the target loss value of the training image is determined based on the candidate pixel values ​​corresponding to each mask pixel. The formula for calculating the target loss value can be:

[0102]

[0103] Where L represents the target loss value of the training image, N represents the number of mask pixels in the predicted portrait mask image, and w i =1+g i G represents the weighting coefficient for the i-th mask pixel. i P represents the weight value of the i-th mask pixel in the weight map. i This represents the candidate loss value for predicting the i-th mask pixel in the portrait mask image.

[0104] In summary, this embodiment provides a method for generating a portrait segmentation model, a portrait segmentation model, and a portrait segmentation method. The method for generating the portrait segmentation model includes inputting training images from preset training samples into a preset network model, determining a predicted portrait mask corresponding to the training images through the preset network model; obtaining portrait edges in the predicted portrait mask, and determining a weight map corresponding to the predicted portrait mask based on the portrait edges; and training the preset network model based on the weight map, the predicted portrait mask, and the target portrait mask corresponding to the training images to obtain a portrait segmentation model. This application determines the weight values ​​of each pixel in the predicted portrait mask during the training process by predicting portrait edges in the predicted portrait mask. By training the preset network model based on these weight values, the recognition effect of the portrait edges corresponding to the trained portrait segmentation model can be improved, thereby improving the accuracy of portrait segmentation.

[0105] Based on the above-mentioned method for generating portrait segmentation models, this embodiment provides a portrait segmentation method, such as... Figure 7 As shown, the portrait segmentation method includes:

[0106] B10. Input the image to be segmented into the portrait segmentation model;

[0107] B20. Using the portrait segmentation model, determine the portrait mask corresponding to the image to be segmented.

[0108] Specifically, the input to the portrait segmentation model is the image to be segmented, and the output is a portrait mask. In other words, after the image to be segmented is input into the portrait segmentation model, the model outputs a portrait mask corresponding to the image to be segmented. After obtaining the portrait mask corresponding to the image to be segmented, the portrait region and background region of the image to be segmented can be determined based on the portrait mask.

[0109] In one implementation of this embodiment, before inputting the image to be segmented into the portrait segmentation model, the method further includes:

[0110] The scene image is acquired by the imaging module, and the image size of the scene image is adjusted to obtain the image to be segmented, wherein the image size of the image to be segmented is smaller than the image size of the scene image.

[0111] Specifically, the imaging module is used to acquire images, and the scene image is obtained by capturing images through an imaging model. The imaging module can include one or more imagers. When the imaging module includes one imager, the scene image is obtained by capturing images through that imager; when the imaging module includes multiple imagers, the scene image is obtained by capturing images through the main imager in the imaging module. Furthermore, the image size of the image to be segmented is smaller than the image size of the scene image, and the image size of the image to be segmented can be determined based on the image size of the scene image. For example, the image size of the image to be segmented is half, a quarter, etc., of the image size of the scene image. In one specific implementation, the image size of the image to be segmented is a quarter of the image size of the scene image. By reducing the size of the scene image, the computational load for obtaining the portrait mask can be reduced, thereby improving the speed of obtaining the portrait mask.

[0112] In one implementation of this embodiment, after determining the portrait mask corresponding to the image to be segmented using the portrait segmentation model, the method further includes:

[0113] Based on the portrait mask, the portrait region and background region of the image to be segmented are determined.

[0114] Specifically, the portrait region is the image region occupied by the portrait in the image to be segmented, and the background region is the image region in the image to be segmented excluding the portrait region. The portrait region can be marked in the portrait mask, and the background region can be determined based on the marked portrait region. Furthermore, the portrait region and background region can be distinguished using different pixel values; for example, all pixels in the portrait region can be set to 1, and all pixels in the background region can be set to 0. In addition, after determining the portrait region and background region of the image to be segmented, the corresponding portrait image and background image can be extracted. Therefore, after determining the portrait region and background region of the image to be segmented based on the portrait mask, the method further includes:

[0115] Based on the portrait region and the background region, the image to be segmented is segmented to obtain the portrait image and / or background image corresponding to the image to be segmented.

[0116] Specifically, the portrait image is the image corresponding to the image region occupied by the portrait in the image to be segmented, that is, the image obtained by removing the background region from the image to be segmented. The background image is the image of the image to be segmented excluding the portrait region. Furthermore, since the image size of the image to be segmented can be smaller than the image size of the acquired scene image when acquiring the portrait mask image, the image size of the portrait image and / or the image size of the background image can be adjusted to the image size of the scene image after obtaining the portrait image and / or background image corresponding to the image to be segmented. Of course, in practical applications, the portrait mask image corresponding to the image to be segmented is first upsampled to the image size of the scene image, and then the scene image is directly segmented based on the upsampled portrait mask to obtain the portrait image and the background image.

[0117] Based on the above-described method for generating a portrait segmentation model, this embodiment provides a portrait segmentation model. The portrait segmentation model includes a first convolutional module, a first feature extraction module, a second feature extraction module, and a second convolutional module cascaded in sequence. The first feature extraction module includes a first preset number of factorized convolutional modules, and the second feature extraction module includes a second preset number of factorized convolutional modules. The factorized convolutional module includes several first convolutional layers, several second convolutional layers, and a fusion layer. The ratio of the width to the height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to the height of the second convolutional kernel. The input of the fusion layer includes the output of the first or second convolutional layer connected to it and the input of the factorized convolutional modules.

[0118] In one implementation, the width of the first convolutional layer or the width of the second convolutional layer is equal to 1.

[0119] In one implementation, the factorized convolution module further includes two third convolutional layers, a plurality of first convolutional layers and a plurality of second convolutional layers alternately connected in series between the two third convolutional layers, and the input of the fusion layer includes the output of the last third convolutional layer and the input of the first third convolutional layer.

[0120] It is also worth noting that the model structure of the portrait segmentation model is the same as that of the preset network model. For details, please refer to the above description of the preset network model, which will not be repeated here.

[0121] Based on the above-described method for generating a human image segmentation model, this embodiment provides a device for generating a human image segmentation model, such as... Figure 8 As shown, the generating apparatus includes:

[0122] The determination module 1000 is used to input training images from preset training samples into a preset network model, and determine the predicted human face mask corresponding to the training images through the preset network model, wherein the training images include human face regions;

[0123] The acquisition module 2000 is used to acquire the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges;

[0124] The training module 3000 is used to train the preset network model based on the weight map, the predicted portrait mask map, and the target portrait mask map corresponding to the training image to obtain a portrait segmentation model.

[0125] Based on the above-mentioned method for generating the human image segmentation model, this embodiment provides an image blurring method, which includes:

[0126] N10. Obtain the portrait mask of the image to be processed, and the candidate preview image corresponding to the image to be processed;

[0127] N20. Based on the portrait mask, determine the disparity map corresponding to the image to be processed, and based on the disparity map and the preview focus of the image to be processed, determine the fusion data corresponding to the image to be processed;

[0128] N30. Based on the fusion data, the candidate preview image is fused with the image to be processed to obtain the preview image corresponding to the image to be processed.

[0129] Specifically, the image to be processed can be an image captured by an imaging module, which includes at least two imagers: a main imager and an auxiliary imager. The main and auxiliary imagers are arranged on the same plane, and can be arranged horizontally or vertically adjacent. The main and auxiliary imagers can be dual cameras of an electronic device (e.g., a smartphone), meaning both the main and auxiliary imagers are cameras. For example, the main and auxiliary imagers can be dual rear cameras or dual front cameras. One of the main and auxiliary imagers can be a color imager and the other a monochrome imager (e.g., the main imager is a color imager, and the auxiliary imager is a monochrome imager). The main and auxiliary imagers can also use imagers with different focal lengths, or they can use the same imager. Of course, the imaging module can also include three imagers (e.g., a smartphone with three cameras), or four imagers, etc.

[0130] The image to be processed can be an image acquired by the imaging module configured within the electronic device itself, or an image acquired by the imaging module of another electronic device via network, Bluetooth, infrared, or other means. In a specific implementation of this embodiment, the image to be processed is captured by the imaging module configured within the electronic device itself, and the image to be processed is an image captured by the main imager of the imaging module. It is understood that the electronic device is configured with an imaging module, which includes at least a main imager and an auxiliary imager; the main imager is used to capture the image to be processed, and the auxiliary imager is used to capture an auxiliary image, which is used to assist in determining the disparity map of the image to be processed. For example, when a mobile phone with dual cameras takes a picture, if the main camera captures image A and the auxiliary camera captures image B, then image A is the image to be processed, and image B is an auxiliary image used to determine the depth map of image A.

[0131] In one implementation of this embodiment, the image to be processed is a preview image formed by the imaging module configured on the electronic device itself in preview mode. In other words, when the electronic device starts the shooting function and enters the preview mode, the preview image is formed by the main imager of the imaging module capturing the scene to be captured. Therefore, the acquisition of the portrait mask of the image to be processed can specifically be as follows: when the electronic device enters the shooting preview function, the main imager of the imaging module configured on the electronic device captures a preview image, and the captured preview image is used as the image to be processed.

[0132] The portrait mask can be used to reflect the portrait region in the image to be processed. The mask image can be obtained based on traditional algorithms or based on a neural network model. Furthermore, as... Figure 9 As shown, in the portrait mask image, the pixel value corresponding to the pixel in the portrait region of the image to be processed is a first preset pixel value, and the pixel value corresponding to the pixel in the image region other than the portrait region of the image to be processed is a second preset pixel value, and the first preset pixel value and the second preset pixel value are different. For example, the first preset pixel value is 1, and the second preset pixel value is 0, etc.

[0133] In one implementation of this embodiment, the candidate preview image corresponding to the image to be processed specifically includes:

[0134] The image size of the image to be processed is adjusted to a preset image size, and the adjusted image to be processed is subjected to Gaussian blurring to obtain a reference blurred image;

[0135] The image size of the reference blurred image is adjusted to the image size of the image to be processed, and the adjusted reference blurred image is used as the candidate preview image corresponding to the image to be processed.

[0136] Specifically, the preset image size can be pre-set or determined based on the image size of the image to be processed. The preset image size is smaller than the image size of the image to be processed. By reducing the image size of the image to be processed, the Gaussian blur processing time can be reduced, thereby improving the real-time performance of the preview image. In one implementation of this embodiment, the preset image size is determined based on the image size of the image to be processed. For example, the image size of the image to be processed is a multiple of the preset image size, such as 2 times, 4 times, etc.

[0137] In one implementation of this embodiment, the disparity map is a binocular disparity map, which may include the disparity corresponding to each first pixel in the image to be processed. The disparity refers to the horizontal distance between the center pixels of two matching image blocks in the left and right binocular images. Therefore, when obtaining the disparity map of the image to be processed, it is necessary to obtain an auxiliary image acquired by the imaging module of the image to be processed, and determine the disparity map corresponding to the image to be processed based on the image to be processed and the auxiliary image. Accordingly, determining the disparity map corresponding to the image to be processed based on the portrait mask specifically includes:

[0138] Determine the auxiliary image corresponding to the image to be processed, and determine the candidate disparity map corresponding to the image to be processed based on the image to be processed and the auxiliary image;

[0139] Based on the portrait mask image, the target disparity region corresponding to the portrait region in the image to be processed is selected in the candidate disparity image;

[0140] The disparity values ​​of each disparity pixel in the target disparity region are set to preset disparity values ​​to obtain the disparity map corresponding to the image to be processed.

[0141] Specifically, the image to be processed is captured by the main imager of the imaging module, and the auxiliary image is captured by the auxiliary imager of the imaging module. The auxiliary image is used to help determine the disparity map of the image to be processed. The candidate disparity map is composed of the disparity between each first pixel and its corresponding second pixel in the image to be processed. Disparity refers to the horizontal distance between the center pixels of two matching image blocks in the left and right binocular images. Each second pixel corresponding to each first pixel is included in the auxiliary image, and the pixel position of each first pixel in the image to be processed is the same as the pixel position of its corresponding second pixel in the auxiliary image.

[0142] In one implementation of this embodiment, to improve the speed of disparity map determination, the image size of the image to be processed and the image size of the auxiliary image can be adjusted before calculating the disparity map. The adjusted image size of the image to be processed is equal to the adjusted image size of the auxiliary image, but smaller than the original image size of the image to be processed. Furthermore, for each first pixel in the adjusted image to be processed, there exists a candidate disparity pixel in the candidate disparity map. This candidate disparity pixel reflects the disparity value between the first pixel and its corresponding second pixel, and the pixel position of the candidate disparity pixel in the candidate disparity map is the same as the pixel position of the first pixel in the adjusted image to be processed. Of course, it is worth noting that in practical applications, the image sizes of the image to be processed and the auxiliary image may not be adjusted, and the candidate disparity map corresponding to the image to be processed may be determined directly based on the image to be processed and the auxiliary image. Furthermore, when determining candidate disparity maps based on the image to be processed and auxiliary images, the SGBM algorithm can be used. After obtaining the candidate disparity maps, post-processing can be performed to improve their accuracy. For example, the post-processing can include hole filling and joint bilateral filtering algorithms. The SGBM algorithm, hole filling, and joint bilateral filtering algorithm are all existing algorithms and will not be described in detail here.

[0143] The target disparity region is the corresponding region in the candidate disparity map of the image region where the target human image is located. For example, when the image size of the image to be processed is equal to the image size of the candidate disparity map, the target disparity region is the same as the image region position in the disparity map where the target human image is located in the image region in the image to be processed. For example, if the image region where the target human image is located in the image to be processed is a rectangular region with four vertices of [100,100], [100,200], [200,200], and [200,100], then the image region position of the target disparity region in the candidate disparity map is a rectangular region with [100,100], [100,200], [200,200], and [200,100]. Furthermore, when the image size of the candidate disparity map is different from the image size of the image to be processed, the image size of the image to be processed can be adjusted to the image size of the candidate disparity map. Then, the target disparity region corresponding to the target human image can be determined in the candidate disparity map. For example, if the image size of the image to be processed is 480*480 and the image size of the candidate disparity map is 240*240, then the image to be processed can be downsampled to adjust the image size of the image to be processed to 240*240.

[0144] The preset disparity value can be pre-set or determined based on the target disparity region. It is used to replace the disparity values ​​of each disparity pixel in the target disparity region, making the disparity values ​​of each disparity pixel in the target disparity region the same. This ensures the consistency of the disparity values ​​of the image region corresponding to the target portrait and avoids false or missed blurring of the portrait during image blurring. In one implementation of this embodiment, the preset disparity value can be the average disparity value of each disparity pixel in the target disparity region. Thus, after obtaining the target disparity region, the disparity values ​​of each disparity pixel in the target disparity region are obtained, and then the average disparity value of each disparity pixel is calculated to obtain the disparity mean of each disparity pixel in the target disparity region. Finally, the disparity values ​​of each disparity pixel in the target disparity region are set as the disparity mean to obtain the disparity map corresponding to the image to be processed. For example, as shown in the example... Figure 10 The disparity map shown is provided. Of course, in practical applications, the average disparity value of each disparity pixel in the target disparity region can also be a weighted average, the average disparity value of some pixels in the target disparity region, or the maximum or minimum disparity value of each disparity pixel in the target disparity region, etc., which will not be elaborated here.

[0145] In one implementation of this embodiment, after obtaining the candidate disparity map corresponding to the image to be processed, it can be detected whether there is a target human image including the preview focus in the image to be processed. If there is a target human image including the preview focus, the target disparity region corresponding to the target human image is selected in the candidate disparity map; if there is no target human image including the preview focus, the candidate disparity map is used as the view corresponding to the image to be processed. In this way, on the one hand, when the preview focus is in the image area where the target human image is located, the target disparity region corresponding to the target human image can be optimized, thereby improving the consistency of the disparity value of the image area corresponding to the target human image and reducing false blurring and omission of human images; on the other hand, when the preview focus is in the image area where the human image is located, or when the image to be processed contains a human image, the image to be processed can be blurred based on the candidate disparity map. Therefore, this embodiment is applicable to both images to be processed that include human images and images to be processed that do not include human images, thereby improving the applicability of the human image segmentation model generation method.

[0146] Furthermore, the preview focus is the focal position of the image to be processed. The preview focus can be automatically generated based on the acquired image, generated based on the user's selection, or sent by an external device. For example, when the imaging device displays the image to be processed, it can receive the user's click operation on the image and obtain the click point as the preview focus, and use the position information of the click point (e.g., the pixel position of the corresponding pixel on the display interface, such as (125, 150)) as the position information of the preview focus.

[0147] The fusion data includes several fusion coefficients, each corresponding to a pixel in the image to be processed, and the pixels corresponding to each fusion coefficient are distinct. Each fusion coefficient is used to fuse the blurred image with the image to be processed. In one implementation of this embodiment, determining the fusion data corresponding to the image to be processed based on the disparity map and the preview focus of the image to be processed specifically includes:

[0148] For each disparity pixel in the disparity map, the disparity value corresponding to the disparity pixel is obtained. Based on the disparity value and the focus disparity value of the preview focus of the image to be processed, the fusion coefficient corresponding to the disparity pixel is determined to obtain the fusion coefficient corresponding to each disparity pixel.

[0149] The fusion coefficients corresponding to each disparity pixel are used as the fusion data corresponding to the image to be processed.

[0150] Specifically, the preview focus is the focus position of the image to be processed, and the focus disparity value corresponding to the preview focus is the pixel value of the disparity pixel point corresponding to the preview focus in the disparity map. For example, if the pixel position of the preview focus in the image to be processed is (100, 100), and the image size of the disparity map is the same as the image size of the image to be processed, then the focus disparity of the preview focus is the pixel value of the disparity pixel point at the pixel position (100, 100) in the disparity map.

[0151] Furthermore, after obtaining the focal disparity value corresponding to the preset focal point, the difference between this disparity value and the focal disparity value is calculated, and the fusion coefficient corresponding to the disparity pixel is calculated based on this difference. The formula for calculating the fusion coefficient can be:

[0152] disp′=Max_Disp*p / (1+p)

[0153] p = exp(-(disp - focus_disp) 2 / σ)

[0154] Where disp′ represents the fusion coefficient, disp represents the disparity value of the disparity pixel, focus_disp represents the focus disparity value, Max_Disp is a constant greater than 256, σ is a constant greater than 0, and exp(·) is an exponential function. In a specific implementation, Max_Disp can be 570, and σ can be 16.

[0155] In one implementation of this embodiment, the preview image is an image used for preview display, for example, such as... Figure 11 The preview image has the same image size as the image to be processed. For each preview pixel in the preview image, its pixel value is calculated based on the pixel value of a first pixel in the image to be processed, the pixel value of a second pixel in the candidate preview image, and the fusion coefficient corresponding to the first pixel. The first pixel is in the image to be processed, the second pixel is in the candidate preview image, and the pixel positions of the first pixel in the image to be processed, the second pixel in the candidate preview image, and the preview pixel in the preview image are all the same. For example, the pixel position of the preview pixel in the preview image is (50, 60), the pixel position of the first pixel in the image to be processed is (50, 60), and the pixel position of the second pixel in the candidate preview image is (50, 60). In a specific implementation, assuming the pixel value of the first pixel is 'a' and the pixel value of the second pixel is 'b', the formula for calculating the pixel value 'c' of the preview pixel can be: c = a * disp' + b * (1 - disp'), where disp' represents the fusion coefficient. Therefore, the fused image can be represented as:

[0156] I_prvBokeh=I*W+I_blur*(1-W)

[0157] Where I_prvBokeh represents the preview image, I represents the image to be processed, W represents the fused data, and I_blur represents the candidate preview image.

[0158] Based on the above image blurring method, this embodiment provides an image blurring device, the image blurring device comprising:

[0159] The acquisition module is used to acquire the portrait mask of the image to be processed, and the candidate preview image corresponding to the image to be processed;

[0160] The determining module is used to determine the disparity map corresponding to the image to be processed based on the portrait mask map, and to determine the fusion data corresponding to the image to be processed based on the disparity map and the preview focus of the image to be processed;

[0161] The fusion module is used to fuse the candidate preview image with the image to be processed based on the fusion data, so as to obtain the preview image corresponding to the image to be processed.

[0162] Based on the above-described method for generating a human portrait segmentation model, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the method for generating a human portrait segmentation model as described in the above embodiment.

[0163] Based on the above-mentioned method for generating the human face segmentation model, this application also provides a terminal device, such as... Figure 12 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0164] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0165] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0166] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0167] Furthermore, the specific working process of the aforementioned human portrait segmentation model generation device, the specific process of loading and executing multiple instruction processors in the storage medium and terminal device have been described in detail in the above method, and will not be repeated here.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a human face segmentation model, characterized in that, The generation method includes: The training images from the preset training samples are input into the preset network model, and the preset network model determines the predicted human face mask corresponding to the training images, wherein the training images include human face regions; Obtain the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges; The preset network model is trained based on the weight map, the predicted human face mask map, and the target human face mask map corresponding to the training image to obtain a human face segmentation model. The step of determining the weight map corresponding to the predicted human image mask based on human image edges specifically includes: For each mask pixel in the predicted human face mask image, the distance of the mask pixel to the edge of the human face is determined, and the distance is used as the weight value of the mask pixel to obtain the weight value corresponding to each mask pixel in the predicted human face mask image. The weight map corresponding to the predicted human face mask is obtained by replacing the pixel value of each mask pixel with the weight value of each mask pixel in the predicted human face mask map.

2. The method for generating a human image segmentation model according to claim 1, characterized in that, The image size of the weight map is the same as the image size of the predicted portrait mask map, and the weight map includes the weight value corresponding to each mask pixel in the predicted portrait mask map.

3. The method for generating a human face segmentation model according to claim 1, characterized in that, The step of training the preset network model based on the weight map, the predicted portrait mask map, and the target portrait mask map corresponding to the training image to obtain the portrait segmentation model specifically includes: For each mask pixel in the predicted portrait mask image, a candidate loss value corresponding to the mask pixel is determined based on the predicted portrait mask image and the target portrait mask image corresponding to the training image, so as to obtain the candidate loss value corresponding to each mask pixel in the predicted portrait mask image. Based on the weight map, the candidate loss values ​​corresponding to each mask pixel are weighted to obtain the target loss value corresponding to the training image. The preset network model is trained based on the target loss value to obtain a human image segmentation model.

4. The method for generating a human image segmentation model according to any one of claims 1-3, characterized in that, The preset network model includes several factorized convolutional modules. Each factorized convolutional module includes several first convolutional layers, several second convolutional layers, and a fusion layer. The ratio of the width to the height of the convolutional kernel of the first convolutional layer is the reciprocal of the ratio of the width to the height of the second convolutional kernel. The input items of the fusion layer include the output items of the first or second convolutional layer connected to it and the input items of the factorized convolutional modules.

5. The method for generating a human image segmentation model according to claim 4, characterized in that, The width of the first convolutional layer or the width of the second convolutional layer is equal to 1.

6. The method for generating a human image segmentation model according to claim 4, characterized in that, The factorized convolution module further includes two third convolutional layers, with several first convolutional layers and several second convolutional layers alternately cascaded between the two third convolutional layers. The input of the fusion layer includes the output of the last third convolutional layer and the input of the first third convolutional layer.

7. The method for generating a human image segmentation model according to claim 4, characterized in that, The preset network model further includes a first convolutional module and a second convolutional module. The first convolutional module and the second convolutional module are arranged in series. A portion of the factorized convolutional modules are cascaded between the first convolutional module and the second convolutional module, and another portion of the factorized convolutional modules are cascaded after the second convolutional module.

8. A method for human face segmentation, characterized in that, The portrait segmentation model is applied to the portrait segmentation model generated by the method for generating portrait segmentation models as described in any one of claims 1-7, wherein the portrait segmentation method specifically includes: Input the image to be segmented into the portrait segmentation model; The portrait segmentation model is used to determine the portrait mask image corresponding to the image to be segmented.

9. The portrait segmentation method according to claim 8, characterized in that, After determining the portrait mask image corresponding to the image to be segmented using the portrait segmentation model, the method further includes: Based on the portrait mask, the portrait region and background region of the image to be segmented are determined, wherein the background region is the image region in the image to be segmented other than the portrait region.

10. The portrait segmentation method according to claim 9, characterized in that, After determining the human image region and background region of the image to be segmented based on the human image mask, the method further includes: Based on the portrait region and the background region, the image to be segmented is segmented to obtain the portrait image and / or background image corresponding to the image to be segmented.

11. The portrait segmentation method according to claim 8, characterized in that, Before inputting the image to be segmented into the portrait segmentation model, the method further includes: The scene image is acquired by the imaging module, and the image size of the scene image is adjusted to obtain the image to be segmented, wherein the image size of the image to be segmented is smaller than the image size of the scene image.

12. The method for generating a human image segmentation model according to claim 7, characterized in that, The width of the first convolutional layer or the width of the second convolutional layer is equal to 1.

13. The method for generating a human portrait segmentation model according to claim 7, characterized in that, The factorized convolution module further includes two third convolutional layers, with several first convolutional layers and several second convolutional layers alternately cascaded between the two third convolutional layers. The input of the fusion layer includes the output of the last third convolutional layer and the input of the first third convolutional layer.

14. An apparatus for generating a human portrait segmentation model, used to implement the method for generating a human portrait segmentation model as described in any one of claims 1-7, characterized in that, The generating apparatus includes: The determination module is used to input training images from preset training samples into a preset network model, and determine the predicted human face mask corresponding to the training images through the preset network model, wherein the training images include human face regions; The acquisition module is used to acquire the human image edges in the predicted human image mask, and determine the weight map corresponding to the predicted human image mask based on the human image edges; The training module is used to train the preset network model based on the weight map, the predicted portrait mask map, and the target portrait mask map corresponding to the training image, so as to obtain a portrait segmentation model.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the method for generating a human face segmentation model as described in any one of claims 1-7, or to implement the steps in the human face segmentation method as described in any one of claims 8-11.

16. A terminal device, characterized in that, include: Processor, memory, and communication bus; the memory stores a computer-readable program that can be executed by the processor; The communication bus enables communication between the processor and the memory; When the processor executes the computer-readable program, it implements the steps in the method for generating a human image segmentation model as described in any one of claims 1-7, or the steps in the human image segmentation method as described in any one of claims 8-11.

Images