Multi-person cutout method, system, device and storage medium

Abstract

The present invention discloses a multi-person cutout method, system, device and storage medium, and relates to the field of information processing technology. The multi-person cutout method comprises the following steps: obtaining multi-person images; performing a first process on the multi-person images to obtain multiple single-person images; performing a second process on the single-person images to obtain a first cutout result of the single-person images; merging the first cutout results of the multiple single-person images to obtain a second cutout result. The second cutout result of the present application has a higher accuracy than that of directly cutting out the multi-person images. The cutout result obtained by the present application has a higher accuracy than that of directly cutting out the multi-person images.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a multi-person cutout method, system, device and storage medium. Background Art

[0002] There are two solutions to the task of extracting foreground: segmentation and matting.

[0003] There are two types of segmentation: semantic segmentation and instance segmentation. Semantic segmentation refers to classifying each pixel of a given image into a certain category, for example, classifying the pixels representing "people" in the image as "people" and the pixels representing "dogs" as "dogs". Instance segmentation distinguishes different instances of objects of the same category with different identities. For example, in the instance segmentation of "people", it is necessary to indicate which pixels belong to "person 1" and which belong to "person 2". Since it is necessary to identify the semantic information in the image for segmentation, machine learning methods are needed. Currently, both semantic and instance segmentation use deep learning methods to learn from a large amount of data to obtain a model that can automatically identify the semantics of pixels in the image.

[0004] There is a certain difference between matting and segmentation. In the concept of "matting", it is assumed that each pixel in the image is composed of a certain proportion of foreground color and a certain proportion of background color. Therefore, instead of hard classification of a certain pixel like semantic segmentation, it is necessary to find out how much foreground proportion each pixel contains. Compared with segmentation, the foreground extracted by matting method can be more naturally synthesized with other background images. Traditional matting relies on the ternary map provided by the user and is a semi-automatic matting.

[0005] Both semantic / instance segmentation and automatic cutout methods rely on deep learning methods and require a large amount of data support. The datasets required for semantic segmentation or instance segmentation are easier to obtain than those for cutouts. This is because the datasets in segmentation tasks are relatively rough, and their object boundaries are usually roughly hard segmented. The datasets for automatic cutouts are more difficult to obtain because a large number of object boundaries need to be carefully annotated to mark how much foreground proportion each pixel contains. This is a real number annotation task, which requires much more manual work than the category annotation of segmentation.

[0006] Due to the above data set situation, the current cutout solution is:

[0007] (1) The semantic / instance segmentation dataset is relatively rich and contains a large number of multi-person image data, so a multi-person image segmentation model can be trained on the dataset. Therefore, the current solutions for foreground extraction of multi-person images are mostly semantic segmentation methods. This method designs a segmentation network and trains it on a multi-person image segmentation dataset to obtain a model that can segment multiple people from an image. However, segmentation is an exact classification of pixels, and it cannot be used to determine the foreground ratio in each pixel. Therefore, although segmentation can roughly segment multi-person images, the segmentation accuracy is insufficient.

[0008] (2) Automatic image cutout datasets are relatively scarce, and existing datasets only contain single-person images and almost no multi-person images. This situation makes it impossible to directly train a multi-person image cutout model on the cutout dataset. The current automatic image cutout model is limited by the dataset. Although it can be trained for single-person cutout, it will cause large-scale errors such as missing cutouts and hollowing out when used for multi-person image cutouts. Summary of the invention

[0009] In order to solve at least one of the above technical problems, the present invention provides a multi-person cutout method, system, device and storage medium to improve the cutout accuracy of multi-person images.

[0010] In a first aspect, an embodiment of the present invention provides a multi-person cutout method, comprising the following steps:

[0011] Get multi-person images;

[0012] Performing a first processing on the multi-person image to obtain a plurality of single-person images;

[0013] Performing a second processing on the single-person image to obtain a first cutout result of the single-person image;

[0014] The first cutout results of the multiple single-person images are combined to obtain a second cutout result.

[0015] In some embodiments, performing a first processing on the multi-person images to obtain a plurality of single-person images comprises the following steps:

[0016] Inputting the multi-person image into a portrait detection model to obtain multiple target detection results;

[0017] The multi-person image is cropped according to the plurality of target detection results to obtain a plurality of single-person images.

[0018] In some embodiments, the target detection result includes a target position, a target length, and a target width.

[0019] In some embodiments, performing a second process on the single-person image to obtain a first cutout result of the single-person image comprises the following steps:

[0020] Inputting the single-person image into a single-person image cutout model to obtain a mask image of the single-person image;

[0021] The mask image is scaled according to the target length and the target width to obtain the first cutout result.

[0022] In some embodiments, before the step of inputting the single-person image into the single-person image cutout model to obtain the mask image of the single-person image, the step of performing a second process on the single-person image to obtain the first cutout result of the single-person image further includes the following steps:

[0023] Obtaining input image specifications of the single-person image matting model;

[0024] The scaled single-person image is obtained by performing scaling processing according to the input image specification.

[0025] In some embodiments, merging the first cutout results of the plurality of single-person images to obtain a second cutout result comprises the following steps:

[0026] Obtaining input image specifications of the multi-person image;

[0027] Generate a background image according to the input image specification;

[0028] The first cutout results of the plurality of single-person images are respectively written into the background image to obtain the second cutout result, wherein the position of the first cutout result of each single-person image in the background image is determined by the target position.

[0029] In some embodiments, merging the first cutout results of the plurality of single-person images to obtain a second cutout result further comprises the following steps:

[0030] Obtain overlapping positions of a plurality of the first cutout results in the background image;

[0031] Acquire multiple pixel values ​​at the overlapping positions to generate a pixel value set;

[0032] The maximum pixel value is taken from the pixel value set as the pixel value of the overlapping position.

[0033] In a second aspect, an embodiment of the present invention further provides a multi-person cutout system, comprising:

[0034] The first module is used to obtain multi-person images;

[0035] The second module is used to perform a first processing on the multi-person image to obtain multiple single-person images;

[0036] A third module is used to perform a second processing on the single-person image to obtain a first cutout result of the single-person image;

[0037] The fourth module is used to merge the first cutout results of the multiple single-person images to obtain a second cutout result.

[0038] In a third aspect, an embodiment of the present invention further provides a multi-person cutout device, comprising:

[0039] Memory, used to store programs;

[0040] A processor, configured to execute the program, for:

[0041] Get multi-person images;

[0042] Performing a first processing on the multi-person image to obtain a plurality of single-person images;

[0043] Performing a second processing on the single-person image to obtain a first cutout result of the single-person image;

[0044] The first cutout results of the multiple single-person images are combined to obtain a second cutout result.

[0045] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, in which a program executable by a processor is stored, and when the program executable by the processor is executed by the processor, it is used to implement the multi-person cutout method as described in the embodiment of the first aspect above.

[0046] The above technical solution of the present invention has at least one of the following advantages or beneficial effects: the present invention performs a first processing on a multi-person image to segment a plurality of single-person images, performs a second processing on the single-person image to obtain a first cutout result of the single-person image, and merges the first cutout results of the plurality of single-person images to obtain a second cutout result. The first cutout result obtained by cutting out the single-person image after the first processing has a high accuracy, and then the first cutout results of the single-person image are merged to obtain the second cutout result. The second cutout result of the present application has a higher accuracy than cutting out directly on a multi-person image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a multi-person cutout method provided according to an embodiment of the present invention;

[0048] Figure 2 2 is a schematic diagram showing an example of a multi-person cutout method provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments described in the embodiments of this application should not be regarded as limitations of this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0050] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] The embodiment of the present invention provides a method for cutting out a multi-person image, which can improve the accuracy of cutting out a multi-person image. Figure 1 The method of the embodiment of the present invention includes but is not limited to step S110, step S120, step S130 and step S140.

[0053] Step S110, obtaining images of multiple persons.

[0054] Specifically, a multi-person image can be obtained from an interactive application, which is installed in a user terminal, and a user inputs a multi-person image through the user terminal. The user terminal can be a device such as a mobile phone, a tablet computer, or a computer. Multi-person images can also be obtained from various data. Since the images input into the user terminal or the images in the database are not necessarily all multi-person images, before the step of obtaining multi-person images, the number of portraits in the input image can be detected by face recognition. When the number of portraits is one, a single-person cutout method is used for cutout; when the number of portraits is greater than one, the multi-person cutout method of this solution is used for cutout.

[0055] Step S120: performing a first processing on the multi-person images to obtain a plurality of single-person images.

[0056] Specifically, step S120 includes step S121 and step S122:

[0057] Step S121, inputting multiple images into a portrait detection model to obtain multiple target detection results;

[0058] The portrait detection model is a model with portrait detection capability. For example, the portrait detection model may be a portrait detection model obtained by deep learning based on the Yolov3 model. The portrait detection model may be constructed in the following manner:

[0059] Get the Yolov3 model code framework;

[0060] Get the MS COCO (Microsoft Common Objects in Context) dataset. It should be noted that this dataset is commonly used for training. It contains a large number of images and provides labels such as detection boxes, segmentation maps, key points, etc. It is also used for many computer vision tasks.

[0061] Train the Yolov3 model code framework on the portrait category in the MS COCO dataset;

[0062] After training to convergence conditions, the portrait detection model is obtained.

[0063] Input the multi-person image into the portrait detection model to obtain target detection results of multiple portraits. The target detection result can be a detection frame, specifically including the target position, target length and target width. The target position can be the coordinate of the upper left corner of the detection frame. There are several target detection results corresponding to the number of portraits in the multi-person image. For example, there are 3 portraits in the multi-person image. After the multi-person image is input into the portrait detection model, 3 target detection results can be obtained. According to the target detection results, the areas of the 3 portraits in the multi-person image can be obtained respectively.

[0064] Step S122: cropping the multi-person image to obtain multiple single-person images according to the multiple target detection results.

[0065] Specifically, the portrait region of a single person in a multi-person image can be confirmed according to the target position, target length and target width of the target detection result. A single-person image can be obtained by cropping the portrait region from the multi-person image. There are as many single-person images as there are portraits in the multi-person image.

[0066] Step S130, performing a second processing on the single-person image to obtain a first cutout result of the single-person image.

[0067] Specifically, step S130 includes step S131 and step S132.

[0068] Step S131, input the single-person image into the single-person image cutout model to obtain a mask image of the single-person image.

[0069] The single-person image cutout model is a model with the ability to cut out portraits. Exemplarily, the single-person image cutout model can be constructed in the following manner:

[0070] Obtain the Deep automatic portrait matting dataset and preprocess the dataset to obtain the training portrait image, the corresponding transparency map, and the corresponding ternary map;

[0071] Input the training portrait image into the pre-built single-person image matting model to generate a training ternary map and a training transparency map;

[0072] The loss of the trained ternary map and the trained transparency map is calculated with the ternary map and the corresponding transparency map of the training portrait image in the dataset, and the parameters of the single-person image cutout model are updated. After the loss convergence condition is met, the trained single-person image cutout model is obtained.

[0073] Inputting a single-person image into the single-person image cutout model can obtain a mask map with the portrait as the foreground. The mask map is a map that marks how much proportion of each pixel in the image belongs to the foreground and how much proportion belongs to the background.

[0074] Step S132, scaling the mask image according to the target length and the target width to obtain a first cutout result.

[0075] Since the length and width of the mask image obtained after the single-person image cutout model are not necessarily the same as the target length and target width of the target detection result of the multi-person image after the portrait detection model, in order to later synthesize the mask images of multiple portraits, the length and width of the mask image need to be scaled to the same as the target length and target width.

[0076] Furthermore, since the single-person image matting model has specific requirements on the size of the input image, before step S131, step S130 also includes step S133 and step S134.

[0077] Step S133, obtaining the input image specifications of the single-person image cutout model.

[0078] Step S134, performing scaling processing according to the input image specification to obtain a scaled single-person image.

[0079] For example, the input image specification of the single-person image cutout model is required to be 200×200, and the single-person image after cropping is 100×100. An image scaling algorithm is required to enlarge the 100×100 single-person image to a 200×200 single-person image, and then input the enlarged single-person image into the single-person image cutout model. It should be noted that the image scaling algorithm can be a bilinear interpolation method, a nearest neighbor interpolation method, or a bicubic interpolation method.

[0080] Step S140: merging the first cutout results of the multiple single-person images to obtain a second cutout result.

[0081] Specifically, step S140 includes step S141, step S142 and step S143.

[0082] Step S141, obtaining input image specifications of multi-person images.

[0083] Step S142, generating a background image according to the input image specification.

[0084] Step S143, writing the first cutout results of the plurality of single-person images into the background image respectively to obtain a second cutout result, wherein the position of the first cutout result of each single-person image in the background image is determined by the target position.

[0085] The background image can be a pure black image with the same length and height as the multi-person image. The first cutout result of the single-person image is written into the background image according to the target position of the single-person image, and then the first cutout results of multiple single-person images are synthesized to obtain the second cutout result of the multi-person image, that is, the mask image of the multi-person image.

[0086] Furthermore, step S140 also includes step S144, step S145 and step S146.

[0087] Step S144, obtaining overlapping positions of multiple first cutout results in the background image.

[0088] Step S145, acquiring multiple pixel values ​​at overlapping positions to generate a pixel value set.

[0089] Step S146, taking the maximum pixel value from the pixel value set as the pixel value of the overlapping position.

[0090] Exemplarily, parts of the two first cutout results overlap, and two pixel values ​​appear at the overlapping position. Taking the larger pixel value as the pixel value of the overlapping position can better preserve the cutout result of the overlapping area.

[0091] Reference Figure 2 , an embodiment of the present invention is described below:

[0092] The multi-person images include person 1, person 2, and person 3. After the multi-person images are input into the portrait detection model, the corresponding detection frames 1, 2, and 3 are obtained. The detection frames are cropped to obtain single-person images 1, 2, and 3. The three single-person images are resized into single-person images 1, 2, and 3 with a size of 200×200. Single-person images 1, 2, and 3 are respectively input into the single-person image cutout model to obtain the mask. Figure 1 , Mask Figure 2 And mask image 3. Figure 1 Scaling to the same target length and target width as detection frame 1 obtains the first cutout result 1, mask Figure 2The first cutout result 2 is obtained by scaling to the same target length and target width as the detection frame 2, and the mask image 3 is scaled to the same target length and target width as the detection frame 3 to obtain the first cutout result 3. A pure black background image with the same specifications as the input multi-person image is generated. The first cutout result 1, the first cutout result 2, and the first cutout result 3 are written into the background image according to the target position of the detection frame, respectively, to obtain the second cutout result of the multi-person image, that is, the multi-person image mask with the same specifications as the multi-person image.

[0093] In addition, detection frame 3 includes a part of person 2 in detection frame 2. When detection frame 3 is cut out, this small part of person 2 is easily regarded as the background by the single-person image cutout model, so the first cutout result of person 2 in cutout result 3 is likely to be 0. In detection frame 2, person 2 is the main body of detection frame 2, so under normal circumstances, person 2 in the first cutout result 2 can be completely cut out, and the result is 1 (or the result pixel value is 255). Then the pixel value set of the overlapping pixel points is {0,255}, so the maximum value of 255 is taken to retain the complete cutout result.

[0094] The embodiment of the present invention further provides a multi-person cutout system, comprising:

[0095] The first module is used to obtain multi-person images;

[0096] The second module is used to perform a first processing on the multi-person images to obtain multiple single-person images;

[0097] The third module is used to perform a second processing on the single-person image to obtain a first cutout result of the single-person image;

[0098] The fourth module is used to merge the first cutout results of multiple single-person images to obtain a second cutout result.

[0099] The embodiment of the present invention further provides a multi-person cutout device, comprising:

[0100] Memory, used to store programs;

[0101] A processor, configured to execute the program, for:

[0102] Get multi-person images;

[0103] Performing a first processing on the multi-person images to obtain a plurality of single-person images;

[0104] Performing a second processing on the single-person image to obtain a first cutout result of the single-person image;

[0105] The first cutout results of multiple single-person images are combined to obtain a second cutout result.

[0106] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more control processors, for example, to execute the steps described in the above embodiment.

[0107] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0108] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for cutting out a picture of multiple people, characterized in that: The following steps are involved: Get multi-person images; Performing a first processing on the multi-person image to obtain a plurality of single-person images; Performing a second processing on the single-person image to obtain a first cutout result of the single-person image; Obtaining input image specifications of the multi-person image; Generate a background image according to the input image specification; The first cutout results of the multiple single-person images are written into the background image respectively to obtain the second cutout results, wherein the position of the first cutout result of each single-person image in the background image is determined by the target position, and the target position is obtained by inputting the multi-person images into the portrait detection model.

2. The multi-person cutout method according to claim 1, characterized in that: The first processing of the multi-person images to obtain a plurality of single-person images comprises the following steps: Inputting the multi-person image into a portrait detection model to obtain multiple target detection results; The multi-person image is cropped according to the plurality of target detection results to obtain a plurality of single-person images.

3. The multi-person cutout method according to claim 2, characterized in that: The target detection result includes the target position, target length and target width.

4. The multi-person cutout method according to claim 3, characterized in that: The performing the second processing on the single-person image to obtain the first cutout result of the single-person image comprises the following steps: Inputting the single-person image into a single-person image cutout model to obtain a mask image of the single-person image; The mask image is scaled according to the target length and the target width to obtain the first cutout result.

5. The multi-person cutout method according to claim 4, characterized in that: Before the step of inputting the single-person image into the single-person image cutout model to obtain the mask image of the single-person image, the step of performing a second process on the single-person image to obtain the first cutout result of the single-person image further includes the following steps: Obtaining input image specifications of the single-person image matting model; The scaled single-person image is obtained by performing scaling processing according to the input image specification.

6. The multi-person cutout method according to claim 5, characterized in that: The multi-person cutout method further comprises the following steps: Obtain overlapping positions of a plurality of the first cutout results in the background image; Acquire multiple pixel values ​​at the overlapping positions to generate a pixel value set; The maximum pixel value is taken from the pixel value set as the pixel value of the overlapping position.

7. A multi-person cutout system, characterized in that: include: The first module is used to obtain multi-person images; The second module is used to perform a first processing on the multi-person image to obtain multiple single-person images; A third module is used to perform a second processing on the single-person image to obtain a first cutout result of the single-person image; The fourth module is used to obtain the input image specifications of the multi-person image; and generate a background image according to the input image specifications; The first cutout results of the multiple single-person images are written into the background image respectively to obtain the second cutout results, wherein the position of the first cutout result of each single-person image in the background image is determined by the target position, and the target position is obtained by inputting the multi-person images into the portrait detection model.

8. A multi-person cutout device, characterized in that: include: Memory, used to store programs; A processor, configured to execute the program, for: Get multi-person images; Performing a first processing on the multi-person image to obtain a plurality of single-person images; Performing a second processing on the single-person image to obtain a first cutout result of the single-person image; Obtaining input image specifications of the multi-person image; Generate a background image according to the input image specification; The first cutout results of the multiple single-person images are written into the background image respectively to obtain the second cutout results, wherein the position of the first cutout result of each single-person image in the background image is determined by the target position, and the target position is obtained by inputting the multi-person images into the portrait detection model.

9. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the multi-person cutout method as described in any one of claims 1-6 when executed by the processor.

Images