Image acquisition model training method for photos, image acquisition method, computer device and computer-readable storage medium

Abstract

The application provides a photograph image acquisition model training method, an image acquisition method, a computer device and a computer readable storage medium. The photograph image acquisition model training method comprises collecting photograph data, inputting the photograph data into an initial image acquisition model to obtain a transparency prediction map, performing global thresholding processing on the transparency prediction map to generate a segmentation mask map, performing augmentation processing on the photograph data to obtain augmented data, training a segmentation model based on a segmentation neural network by taking the augmented data as input, segmenting the augmented data by using the trained segmentation model to obtain an inference mask map, performing morphological dilation processing on the inference mask map to generate an inference mask morphology map, splicing channels of the augmented data and the corresponding inference mask map to form a four-channel map, and training an image acquisition model based on an image acquisition neural network by taking the four-channel map as input. The method can improve the recognition ability of the image acquisition model for the edge of the foreground image.

Description

Technical Field

[0001] This invention relates to the field of image acquisition, and more specifically to a training method for acquiring ID photos, an image acquisition method, a computer device, and a computer-readable storage medium. Background Technology

[0002] ID photos are typically taken in photo studios. However, some ID photos are created by cutting out images from everyday photos or mobile phone cameras because the person cannot be present in person to have their photo taken. In this method of creating ID photos from cutout images, the image is generated by identifying the main subject in a front-facing portrait photo taken against a non-green screen background, extracting the foreground of the subject and its transparency image, cropping it, and then overlaying the foreground and background colors based on the transparency image to generate a suitable ID photo.

[0003] Traditional image matting techniques can train segmentation models using static image acquisition algorithms (Trimap), but these models require both the original image and the trimap image as input. However, the trimap image is a grayscale image, which is difficult to obtain and increases the computational burden.

[0004] Traditional image matting techniques can be trained using matting neural networks such as Modnet, which do not require image acquisition processing of trimap images. However, this image acquisition model is generally trained using relatively clear image datasets. When using general portrait image acquisition models, the image acquisition effect is generally better with relatively clear images. However, since the images used to make ID photos may be low-resolution photos taken in low-light environments, the image acquisition effect of these images is not good. Summary of the Invention

[0005] The first objective of this invention is to provide a method for training an image acquisition model for photographs that improves the ability to distinguish the edges of foreground portraits.

[0006] A second objective of this invention is to provide an image acquisition method using an image acquisition model.

[0007] A third objective of this invention is to provide a computer device that applies the above-described image acquisition model training method and image acquisition method.

[0008] A fourth objective of this invention is to provide a computer-readable storage medium for implementing the above-described image acquisition model training method and image acquisition method.

[0009] To achieve the aforementioned first objective, the present invention provides an image acquisition model training method, comprising: acquiring photographic data; inputting the photographic data into an initial image acquisition model to obtain a transparency prediction map; performing global thresholding on the transparency prediction map to generate a segmentation mask map; performing augmentation processing on the photographic data to obtain augmented data; using the augmented data as input and the segmentation mask map as labels, training a segmentation model based on a segmentation neural network; using the trained segmentation model to segment the augmented data to obtain an inference mask map; performing morphological dilation processing on the inference mask map to generate an inference mask morphology map; concatenating the augmented data and its corresponding inference mask map through channels to form a four-channel image; using the four-channel image as input and the transparency prediction map as labels, training an image acquisition model based on an image acquisition neural network.

[0010] As can be seen from the above scheme, this invention trains a segmentation model and an image acquisition model, and performs image acquisition operations through these two models. After amplifying the data and inputting it into the segmentation model, an inference mask image is obtained. This inference mask image undergoes morphological dilation to generate an inference mask morphology image. The inference mask morphology image is then concatenated with the image to form a four-channel image. The inference mask morphology image in the four-channel image can serve as auxiliary prior information for distinguishing foreground images. The image acquisition model uses the inference mask morphology image to eliminate complex background information, enhancing its ability to distinguish image edges.

[0011] In a further proposed approach, the convolutional layer of the image acquisition neural network has four channels.

[0012] Therefore, since the input to the image acquisition neural network is a four-channel image, the number of channels in the convolutional layer of the image acquisition neural network should also be changed to four channels.

[0013] In a further scheme, global thresholding processing is performed on the transparency prediction map to generate a segmentation mask map, including: setting a pixel threshold; determining whether each pixel in the transparency prediction map is greater than or equal to the pixel threshold; if a pixel in the transparency prediction map is greater than or equal to the pixel threshold, then the pixel is set to 255; if a pixel in the transparency prediction map is less than the pixel threshold, then the pixel is set to 0.

[0014] Therefore, the pixel threshold is set to 120.

[0015] In a further proposed solution, the photo data collection includes: collecting portrait images from the network; removing images with two or more faces using a face detection model; removing images with two or more people using a person detection model; removing images with three Euler angles greater than a preset angle threshold using a head pose estimation model; removing images of people wearing hats using a hat detection model; and removing images where faces are occluded using an occlusion detection model, thus obtaining the photo data.

[0016] Therefore, it can be seen that photos can be selected online, and celebrity photos or people photos can be obtained through multiple channels. Unobstructed single-person front-facing photos can be selected by using face detection models, person detection models, head pose estimation models, hat detection models, and occlusion detection models.

[0017] In a further embodiment, the photo data is augmented to obtain augmented data including: facial analysis of the photo data using a face analysis model; adjustments to each image in the photo data to generate first augmented photo data; dividing the photo data into a first group and a second group of photos with equal numbers, and placing the first augmented photo data into their respective groups; using a makeup transfer neural network model, using the first group of photos as a reference, transferring the makeup from the first group of photos to the second group of photos to generate second augmented photo data; replacing the portraits in the second group of photos with images from the first group to generate third augmented photo data; selecting a background image, and replacing the background of the photo data, the first augmented photo data, the second augmented photo data, and the third augmented photo data to generate fourth augmented photo data; the photo data, the first augmented photo data, the second augmented photo data, the third augmented photo data, and the fourth augmented photo data are combined to form augmented data.

[0018] Therefore, digital augmentation alters the texture, color, and makeup of different parts of a person's image, as well as the background pattern. However, it does not change the area of ​​the image during the augmentation process. This improves the model's stability and robustness in identifying the background, and increases the model's accuracy when acquiring images from low-quality pictures.

[0019] In a further embodiment, generating the fourth amplified photo data by replacing the background of the photo data, the first amplified photo data, the second amplified photo data, and the third amplified photo data includes: normalizing the transparency prediction image and denoting it as alpha, denoting the amplified data as F, denoting the background image as B, and denoting the fourth amplified photo data as I; then the formula for generating the fourth amplified photo data is I = alpha * F + (1 - alpha) * B.

[0020] Therefore, the fourth amplified image is obtained by fusing the background image and the foreground image using a formula.

[0021] In a further proposed solution, the background image is either a monotonous background or a background with no human figures.

[0022] This shows that different background images can improve the ability of image acquisition models to distinguish backgrounds.

[0023] To achieve the second objective mentioned above, the present invention provides an image acquisition method, comprising: acquiring an image to be acquired; segmenting the image to be acquired using a segmentation model to generate a mask image; concatenating the mask image with the image to be acquired through channels to generate a four-channel image to be acquired; and performing an image acquisition operation on the four-channel image to be acquired using an image acquisition model.

[0024] As can be seen from the above scheme, in the image acquisition process, the mask image obtained by the segmentation model is concatenated with the image to be acquired to obtain a four-channel image. The image acquisition model then performs image acquisition operations on this four-channel image. Using a dual-model approach for image acquisition can improve the ability to distinguish image edges.

[0025] To achieve the third objective described above, the computer device provided by the present invention includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the above-described image acquisition model training method and the above-described image acquisition method.

[0026] To achieve the fourth objective mentioned above, the present invention provides a computer-readable storage medium storing a computer program thereon, characterized in that: when the computer program is executed, it implements the above-mentioned image acquisition model training method for photographs and the above-mentioned image acquisition method. Attached Figure Description

[0027] Figure 1 This is a flowchart of an embodiment of the image acquisition model training method for photographs according to the present invention.

[0028] Figure 2 This is a flowchart illustrating the process of collecting photo data in an embodiment of the image acquisition model training method for photos according to the present invention.

[0029] Figure 3 This is a flowchart illustrating the global thresholding process of the transparency prediction map in an embodiment of the image acquisition model training method for photographs according to the present invention.

[0030] Figure 4 This is a flowchart of the photo data augmentation process in an embodiment of the image acquisition model training method of the present invention.

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0032] Example of image acquisition model training method for photos:

[0033] See Figure 1 , Figure 1This is a flowchart illustrating an embodiment of the image acquisition model training method for this invention. The segmentation model and image acquisition model trained by this invention improve the ability to distinguish image edges. First, step S11 is executed to collect photo data, which is then input into the initial image acquisition model to obtain a transparency prediction map. The photo data is collected from the internet, and the initial image acquisition model is an image acquisition model trained on a publicly available dataset. Technicians manually inspect the transparency prediction maps, and any flawed transparency prediction maps are modified and adjusted using image editing software.

[0034] After step S11 is completed, step S12 is executed to perform global thresholding on the transparency prediction map, generating a segmentation mask map. The transparency prediction map after global thresholding can better separate the image from the background.

[0035] After generating the segmentation mask, step S13 is executed to augment the photo data, resulting in augmented data. By altering the texture, color, makeup, and other features of different parts of the portrait in the photo data, or by changing the background pattern, the robustness of the segmentation and image acquisition models in identifying backgrounds can be improved during training. This allows the segmentation and image acquisition models to better utilize complex backgrounds and improves the accuracy of image acquisition from low-quality images.

[0036] After obtaining the amplified data, step S14 is executed, using the amplified data as input and the segmentation mask image as the label, to train a segmentation model based on the segmentation neural network. The segmentation neural networks used in this invention are the BiseNet neural network and a lightweight U-Net neural network. The BiseNet neural network has high accuracy and precision, but consumes more resources; it is suitable for images with complex backgrounds or low-quality images. The lightweight U-Net neural network combines the U-Net structure with the MobileNet-V2 structure, which can improve the execution efficiency of the segmentation model and has lower resource consumption; it is suitable for high-quality images and images with simple backgrounds. During the segmentation model training process, random cropping, flipping, brightness or contrast changes can be performed on the amplified data, which can improve the convergence and robustness of the segmentation model.

[0037] After the segmentation model is trained, step S15 is executed to segment the augmented data using the trained segmentation model, obtaining an inference mask image. After obtaining the inference mask image, step S16 is executed to perform morphological dilation on the inference mask image, generating an inference mask morphology image. Morphological dilation expands the boundary points of the inference mask morphology image outwards, making the boundaries of the foreground image clearer.

[0038] After generating the inference mask morphology image, step 17 is executed to concatenate the amplified data with its corresponding inference mask image to form a four-channel image. The amplified data is an RGB image with three channels, and the inference mask image is a one-channel image. Concatenating the amplified data with the inference mask image results in a four-channel image.

[0039] After forming the four-channel image, step S18 is executed, using the four-channel image as input and the transparency prediction image as label, to train the image acquisition model based on the image acquisition neural network. The image acquisition neural network used in this invention is the MODNet neural network, which features high accuracy and strong generalization performance. However, the image acquisition model trained by the MODNet neural network may exhibit flaws due to poor image edge processing. This invention changes the input of the MODNet neural network to a four-channel image and also changes the number of channels in the convolutional layers to four. During the training process of the image acquisition model, the augmented data can be randomly cropped, flipped, and its brightness or contrast can be changed, which can improve the convergence and robustness of the segmentation model. The trained image acquisition model has a strong ability to recognize foreground images.

[0040] See Figure 2 , Figure 2 This is a flowchart illustrating the image acquisition model training method for photos according to an embodiment of the present invention, showing the process of collecting photo data. First, step S31 is executed to collect portrait images from the network. 34,438 photos of Asian celebrities or other people are collected from the network. After collecting the portrait images from the network, steps S32 and S33 are executed to remove two or more face images using a face detection model and two or more person images using a person detection model, resulting in a single-person photo.

[0041] After obtaining the individual photo, step S35 is executed, using a head pose estimation model to remove images where the three Euler angles are greater than a preset threshold. The three Euler angles are pitch, yaw, and roll, with a preset threshold of 15 degrees. When the pitch, yaw, and roll angles are less than 15 degrees, the photo is a frontal view.

[0042] Steps S36 and S37 are executed to remove images of people wearing hats using a hat detection model and images of faces being occluded using an occlusion detection model, resulting in 4000 unoccluded, front-facing, single-person photos. Since this invention is applied to image acquisition for ID photos, images of Asian people with relatively standardized facial poses and no occlusions are used as training data.

[0043] See Figure 3 , Figure 3This is a flowchart illustrating the global thresholding process of the transparency prediction map in an embodiment of the image acquisition model training method for photographs according to the present invention. First, step S21 is executed to set a pixel threshold. The pixel threshold set in this invention is 120.

[0044] After setting the pixel threshold, step S22 is executed to determine whether each pixel in the transparency prediction map is greater than or equal to the pixel threshold. If a pixel in the transparency prediction map is greater than or equal to the pixel threshold, step S23 is executed to set the pixel value to 255. If a pixel in the transparency prediction map is less than the pixel threshold, step S24 is executed to set the pixel value to 0. The transparency prediction map after global thresholding can better separate the image from the background.

[0045] See Figure 4 , Figure 4 This is a flowchart of the photo data augmentation process in an embodiment of the image acquisition model training method of the present invention. First, step S41 is executed, using a face analysis model to perform facial analysis on the photo data, distinguishing parts such as hair, face, and clothing.

[0046] Then, step S42 is executed, where adjustments are made to each photo in the photo data to generate the first augmented photo data. Adjustments include enhancing brightness or contrast of the overall image or specific areas, whitening or darkening, adding shadows or highlights, etc. One or more adjustment operations are randomly selected for each photo. Each photo undergoes two adjustment operations, generating a total of 8000 first augmented photo data images.

[0047] After generating the first augmented photo data, step S43 is executed to divide the photo data into two groups of equal size: a first group and a second group. The first augmented photo data is then categorized into their respective groups. If the 4000 photos are divided into a first group and a second group, each containing 2000 photos, and the first augmented data is categorized into their respective groups, then both the first and second groups will contain 6000 photos.

[0048] After step S43 is completed, step S44 is executed, using a makeup transfer neural network model to transfer the makeup from the first set of photos to the second set of photos, generating the second augmented photo data. The second augmented photo data consists of 6000 photos.

[0049] After generating the second augmented photo data, step S45 is executed to swap the portraits from the second set of photos with those from the first set of photos, generating the third augmented photo data. The technique used to swap the portraits from the second set of photos with those from the first set of photos is image face swapping technology, and the third augmented photo data consists of 6000 photos.

[0050] After generating the third amplified photo data, step S46 is executed: a background image is selected, and the backgrounds of the photo data, the first amplified photo data, the second amplified photo data, and the third amplified photo data are replaced to generate the fourth amplified photo data. The total number of photos (photo data, first amplified photo data, second amplified photo data, and third amplified photo data) is 24,000. After replacing the backgrounds of these 24,000 photos, 24,000 new photos are obtained. Some of the clearer photos undergo multiple background replacements, resulting in 26,000 photos in the fourth amplified photo data set.

[0051] The background replacement method is as follows: Normalize the transparency prediction image and denote it as alpha; denote the amplified data as F; denote the background image as B; and denote the fourth amplified photo data as I. The formula for generating the fourth amplified photo data is then I = alpha * F + (1 - alpha) * B. A monotonous background image or a background image of an unoccupied scene is selected as the background image.

[0052] After executing step S46, step S47 is executed. The photo data, the first amplified photo data, the second amplified photo data, the third amplified photo data, and the fourth amplified photo data are combined to form amplified data, which consists of 50,000 images. By changing the texture, color, makeup, and other features of different parts of the portrait in the photo data, or by changing the background pattern of the photo data, the robustness of the segmentation model and the image acquisition model in distinguishing backgrounds can be improved when training the segmentation model and the image acquisition model. This allows the segmentation model and the image acquisition model to better utilize complex backgrounds and improve the accuracy of image acquisition from low-quality images.

[0053] Image acquisition method example:

[0054] Obtain the image to be acquired; segment the image to be acquired using a segmentation model to generate a mask image; concatenate the mask image with the image to be acquired to generate a four-channel image to be acquired; perform image acquisition operations on the four-channel image to be acquired using an image acquisition model.

[0055] When using an image acquisition model to acquire images from four channels, the image contains a mask image obtained using a segmentation model. This mask image can serve as prior information for distinguishing images, helping the image acquisition model eliminate a large amount of background interference information, increasing the accuracy of image acquisition, and thus improving the image acquisition model's ability to distinguish images.

[0056] Computer device embodiment:

[0057] The computer device in this embodiment includes a processor and a memory. The processor stores a computer program, which is executed by the processor to implement the above-described image acquisition model training method and image acquisition method.

[0058] Computer-readable storage media:

[0059] The image acquisition model training method and image acquisition method for photographs in a computer device described in the above embodiments can be stored in a computer-readable storage medium as a computer program. When the computer program is executed by a processor, it can complete the steps of the above embodiments of the image acquisition model training method and image acquisition method for photographs in a computer device. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0060] The above are merely preferred embodiments of the present invention, but the design concept of the invention is not limited thereto. Without departing from the concept of the present invention, many other equivalent embodiments may be included. Those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the protection scope of the present invention.

Claims

1. A method for training an image acquisition model for photographs, comprising: Collect photo data and input the photo data into the initial image acquisition model to obtain a transparency prediction map; The transparency prediction map is globally thresholded to generate a segmentation mask map; The photo data is then amplified to obtain amplified data; Using the amplified data as input and the segmentation mask image as a label, a segmentation model is trained based on a segmentation neural network; The trained segmentation model is used to segment the augmented data to obtain an inference mask image; The inference mask image is morphologically dilated to generate an inference mask morphology image. Its features are: The amplified data and its corresponding inference mask morphology map are concatenated to form a four-channel image; Using the four-channel image as input and the transparency prediction image as a label, an image acquisition model is trained based on an image acquisition neural network. The photo data is amplified to obtain amplified data including: The photo data was analyzed using a face analysis model. Adjustments are made to each image in the photo data to generate the first augmented photo data; The adjustment operations include enhancing the brightness and contrast of the overall image or a partial image, whitening or darkening the image, and adding shadows or glimmers. The photo data is divided into a first group of photos and a second group of photos with the same number of photos, and the first amplified photo data is classified into the corresponding group of photo data. Using a makeup transfer neural network model, the makeup from the first set of photos is transferred to the second set of photos to generate second augmented photo data; Replace the portraits in the second set of photos with those in the first set of photos to generate the third augmented photo data; Select a background image, and replace the background of the photo data, the first amplified photo data, the second amplified photo data, and the third amplified photo data to generate the fourth amplified photo data. The photo data, the first amplified photo data, the second amplified photo data, the third amplified photo data, and the fourth amplified photo data are combined to form amplified data.

2. The image acquisition model training method for photographs according to claim 1, characterized in that: The convolutional layer of the image acquisition neural network has four channels.

3. The image acquisition model training method for photographs according to claim 1 or 2, characterized in that: The transparency prediction map is globally thresholded to generate a segmentation mask map including: Set a pixel threshold; Determine whether each pixel in the transparency prediction map is greater than or equal to the pixel threshold; If a pixel in the transparency prediction map is greater than or equal to the pixel threshold, then the pixel is set to 255. If a pixel in the transparency prediction map is smaller than the pixel threshold, then that pixel is set to 0.

4. The image acquisition model training method for photographs according to claim 3, characterized in that: The collected photo data includes: Collecting portrait images from the Internet; Images that remove two or more faces using a face detection model; Remove images with two or more people using a human detection model; Images with three Euler angles greater than a preset angle threshold are removed using a head pose estimation model; Remove images of people wearing hats using a hat detection model; The occlusion detection model is used to remove faces from the image to obtain the photo data.

5. The image acquisition model training method for photographs according to claim 4, characterized in that: The process of generating the fourth amplified photo data by replacing the background of the photo data, the first amplified photo data, the second amplified photo data, and the third amplified photo data includes: The transparency prediction map is normalized and denoted as alpha, the amplified data is denoted as F, the background image is denoted as B, and the fourth amplified photo data is denoted as I; The formula for generating the fourth amplified image data is I = alpha F + (1-alpha) B.

6. The image acquisition model training method for photographs according to claim 5, characterized in that: The background image is a monotonous background or a background with no people.

7. An image acquisition method, characterized in that, include: Get the image to be retrieved; The image to be acquired is segmented using a segmentation model to generate a mask image; The mask image and the image to be acquired are concatenated by channels to generate a four-channel image to be acquired; The image acquisition model as described in any one of claims 1 to 6 is used to perform image acquisition operations on the four-channel images to be acquired.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the image acquisition model training method for photographs as described in any one of claims 1 to 6 and the image acquisition method as described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, it implements the image acquisition model training method for photographs as described in any one of claims 1 to 6 and the image acquisition method as described in claim 7.

Images