A method and system for changing the color of comic special effects images

Abstract

The present invention discloses a method and system for changing the color of a comic special effect image, comprising: obtaining an input original image, the original image being an image of the same type; using an AI comic algorithm to perform comic processing on the original image to obtain a first comic image; establishing a template image of the same type as the original image as a template image for color migration; using a semantic segmentation algorithm to segment the first comic image and the template image to obtain a first comic interest segmentation image and a template interest segmentation image; using a color migration algorithm to migrate the corresponding color of each interest region in the template interest style image to the corresponding interest region of the first comic interest segmentation image to obtain a second comic image. The comic template image is semantically segmented to generate a corresponding template region, and the color of the template region is migrated to the corresponding original image interest region using a color migration algorithm according to the template region, thereby realizing rapid comic special effect migration.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for changing the color of comic special effects images. Background Art

[0002] At present, various comic-style special effects are often involved in mobile apps or PC image processing software, which process users' real photos into comic styles. These special effects often have high market recognition and are loved by users, so they have high commercial value. However, the production of existing comic special effects is very complicated. Taking AI comic algorithms as an example, the research and development of a comic algorithm often requires many designers and algorithm developers to invest a lot of manpower and money in photo editing, model training, and parameter adjustment iteration. The time cycle is long and the effect is single. A comic often has only one color style, and the color style of the comic cannot be modified or customized. If you want to modify the style, you need to re-develop it, which is costly. Therefore, how to quickly generate comics in a variety of color styles to meet the needs of different groups of people has become a technical difficulty. Summary of the invention

[0003] One of the inventive purposes of the present invention is to provide a method and system for changing the color of comic special effects images. The method and system perform target cutting on the comic original image based on a semantic segmentation model, obtain different interest regions of the original image, and establish a comic template image. Semantic segmentation is performed based on the comic template image to generate a corresponding template region. A color migration algorithm is used based on the template region to migrate the color of the template region to the corresponding interest region of the original image, thereby realizing rapid comic special effects migration.

[0004] Another object of the present invention is to provide a method and system for changing the color of comic special effects images. The method and system first perform comic processing on real pictures using an AI comic algorithm, and then further perform semantic segmentation on the comic-processed images to reduce the problem of unclear interest boundaries caused by direct migration of the original image colors.

[0005] Another inventive object of the present invention is to provide a method and system for changing the color of comic special effects images. The method and system perform comic processing based on an AI comic algorithm, generate an image set with a similar style through training through the AI ​​comic algorithm, divide the comic set of the same style into regions of interest, and configure the same or similar template as the basis for color migration, so that the template color can be migrated quickly and accurately.

[0006] In order to achieve at least one of the above-mentioned invention objects, the present invention further provides a method for changing the color of a comic special effects image, the method comprising:

[0007] Obtaining an input original image, where the original image is an image of the same type;

[0008] Using an AI comic algorithm to process the original image into a comic image to obtain a first comic image;

[0009] Creating a template image of the same type as the original image as a template image for color migration;

[0010] Using a semantic segmentation algorithm to segment the first comic image and the template image to obtain a first comic interest segmentation image and a template interest segmentation image;

[0011] A color migration algorithm is used to migrate the corresponding color of each interest region in the template interest style image to the corresponding interest region of the first comic interest segmentation image to obtain a second comic image.

[0012] According to one of the preferred embodiments of the present invention, the method for semantic segmentation of the original image includes: setting the size of the network input image, setting three-channel RGB data, and outputting a three-channel RGB image of the same size as the input image, wherein corresponding semantic labels are respectively configured according to the segmented areas of each original image, and different RGB colors are configured for each semantic label area to construct a sample atlas for semantic segmentation, and a segmentation network model is used to train the sample atlas to obtain a trained semantic segmentation model of the corresponding style, and the target segmentation area of ​​the original image is obtained according to the trained semantic segmentation model to form a first comic interest segmentation image.

[0013] According to another preferred embodiment of the present invention, the template image segmentation method includes: obtaining a template image of the same type as the original image, inputting the template image into the trained semantic segmentation model, and outputting a template interest segmentation image having a corresponding target segmentation area.

[0014] According to another preferred embodiment of the present invention, after completing the semantic segmentation of the first comic image and the template image, a color migration operation of the target segmentation area is further performed, wherein the color migration operation includes: after obtaining the target segmentation area set of the first comic interest segmentation image, calculating the mean and variance of the pixel values ​​contained in each target segmentation area in the first comic interest segmentation image to the LAB color space component, and calculating the mean and variance of the pixel RGB values ​​contained in each target segmentation area in the template interest segmentation image to the LAB color space component.

[0015] According to another preferred embodiment of the present invention, the method for calculating the pixel values ​​of the target area of ​​the first comic interest segmentation image to the LAB color space components includes:

[0016]

[0017] L, M, and S are conversion parameters.

[0018] According to another preferred embodiment of the present invention, the method for calculating the mean and variance of the pixel values ​​contained in the first comic interest segmentation image to the LAB color space components includes:

[0019] K[n]=Sum(SetA[n])

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] Among them l s_mean , a s_mean , b s_mean is the mean value of the corresponding components of the pixel values ​​contained in the first comic interest segmentation image to the LAB color space; l s_var , a s_var , b s_var is the variance of the pixel values ​​contained in the first comic interest segmentation image to the corresponding components in the LAB color space, where (i, j) is defined as the pixel position of the target area, l(i, j) represents the L component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space; a(i, j) represents the A component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space, and b(i, j) represents the B component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space; where setA represents the target area set of the first comic interest cutting image, and n represents the number of target areas of the first comic interest cutting image.

[0027] According to another preferred embodiment of the present invention, the target area in the first comic interest segmentation image is traversed, each pixel value (l, a, b) in the target area is calculated, and the pixel value (l', a', b') after color migration is calculated:

[0028]

[0029]

[0030]

[0031] According to another preferred embodiment of the present invention, an RGB conversion formula is configured to further convert the pixel value (1', a', b') after color migration into a corresponding RGB value to obtain the second comic image that is successfully migrated and converted.

[0032] In order to achieve at least one of the above-mentioned invention purposes, the present invention further provides a comic special effects image color changing system, wherein the system executes the above-mentioned comic special effects image color changing method.

[0033] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the method for changing the color of a comic special effect image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Shown is a flow chart of a method for changing the color of a comic special effects image according to the present invention.

[0035] Figure 2 Shown is a schematic diagram of the target area after semantic segmentation of the original image processed by AI cartoons in the present invention.

[0036] Figure 3 Shown is a schematic diagram of the target area after semantic segmentation of the template image in the present invention.

[0037] Figure 4 Shown is a schematic diagram of a first comic image processed by AI comics in the present invention.

[0038] Figure 5 Shown is a schematic diagram of a template image in the present invention.

[0039] Figure 6 Shown is a schematic diagram of a second comic image after color migration in the present invention. DETAILED DESCRIPTION

[0040] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0041] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0042] Please combine Figure 1-Figure 6 The present invention discloses a method and system for changing the color of comic special effects images. The method mainly includes the following steps: first, it is necessary to select the image type, comic type and comic style, and use the AI ​​comic algorithm to process the original image into a comic, wherein the original image can be a real shot image, and the AI ​​comic algorithm can process the same type of photos into a comic, and obtain a comic image with the same or similar style of comic processing. Further, a template image with the same type and style as the original image is configured, the comic processed image and the simulated image are semantically segmented, and the segmented image is color migrated to finally obtain the template image with the required special effect color conversion.

[0043] For more information, see Figure 4 The present invention takes a face image as an example. The original image is a photographed image S, which is processed by the AI ​​comic algorithm to obtain the following Figure 4 The first comic picture is displayed, wherein the AI ​​comic algorithm includes but is not limited to CartoonGAN algorithm, Pix2Pix algorithm, StarGAN algorithm and StyleGAN algorithm, etc. The present invention takes Pix2Pix algorithm as an example: after obtaining the original face image, the designer draws a comic of the face image of the same type and style according to the face image, draws a batch of comic images of the same type and style based on the original image, generates a comic data set, and the comic data set contains at least 10,000 comic samples. The comic samples are input into the Pix2Pix algorithm for model training, and training parameters such as hyperparameters that meet the requirements of the loss function are obtained to obtain a trained Pix2Pix comic model. The facial photo after taking the photo is input into the trained Pix2Pix comic model to obtain the following. Figure 4 The first comic image displayed. It should be noted that, since the comic image is processed based on the comic image of the original image of the same type and style, the output first comic image also has the same or similar comic style.

[0044] Furthermore, after obtaining the first comic image after the comic processing of the original image, a semantic segmentation algorithm is further used to perform semantic segmentation on the first comic image, wherein the semantic segmentation algorithm includes but is not limited to the Unet algorithm, PSPNet algorithm, SegNet algorithm, DeepLab algorithm, E-Net algorithm, etc., wherein the above semantic segmentation algorithm is an existing public algorithm, and the present invention will not be described in detail. In order to better illustrate the technical effect of the present invention, the present invention takes the Unet semantic segmentation algorithm as an example, and the method includes the following steps: setting the input image size of the Unet semantic segmentation network, for example, a 256×256 size image can be set, and the three-channel RGB data is set, so that the image passing through the semantic segmentation network is a 256×256×3 color three-channel image. Further configure the semantic label, and divide the original image into 4 target areas through the semantic segmentation algorithm, namely 1, portrait skin target area; portrait hair target area; 3, portrait clothing target area; 4, portrait background target area. The four target areas are further configured with corresponding semantic labels and RGB color values, wherein the RGB color configurations of the four target areas can be respectively configured as target area 1: RGB (255, 0, 0); target area 2: RGB (0, 255, 0); target area 3: RGB (0, 0, 255); target area 4: RGB (0, 0, 0). The semantic labels corresponding to target areas 1-4 are portrait skin, portrait hair, portrait clothes and portrait background, respectively. After all sample images in the drawn data set are configured with semantic labels, the Unet network is used to input the drawn data set samples for semantic segmentation training. According to the trained semantic segmentation model, the following is obtained: Figure 2 The first comic interest segmentation image based on the target area set of the original image is shown.

[0045] After completing the semantic segmentation of the first comic image corresponding to the original image, a template image is further obtained for semantic segmentation, wherein the template image is a comic image of the same type and style as the first comic image. For example, in the present invention, the first comic image is a female portrait, and the corresponding template image of the same type and style is also a portrait (such as Figure 5 Display), wherein the template image is input into a pre-trained Unet semantic segmentation model with the same semantic segmentation as the first comic image, to obtain a template interest segmentation image after the semantic segmentation of the template image.

[0046] in Figure 3What is shown is a comparative schematic diagram of the template image after semantic segmentation of the present invention. In the template interest segmentation image, target area 1 is the corresponding portrait hair label; target area 2 is the corresponding portrait skin label; target area 3 is the portrait clothing label, and target area 4 is the portrait background label. In some preferred embodiments of the present invention, by setting a custom migration strategy, the colors of different areas in the template interest segmentation image are replaced, and a color migration operation is further performed after the replacement.

[0047] In order to better illustrate the technical effect of the present invention, the present invention further provides an image target area color migration algorithm based on semantic segmentation, the color migration algorithm comprises the following steps: obtaining a target area set in the first comic interest segmentation map: SetA = {hair, skin, clothes, background}, further calculating the mean value l of the pixel RGB value corresponding to the LAB color space component of each target area in SetA in map A s_mean , a s_mean , b s_mean and variance l s_var , a s_var , b s_var ; Where l, a, b represent different components of the corresponding target area in the LAB color space. The calculation method for converting the pixel RGB value into the LAB color space component includes:

[0048]

[0049] Where L, M, S are conversion parameters. The method for calculating the mean and variance of the pixel RGB values ​​contained in each target area in SetA in Figure A corresponding to the LAB color space components is:

[0050] K[n]=Sum(SetA[n])

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] Wherein n represents the number of targets in the target area set, and there are 4 target areas in the first comic interest cut image, representing the hair, skin, clothing and background target areas respectively; further define (i, j) represents the pixel position, l(i, j) represents the L component value of the pixel at position (i, j) in the first comic interest cut image in the LAB color space, a(i, j) represents the A component value of the pixel at position (i, j) in the first comic interest cut image in the LAB color space, and b(i, j) represents the B component value of the pixel at position (i, j) in the first comic interest cut image in the LAB color space; wherein setA represents the target area set of the first comic interest cut image, and n represents the number of target areas of the first comic interest cut image.

[0058] For the template interest segmentation image M, the corresponding target area set is SetM = {hair, skin, clothes, background}, and the mean value l of the pixel RGB value corresponding to the LAB color space component in the template interest segmentation image M is further calculated. m_mean , a m_mean , b m_mean and variance l m_var , a m_var , b m_var The calculation method of the mean value and method of the LAB color space component corresponding to the RGB value of the SetM pixel is the same as the mean value of the first comic image, and the present invention will not repeat it again.

[0059] After completing the calculation of the mean and variance of the LAB color space components corresponding to the segmented areas of the first comic interest segmentation area and the template interest image, for each first comic interest segmentation area n∈{hair, skin, clothing, background}, traverse the pixel (l, a, b) values ​​of the corresponding SetA[n] area (i, j) position in the first comic interest segmentation area map, and calculate the pixel value (l', a', b') after the color of the target area of ​​the template image is migrated to the target area of ​​the first comic image according to the color migration algorithm. The method for calculating the pixel value after migration is:

[0060]

[0061]

[0062]

[0063] The pixel values ​​(l', a', b') after color migration are further converted to RGB values, as shown in the following figure: Figure 6 The final output of the second comic effect is shown, where the formula for converting pixel values ​​(l', a', b') to RGB values ​​is:

[0064]

[0065]

[0066] In the above conversion formula, L, M, and S are conversion parameters, and the template image can be a customized image of the same type. The technical effects of the above complete technical solution of the present invention include: rapid color migration of the original image can be achieved through intelligent AI comic processing and semantic segmentation. There is no need to repeatedly train the comic model, and the resource consumption of designers retouching and algorithm engineers training is greatly reduced, making the above method and system more adaptable.

[0067] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wire segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0068] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be deformed or modified in any way.

Claims

1. A method for changing the color of a comic special effect image, characterized in that: The method comprises: Obtaining an input original image, where the original image is an image of the same type; Using an AI comic algorithm to process the original image into a comic image to obtain a first comic image; Creating a template image of the same type as the original image as a template image for color migration; Using a semantic segmentation algorithm to segment the first comic image and the template image to obtain a first comic interest segmentation image and a template interest segmentation image; Using a color migration algorithm, migrate the corresponding color of each interest region in the template interest style image to the corresponding interest region of the first comic interest segmentation image to obtain a second comic image; The method for semantically segmenting the original image includes: setting the size of the network input image, setting three-channel RGB data, and outputting a three-channel RGB image of the same size as the input image, wherein corresponding semantic labels are respectively configured according to the segmented areas of each original image, and different RGB colors are configured for each semantic label area to construct a sample atlas of semantic segmentation, and a segmentation network model is used to train the sample atlas to obtain a trained semantic segmentation model of the corresponding style, and a target segmentation area of ​​the original image is obtained according to the trained semantic segmentation model to form a first comic interest segmentation image; The template image segmentation method comprises: obtaining a template image of the same type as the original image, inputting the template image into the trained semantic segmentation model, and outputting a template interest segmentation image having a corresponding target segmentation area; After completing the semantic segmentation of the first comic image and the template image, a color migration operation of the target segmentation area is further performed, wherein the color migration operation includes: after obtaining the target segmentation area set of the first comic interest segmentation image, calculating the mean and variance of the pixel values ​​contained in each target segmentation area in the first comic interest segmentation image to the LAB color space component, and calculating the mean and variance of the pixel RGB values ​​contained in each target segmentation area in the template interest segmentation image to the LAB color space component.

2. A method for changing the color of comic special effects images according to claim 1, characterized in that: The method for calculating the pixel values ​​of the target area of ​​the first comic interest segmentation image into LAB color space components includes: L, M, and S are conversion parameters.

3. A method for changing the color of comic special effects images according to claim 2, characterized in that: The method for calculating the mean and variance of the pixel values ​​contained in the first comic interest segmentation image to the LAB color space components includes: Among them l s_mean , a s_mean , b s_mean is the mean value of the corresponding components of the pixel values ​​in the first comic interest segmentation image to the LAB color space; l s_var , a s_var , b s_var is the variance of the pixel values ​​contained in the first comic interest segmentation image to the corresponding components in the LAB color space, where l s_mean , a s_mean , b s_mean is the mean value of the corresponding components of the pixel values ​​in the first comic interest segmentation image to the LAB color space; l s_var , a s_var , b s_var is the variance of the pixel values ​​contained in the first comic interest segmentation image to the corresponding components in the LAB color space, where (i, j) is defined as the pixel position of the target area, l(i, j) represents the L component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space; a(i, j) represents the A component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space, and b(i, j) represents the B component value of the pixel at position (i, j) in the first comic interest cutting image in the LAB color space; where setA represents the target area set of the first comic interest cutting image, and n represents the number of target areas of the first comic interest cutting image.

4. A method for changing the color of comic special effects images according to claim 3, characterized in that: Traverse the target area in the first comic interest segmentation image, calculate each pixel value (l, a, b) in the target area, and calculate the pixel value (l', a', b') after color migration:

5. A method for changing the color of comic special effects images according to claim 4, characterized in that: An RGB conversion formula is configured to further convert the pixel value (1', a', b') after the color migration is completed into a corresponding RGB value, so as to obtain the second comic image that is finally successfully migrated and converted.

6. A comic special effects image color changing system, characterized in that: The system executes a comic special effects image color changing method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement a comic special effect image color changing method as described in any one of claims 1 to 5.

Images