A method for generating stick figures based on artificial intelligence

Abstract

The present invention discloses a method for generating stick figures based on artificial intelligence, and relates to the field of artificial intelligence technology. The present invention greatly reduces the difficulty of data collection by adopting unpaired training data as training samples for deep learning. At the same time, shared data sets or data on the Internet can be used as training samples for training, and more challenging photos can be included in the training set, thereby expanding the training set and enriching the data categories. The problem of paired training data is solved, and the effect of stick figure generation is greatly improved, which provides strong data support for the algorithm for generating stick figures. It is also one of the important reasons why the present invention finally achieves good results. The present invention enhances the generation performance of facial features. On the basis of the original network, multiple local discriminant networks are used, including local discriminant networks for eyes, nose and lips, to enhance the expression of regional information, and well preserve all important facial features in the generated portrait.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and in particular relates to a method for generating stick figures based on artificial intelligence. Background Art

[0002] Stick figure drawing is a common form of artistic expression, depicting human perception of nature with a high degree of generalization and abstraction. In recent years, with the rapid development of artificial intelligence (AI), integrating AI technology with daily life has become a widespread trend. Many scholars have attempted to apply AI technology to artistic creation, such as the production of stylized paintings and the transfer of color, light, and shadow styles.

[0003] However, despite numerous similar applications in the industry and promising results, using AI technology to automatically generate stick figure algorithms is a bold and challenging endeavor. The difficulties lie in: 1) Deep learning struggles to distinguish between necessary lines and interference from complex gradient transformations; 2) The resulting stick figure lines are overly complex; 3) The effects of light changes and uneven lighting in real scenes; and 4) background interference and overlap. Existing technologies suffer from the following shortcomings:

[0004] A: Currently, many algorithms can achieve diverse styles of painting, but algorithms for generating simple facial portraits are rare and often produce unsatisfactory results. For example, facial portraits have unsmooth contours, incomplete facial features, overly complex lines, lack of simplicity and generalization, and are affected by uneven lighting.

[0005] B: Stick figures are highly summarized and presented as representations of the original painting. Existing AI algorithms are often trained using paired data. This is not only expensive to collect, but also suffers from the imbalance between the outline (foreground) and background of the stick figure, which can lead to loss of important facial features. Therefore, limited by the amount of data and data imbalance, these methods often fail to adapt well to diverse real-world scenarios. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for generating stick figures based on artificial intelligence. By adopting an unpaired data training method from face photos to generate stick figures, the problems existing in paired training data are solved and the stick figure generation effect is greatly improved.

[0007] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0008] The present invention is a method for generating stick figures based on artificial intelligence. CycleGAN is used as the backbone network of the stick figure generation algorithm. Unpaired training data is used as training samples for deep learning. The method specifically includes the following steps:

[0009] Step SS01: Obtain original character portrait data;

[0010] Step SS02: pre-processing the portrait data;

[0011] Step SS03: Generate a stick figure according to the stick figure model;

[0012] Step SS04: Generate a stick figure image;

[0013] The network in the stick figure model is trained in an adversarial manner. The unpaired method does not require the alignment of the photo and the original drawing, only that the stick figure style remain consistent. This greatly reduces the difficulty of data collection. At the same time, data from a common dataset or the Internet can be used as training samples for training, and more challenging photos can be included in the training set, expanding the training set while enriching the data categories. This provides strong data support for the stick figure generation algorithm of this method and is one of the important reasons why the present invention ultimately achieves good results. The method for preparing the dataset during training is as follows:

[0014] The face area is obtained by face detection algorithm for photos of people in real scenes, and the face image is cropped by expanding the center of the face area.

[0015] Key point detection algorithm is used to detect key point information, and key points are used to locate the eye, nose, and mouth areas, and image masks are created for the eye, nose, and mouth areas respectively;

[0016] Draw the corresponding stick figure data according to the cropped face image.

[0017] Furthermore, the face area is a rectangular area, and the face area includes a face area 1 and a face area 2;

[0018] The facial region 1 is defined by the hairline, chin, and ears;

[0019] The second facial area is bounded by the hairline, shoulders, and ears.

[0020] Furthermore, the method of enlarging the area and cropping the face image based on the center of the face area is as follows:

[0021] Step C001: Obtain the center points of face region 1 and face region 2, and mark them as center point Z1 and center point Z2 respectively;

[0022] Step C002: Determine the relative position of the nose tip and the center point Z1;

[0023] If the nose tip is above the horizontal line where the center point Z1 is located, the upper edge line of face area 1 and the line connecting the two ends of the upper edge line of face area 1 to the lower edge line of face area 2 are used as boundaries to enclose a triangular area;

[0024] If the nose tip is below the horizontal line where the center point Z1 is located, the triangle area is defined by the line connecting the upper edge line of face area 1 to edge line 1 and the two ends of edge line 1 to the lower edge line of face area 2.

[0025] Step C003: cropping the face image according to the triangular area circled in step C002.

[0026] Furthermore, this invention enhances the generation of facial features. The depiction of facial features has a significant impact on the overall painting, and can even determine the success or failure of a painting. We found that if only the global network is used, the facial features are easily affected by environmental lighting, angles, shadows, and other factors, resulting in missing, distorted, inconsistent, and unsightly lines in the facial features. Therefore, this method uses multiple local discriminant networks based on the original network, including local discriminant networks for the eyes, nose, and lips, to enhance the expression of regional information. Our method effectively preserves all important facial features in the generated portrait image, including key points such as positioning key points and auxiliary key points:

[0027] The key points of the positioning are: each eye, nose, and mouth in the face image is covered with a rectangular area; the boundary line of each rectangular area is tangent to the outermost edge points of the eye, nose, and mouth, and the rectangular area is defined as the rectangle around the eye, rectangle around the nose, and rectangle around the mouth; the four corner points of each rectangular area are used as key points;

[0028] Auxiliary key points include: center point Z1, center point Z2;

[0029] The auxiliary key points also include the intersections of the edge lines of the rectangle around the eyes, the rectangle around the nose, and the rectangle around the mouth with the diagonals of the face area 1 and the face area 2, respectively.

[0030] Furthermore, the key point information is obtained by key point detection algorithm, and the method of using key points to locate the eye, nose, and mouth areas is as follows:

[0031] Step J00: Acquire a face image;

[0032] Step J01: Detect the rectangles around the eyes, nose, and mouth, and obtain the four corner points corresponding to the rectangles around the eyes, nose, and mouth as positioning key points;

[0033] Step J02: Obtain face region 1 and face region 2, and detect the intersection points of center point Z1, center point Z2, and the edge lines of the rectangles around the eyes, nose, and mouth, with the diagonals of face region 1 and face region 2, respectively, as auxiliary key points;

[0034] Step J03: Establish a rectangular coordinate system with the center point of the lower edge line of the face area as the origin;

[0035] Step J04: Obtain the position of each positioning key point and auxiliary key point in the rectangular coordinate system, as well as the relative position between each positioning key point and the auxiliary key point;

[0036] Step J05: Determine the eye, nose, and mouth regions based on the positioning key points, and determine the relative positions of the eye, nose, and mouth regions based on the relative positions between each positioning key point and the auxiliary key points.

[0037] Furthermore, the relative position is represented by a vector between the positioning key point and the auxiliary key point.

[0038] Furthermore, the stick figure model includes:

[0039] Generator J, which is used to convert face photos into stick figures;

[0040] Generator R, which is used to convert stick figures into face photos;

[0041] Discriminator P_j, which is used to distinguish generated stick figures from real stick figures;

[0042] The discriminator P_r is used to distinguish between generated photos and real photos;

[0043] Four local discriminators are also included:

[0044] Discriminators P_y1, P_y2, P_b, and P_z are used to discriminate the images of the two eyes, nose, and mouth, respectively, to enhance the quality of facial feature image generation.

[0045] The network structure of Discriminator P_j, Discriminator P_r, Discriminator P_y1, Discriminator P_y2, Discriminator P_b, and Discriminator P_z is based on the PatchGAN network.

[0046] Furthermore, the generator J is an encoder-decoder including a residual module, and the method by which the generator J converts a face photo into a stick figure is:

[0047] Step 1: Encode the face photo and extract features through a convolutional layer and two downsampling modules;

[0048] Step 2: Construct features through 9 residual modules with the same structure and convert them to the eye, nose, and mouth regions;

[0049] In step three, the output graph is then reconstructed through two upsampling modules and a final convolutional layer.

[0050] Furthermore, the network in the stick figure model is trained in an adversarial manner, and the training method includes the following steps:

[0051] Step S1: obtaining original character portrait data;

[0052] Step S2: pre-processing the portrait data;

[0053] Step S3: Create a training set and create image masks for the eye, nose, and mouth regions respectively;

[0054] Step S4: drawing corresponding stick figure data;

[0055] Step S5: training to generate a stick figure model.

[0056] Furthermore, the discriminators P_j, P_r, P_y1, P_y2, P_b, and P_z are trained to maximize the probability of the real photos and the synthesized stick figures.

[0057] Generators G and F are trained to minimize the probability of assigning the correct label.

[0058] The present invention has the following beneficial effects:

[0059] By using unpaired training data as training samples for deep learning, this method significantly reduces the difficulty of data acquisition. It can also use shared datasets or online data as training samples, and incorporate more challenging photos into the training set, expanding the training set and enriching the data types. This provides strong data support for the algorithm used to generate stick figures, and is one of the key reasons for the method's ultimately successful results.

[0060] This invention enhances the generation performance of facial features. Based on the original network, multiple local discriminant networks are used, including local discriminant networks for eyes, nose and lips, to enhance the expression of regional information and well preserve all important facial features in the generated portrait image.

[0061] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 This is the flowchart of the present invention's stick figure drawing algorithm training;

[0064] Figure 2 is a schematic diagram of an embodiment;

[0065] Figure 3 A schematic diagram of an embodiment. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] In the description of the present invention, it should be understood that the terms "left", "upper", "lower", "right", "edge", "tangent", etc., which indicate orientation or positional relationships, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0068] See also Figure 1 As shown, the present invention is a method for generating stick figures based on artificial intelligence, using CycleGAN as the backbone network of the stick figure production algorithm and using unpaired training data as training samples for deep learning. The method specifically includes the following steps:

[0069] Step SS01: Obtain original character portrait data;

[0070] Step SS02: pre-processing the portrait data;

[0071] Step SS03: Generate a stick figure according to the stick figure model;

[0072] Step SS04: Generate a stick figure image;

[0073] The network in the stick figure model is trained in an adversarial manner. The unpaired method does not require the alignment of the photo and the original drawing, only that the stick figure style remain consistent. This greatly reduces the difficulty of data collection. At the same time, data from a common dataset or the Internet can be used as training samples for training, and more challenging photos can be included in the training set, expanding the training set while enriching the data categories. This provides strong data support for the stick figure generation algorithm of this method and is one of the important reasons why the present invention ultimately achieves good results. The method for preparing the dataset during training is as follows:

[0074] The face area is obtained by face detection algorithm for photos of people in real scenes, and the face image is cropped by expanding the center of the face area.

[0075] Key point detection algorithm is used to detect key point information, and key points are used to locate the eye, nose, and mouth areas, and image masks are created for the eye, nose, and mouth areas respectively;

[0076] Draw the corresponding stick figure data according to the cropped face image.

[0077] As an embodiment provided by the present invention, preferably, Figure 2 、 3 As shown, the face area is a rectangular area, and the face area includes a face area 1 and a face area 2;

[0078] The facial region 1 is defined by the hairline (the highest or lowest point of the edge line between the hair and the face), the chin, and the ears.

[0079] The second facial area is bounded by the hairline, shoulders, and ears.

[0080] As an embodiment provided by the present invention, preferably, a method for expanding the center of the face region to crop the face image is as follows:

[0081] Step C001: Obtain the center points of face region 1 and face region 2, and mark them as center point Z1 and center point Z2 respectively;

[0082] Step C002: Determine the relative position of the nose tip and the center point Z1;

[0083] If the nose tip is above the horizontal line where the center point Z1 is located, the upper edge line of face area 1 and the line connecting the two ends of the upper edge line of face area 1 to the lower edge line of face area 2 are used as boundaries to enclose a triangular area;

[0084] If the nose tip is below the horizontal line where the center point Z1 is located, the triangle area is defined by the line connecting the upper edge line of face area 1 to edge line 1 and the two ends of edge line 1 to the lower edge line of face area 2.

[0085] Step C003: cropping the face image according to the triangular area circled in step C002.

[0086] As an embodiment provided by the present invention, preferably, the present invention enhances the generation performance of facial features. The depiction of facial features has a huge impact on the overall painting and can even determine the success or failure of a painting. We found that if only the global network is used, the facial features information of the face is easily affected by environmental lighting, angles, shadows, etc., resulting in missing, disordered, inconsistent with reality, and unsightly lines generated in the facial features. Therefore, this method uses multiple local discriminant networks based on the original network, including local discriminant networks for the eyes, nose, and lips, to enhance the expression of regional information. Our method well preserves all important facial features in the generated portrait image. Key points include positioning key points and auxiliary key points:

[0087] The key points of the positioning are: each eye, nose, and mouth in the face image is covered with a rectangular area; the boundary line of each rectangular area is tangent to the outermost edge points of the eye, nose, and mouth, and the rectangular area is defined as the rectangle around the eye, rectangle around the nose, and rectangle around the mouth; the four corner points of each rectangular area are used as key points;

[0088] Auxiliary key points include: center point Z1, center point Z2;

[0089] The auxiliary key points also include the intersections of the edge lines of the rectangle around the eyes, the rectangle around the nose, and the rectangle around the mouth with the diagonals of the face area 1 and the face area 2, respectively.

[0090] As an embodiment provided by the present invention, preferably, the key point information is obtained by detecting with a key point detection algorithm, and the method of using the key points to locate the eye, nose, and mouth areas is as follows:

[0091] Step J00: Acquire a face image;

[0092] Step J01: Detect the rectangles around the eyes, nose, and mouth, and obtain the four corner points corresponding to the rectangles around the eyes, nose, and mouth as positioning key points;

[0093] Step J02: Obtain face region 1 and face region 2, and detect the intersection points of center point Z1, center point Z2, and the edge lines of the rectangles around the eyes, nose, and mouth, with the diagonals of face region 1 and face region 2, respectively, as auxiliary key points;

[0094] Step J03: Establish a rectangular coordinate system with the center point of the lower edge line of the face area as the origin;

[0095] Step J04: Obtain the position of each positioning key point and auxiliary key point in the rectangular coordinate system, as well as the relative position between each positioning key point and the auxiliary key point;

[0096] Step J05: Determine the eye, nose, and mouth regions based on the positioning key points, and determine the relative positions of the eye, nose, and mouth regions based on the relative positions between each positioning key point and the auxiliary key points.

[0097] As an embodiment provided by the present invention, preferably, the relative position is represented by a vector between the positioning key point and the auxiliary key point.

[0098] As an embodiment provided by the present invention, preferably, the stick figure model includes:

[0099] Generator J, which is used to convert face photos into stick figures;

[0100] Generator R, which is used to convert stick figures into face photos;

[0101] Discriminator P_j, which is used to distinguish generated stick figures from real stick figures;

[0102] The discriminator P_r is used to distinguish between generated photos and real photos;

[0103] Four local discriminators are also included:

[0104] Discriminators P_y1, P_y2, P_b, and P_z are used to discriminate the images of the two eyes, nose, and mouth, respectively, to enhance the quality of facial feature image generation.

[0105] The network structure of Discriminator P_j, Discriminator P_r, Discriminator P_y1, Discriminator P_y2, Discriminator P_b, and Discriminator P_z is based on the PatchGAN network.

[0106] As an embodiment provided by the present invention, preferably, the generator J is an encoder-decoder including a residual module, and the method by which the generator J converts a face photo into a stick figure is:

[0107] Step 1: Encode the face photo and extract features through a convolutional layer and two downsampling modules;

[0108] Step 2: Construct features through 9 residual modules with the same structure and convert them to the eye, nose, and mouth regions;

[0109] In step three, the output graph is then reconstructed through two upsampling modules and a final convolutional layer.

[0110] As an embodiment provided by the present invention, preferably, the network in the stick figure model is trained in an adversarial manner, and the training method includes the following steps:

[0111] Step S1: obtaining original character portrait data;

[0112] Step S2: pre-processing the portrait data;

[0113] Step S3: Create a training set and create image masks for the eye, nose, and mouth regions respectively;

[0114] Step S4: drawing corresponding stick figure data;

[0115] Step S5: training to generate a stick figure model.

[0116] As an embodiment provided by the present invention, preferably, the discriminator P_j, the discriminator P_r, the discriminator P_y1, the discriminator P_y2, the discriminator P_b, and the discriminator P_z are trained to maximize the probability of the real photo and the synthesized stick figure;

[0117] Generators G and F are trained to minimize the probability of assigning the correct label.

[0118] A method for generating stick figures based on artificial intelligence (AI) greatly reduces the difficulty of data collection by using unpaired training data as training samples for deep learning. It can also use shared datasets or data from the Internet as training samples for training, and incorporates more challenging photos into the training set, expanding the training set while enriching the data categories. This provides strong data support for the algorithm used to generate stick figures and is one of the important reasons for the good results achieved by the present invention. The present invention enhances the generation performance of facial features. Based on the original network, multiple local discriminant networks are used, including local discriminant networks for the eyes, nose, and lips, to enhance the expression of regional information and well preserve all important facial features in the generated portrait.

[0119] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0120] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating stick figures based on artificial intelligence, characterized in that: The method specifically comprises the following steps: Step SS01: Obtain original character portrait data; Step SS02: pre-processing the portrait data; Step SS03: Generate a stick figure according to the stick figure model; Step SS04: Generate a stick figure image; The network in the stick figure model is trained in an adversarial manner, and the method for preparing the training data set is as follows: The face area is obtained by face detection algorithm for photos of people in real scenes, and the face image is cropped by expanding the center of the face area. Key point detection algorithm is used to detect key point information, and key points are used to locate the eye, nose, and mouth areas, and image masks are created for the eye, nose, and mouth areas respectively; Draw the corresponding stick figure data according to the cropped face image; The face area is a rectangular area, and the face area includes a face area 1 and a face area 2; The facial region 1 is defined by the hairline, chin, and ears; The second facial area is bounded by the hairline, shoulders, and ears; The method of expanding the area and cropping the face image based on the center of the face area is: Step C001: Obtain the center points of face region 1 and face region 2, and mark them as center point Z1 and center point Z2 respectively; Step C002: Determine the relative position of the nose tip and the center point Z1; If the nose tip is above the horizontal line where the center point Z1 is located, the upper edge line of face area 1 and the line connecting the two ends of the upper edge line of face area 1 to the lower edge line of face area 2 are used as boundaries to enclose a triangular area; If the nose tip is below the horizontal line where the center point Z1 is located, the triangle area is defined by the line connecting the upper edge line of face area 1 to edge line 1 and the two ends of edge line 1 to the lower edge line of face area 2. Step C003: cropping the face image based on the triangular area circled in step C002; Key points include positioning key points and auxiliary key points: The key points of the positioning are: each eye, nose, and mouth in the face image is covered with a rectangular area; the boundary line of each rectangular area is tangent to the outermost edge points of the eye, nose, and mouth, and the rectangular area is defined as the rectangle around the eye, rectangle around the nose, and rectangle around the mouth; the four corner points of each rectangular area are used as key points; Auxiliary key points include: center point Z1, center point Z2; Auxiliary key points also include the intersection points of the edge lines of the rectangle around the eyes, the rectangle around the nose, and the rectangle around the mouth with the diagonals of the face area 1 and the face area 2 respectively; The key point information is obtained through key point detection algorithm. The method of using key points to locate the eye, nose and mouth areas is as follows: Step J00: Acquire a face image; Step J01: Detect the rectangles around the eyes, nose, and mouth, and obtain the four corner points corresponding to the rectangles around the eyes, nose, and mouth as positioning key points; Step J02: Obtain face region 1 and face region 2, and detect the intersection points of center point Z1, center point Z2, and the edge lines of the rectangles around the eyes, nose, and mouth, with the diagonals of face region 1 and face region 2, respectively, as auxiliary key points; Step J03: Establish a rectangular coordinate system with the midpoint of the lower edge of the face area as the origin; Step J04: Obtain the position of each positioning key point and auxiliary key point in the rectangular coordinate system, as well as the relative position between each positioning key point and the auxiliary key point; Step J05: Determine the eye, nose, and mouth regions based on the positioning key points, and determine the relative positions of the eye, nose, and mouth regions based on the relative positions between each positioning key point and the auxiliary key points; The relative position is represented by a vector between the positioning key point and the auxiliary key point.

2. The method for generating stick figures based on artificial intelligence according to claim 1, wherein: Stick figure models include: Generator J, which is used to convert face photos into stick figures; Generator R, which is used to convert stick figures into face photos; Discriminator P_j, which is used to distinguish generated stick figures from real stick figures; The discriminator P_r is used to distinguish between generated photos and real photos; Four local discriminators are also included: Discriminator P_y1, discriminator P_y2, discriminator P_b, and discriminator P_z are used to discriminate the images of two eyes, nose, and mouth respectively; The network structure of Discriminator P_j, Discriminator P_r, Discriminator P_y1, Discriminator P_y2, Discriminator P_b, and Discriminator P_z is based on the PatchGAN network.

3. The method for generating stick figures based on artificial intelligence according to claim 2, wherein: The generator J is an encoder-decoder including a residual module. The method of the generator J to convert a face photo into a stick figure is as follows: Step 1: Encode the face photo and extract features through a convolutional layer and two downsampling modules; Step 2: Construct features through 9 residual modules with the same structure and convert them to the eye, nose, and mouth regions; In step three, the output graph is then reconstructed through two upsampling modules and a final convolutional layer.

4. The method for generating stick figures based on artificial intelligence according to claim 3, wherein: The network in the stick figure model is trained in an adversarial manner, and the training method includes the following steps: Step S1: obtaining original character portrait data; Step S2: pre-processing the portrait data; Step S3: Create a training set and create image masks for the eye, nose, and mouth regions respectively; Step S4: drawing corresponding stick figure data; Step S5: training and generating a stick figure model.

5. The method for generating stick figures based on artificial intelligence according to claim 4, wherein: Train the discriminators P_j, P_r, P_y1, P_y2, P_b, and P_z to maximize the probability of real photos and synthetic stick figures; Generators G and F are trained to minimize the probability of assigning the correct label.

Images