Learning model generation method, illustration conversion method, learning model generation device, illustration conversion device, learning model generation program, and illustration conversion program

Abstract

A trained model generation method according to one embodiment, which is executed by an illustration conversion device, comprises: acquiring input data; extracting style feature data and content feature data pertaining to the input data by using a first neural network; identifying a second habit label from the style feature data by using a second neural network; training parameters that are used by the first and second neural networks, through reverse propagation, on the basis of a first loss function calculated from the input data and the second habit label; generating habit-removing illustration data from at least the content feature data; and training the parameters that are used by the first neural network, through reverse propagation, on the basis of a second loss function calculated from the habit-removing illustration data and the input data.

Description

[Technical Field]

[0001] The present invention relates to a learning model generation method, an illustration conversion method, a learning model generation device, an illustration conversion device, a learning model generation program, and an illustration conversion program. [Background technology]

[0002] Beginner users who have just begun learning how to create illustrations may create line drawings. Such users often have little or no desire to receive feedback (evaluation) from experts on their illustrations. Without expert feedback, the illustrations created by users often end up with quirks. Even if users repeatedly practice with quirks, it is difficult for them to reach an advanced level. Therefore, there is a technology that automatically corrects illustrations created by users.

[0003] For example, Non-Patent Document 1 proposes a technique for automatically drawing illustrations from rough sketches.

[0004] Non-Patent Document 2 proposes a technique for creating a mask on a part of an illustration created by a user and performing image interpolation on that mask. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Edgar Simo-Serra et al. “Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup”, ACM Transactions on Graphics(SIGGRAPH), 2016 [Non-patent document 2] Keita Awane et al., "Supporting Line Drawing Creation by Image Completion of Mask Missing Areas," 3rd Comic Engineering Research Meeting, March 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] The quality of the lines in the illustrations generated using Non-Patent Document 1 or Non-Patent Document 2 is low, so there is a problem that they cannot be applied to actual line drawings of illustrations.

[0007] Furthermore, illustrations are often evaluated subjectively based on the shape, breaks, and sharpness of the lines, and no clean line modeling is performed. Therefore, there is a problem that the quirks of illustrations created by users can only be evaluated by experts and cannot be evaluated by applications that edit user illustrations.

[0008] This invention was made with the above-mentioned circumstances in mind, and its purpose is to provide a technology that can capture the tendencies of illustrations created by users and automatically display illustrations with beautiful lines that correspond to those tendencies. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, one aspect of the present invention is a learning model generation method executed by an illustration conversion device, in which a data acquisition unit of the illustration conversion device acquires input data including quirk illustration data, a first quirk label corresponding to the quirk illustration data, training illustration data, and correct answer illustration data; a feature extraction unit of the illustration conversion device uses a first neural network to extract style feature data that characterizes the individuality including the user's drawing style, and content feature data that characterizes the training illustration data; a quirk label identification unit of the illustration conversion device identifies a second quirk label from the style feature data using a second neural network; and a classification result processing unit of the illustration conversion device calculates a first loss function based on the first quirk label and the second quirk label. the identification result processing unit learns parameters to be used by the first neural network and the second neural network by back propagation based on the first loss function; the quirk removal illustration generation unit of the illustration conversion device generates quirk removal illustration data from at least the content feature data; the quirk removal illustration generation unit calculates a second loss function based on the quirk removal illustration data and the correct illustration data; the quirk removal illustration generation unit learns parameters to be used by the first neural network by back propagation based on the second loss function; and the memory unit of the illustration conversion device stores the parameters to be used by the first neural network and the second neural network. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to capture the tendencies of a line drawing created by a user and automatically correct the created line drawing to create clean lines in accordance with the tendencies. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 is a block diagram showing an example of a hardware configuration of an illustration conversion device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of the illustration conversion device according to the embodiment in relation to the hardware configuration shown in FIG. [Figure 3] FIG. 3 is a block diagram showing feedback for performing parameter learning of the neural network used by each unit of the illustration conversion device in this embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of a learning operation for illustration conversion by the illustration conversion device. [Figure 5] FIG. 5 is a diagram showing an example of illustration pattern data. [Figure 6] FIG. 6 is a diagram illustrating an example of learning input data. [Figure 7] FIG. 7 is a flowchart showing an example of the operation of step S103 in more detail. [Figure 8] FIG. 8 is a diagram illustrating an example of a feature extraction network of VGG19. [Figure 9] FIG. 9 is a diagram showing an example of a neural network used by the habit label identifying unit. [Figure 10] FIG. 10 is a flowchart showing an example of the operation of step S104 in more detail. [Figure 11] FIG. 11 is a flowchart showing an example of the process of step S108 in more detail. [Figure 12] FIG. 12 is a flowchart showing an example of a quirk-free illustration generating operation for generating an illustration in which the quirk has been corrected. [Figure 13] FIG. 13 is a diagram showing an example of a display associated with a habit label. [Figure 14] FIG. 14 is a diagram showing an example of display image data. [Figure 15] FIG. 15 shows an example of a table of input dates and times and habit labels stored in the storage unit. [Figure 16]FIG. 16 is a flowchart showing an example of a learning operation for illustration conversion by the illustration converting device. [Figure 17] FIG. 17 is a diagram showing an example in which learning input data is divided into N blocks. [Figure 18] FIG. 18 is a flowchart showing an example of a quirk-free illustration generating operation for generating an illustration in which the quirk has been corrected. [Figure 19] FIG. 19 is a diagram showing an example of the habit label identification results and generated illustration data displayed on the output device. [Figure 20] FIG. 20 is a block diagram showing the software configuration of the illustration conversion device according to the embodiment in relation to the hardware configuration shown in FIG. [Figure 21] FIG. 21 is a block diagram showing feedback for performing parameter learning of the neural network used by each unit of the illustration conversion device in this embodiment. [Figure 22] FIG. 22 is a flowchart showing an example of a learning operation for illustration conversion by the illustration converting device. [Figure 23] FIG. 23 is a diagram showing an example in which the input image is divided into N blocks by the data acquisition unit. [Figure 24] FIG. 24 is a diagram showing an example of a method for dividing input image data into blocks when the input image data includes eyes. [Figure 25] FIG. 25 is a flowchart showing an example of the operation of step S805 in more detail. [Figure 26] FIG. 26 is a flowchart showing an example of the operation of step S806 in more detail. [Figure 27] FIG. 27 is a flowchart showing an example of the process of step S810 in more detail. [Figure 28] FIG. 28 is a flowchart showing an example of a quirk-free illustration generating operation for generating an illustration in which the quirk has been corrected. [Figure 29] FIG. 29 is a diagram showing an example of the habit label identification results and generated illustration data displayed on the output device. [Figure 30] FIG. 30 is a diagram showing another example of the habit label identification results and generated illustration data displayed on the output device. [Figure 31] FIG. 31 is a diagram showing an example of the habit label identification result displayed on the output device. [Figure 32] FIG. 32 is a diagram showing an example of ranking data of habit labels displayed on the output device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that, hereinafter, elements that are the same as or similar to elements that have already been described will be designated by the same or similar reference numerals, and duplicated descriptions will basically be omitted.

[0013] [First embodiment] (composition) FIG. 1 is a block diagram showing an example of the hardware configuration of an illustration conversion apparatus 100 according to an embodiment. The illustration conversion device 100 may be, for example, a learning model generation device. The illustration conversion device 100 may be, for example, a user terminal used by a user who creates illustrations. Here, the user terminal may be any computer that is generally usable by a user, such as a PC (Personal Computer), a smartphone, a tablet terminal, or a wearable terminal. The illustration conversion device 100 may also be a server to which the user terminal is connected via a network. The server may also be any computer that can be used as a server.

[0014] The illustration conversion device 100 includes a control unit 1, a storage unit 2, and an input / output interface 3. The control unit 1, the storage unit 2, and the input / output interface 3 are communicably connected to one another via a bus. The input / output interface 3 is also communicably connected to an input device 4 and an output device 5.

[0015] The control unit 1 controls the illustration conversion device 100. The control unit 1 includes a hardware processor such as a central processing unit (CPU).

[0016] The storage unit 2 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written to and read from at any time with a nonvolatile memory such as read-only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs necessary for executing various control processes according to one embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The storage unit 2 may further include a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from at any time with a volatile memory such as random access memory (RAM).

[0017] The input / output interface 3 is connected to the input device 4, the output device 5, etc. The input / output interface 3 is an interface that enables transmission and reception of information between the input device 4 and the output device 5. The input / output interface 3 may also include a communication interface. For example, the illustration conversion device 100 and at least one of the input device 4 or the output device 5 may be wirelessly connected using short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology. The communication interface may also include, for example, a communication module that provides a wired or wireless connection to a device or server used by another user via a network.

[0018] The input device 4 includes, for example, a keyboard, a pointing device, etc., which are used by the owner (e.g., a user) of the illustration conversion device 100 to input instructions to the illustration conversion device 100. The input device 4 may also include a reader for reading data to be stored in the storage unit 2 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0019] The output device 5 includes a display that displays output data to be presented to the user from the illustration conversion device 100, a printer that prints the output data, and the like.

[0020] FIG. 2 is a block diagram showing the software configuration of the illustration conversion apparatus 100 according to the embodiment in relation to the hardware configuration shown in FIG. FIG. 3 is a block diagram showing feedback for performing parameter learning of the neural network used by each unit of the illustration conversion apparatus 100 according to the embodiment.

[0021] The storage unit 2 includes a feature extraction parameter storage unit 21 and an illustration refinement parameter storage unit 22 .

[0022] The feature extraction parameter storage unit 21 is used to store the learning results of the feature extraction unit 12, the quirk label identification unit 13, and the quirk removal illustration generation unit 15 of the control unit 1, which will be described later.

[0023] The illustration refinement parameter storage unit 22 is used to store the learning results of the generated illustration refinement unit 16 of the control unit 1, which will be described later.

[0024] The control unit 1 includes a data acquisition unit 11 , a feature extraction unit 12 , a habit label identification unit 13 , an identification result processing unit 14 , a habit removal illustration generation unit 15 , a generated illustration refinement unit 16 , and an output control unit 17 .

[0025] The data acquisition unit 11 acquires various data from the input device 4 via the input / output interface 3. For example, when the illustration conversion device 100 learns parameters to be used in a neural network, the acquired various data may be, for example, image data created by the user (habit illustration data), correct image data, and learning input data including habit labels that explain the habit of the image. Furthermore, when the illustration conversion device 100 corrects an illustration, the acquired data may include illustration data drawn by the user (habit illustration data). The data acquisition unit 11 outputs at least a part of the acquired data to the feature extraction unit 12, the classification result processing unit 14, and the generated illustration refinement unit 16.

[0026] The feature extraction unit 12 includes a habit feature extraction unit 121 , a style feature extraction unit 122 , and a content feature extraction unit 123 .

[0027] The habit feature extraction unit 121 extracts feature data from the illustration data received from the data acquisition unit 11. The habit feature extraction unit 121 may extract the habit feature data from the illustration data using, for example, a convolutional neural network (CNN). Here, the habit feature data may be data that characterizes the drawing habits of the user. A specific method for extracting the habit feature data will be described later. Furthermore, the habit feature extraction unit 121 outputs the extracted habit feature data to the habit label identification unit 13.

[0028] The style feature extraction unit 122 extracts style feature data from the illustration data received from the data acquisition unit 11. For example, the style feature extraction unit 122 extracts style feature data from the illustration data using CNN. Here, the style feature data may be data that characterizes the individuality of the user, such as their drawing style. The style feature data may also include quirk feature data. A specific method for extracting style feature data will be described later. The style feature extraction unit 122 outputs the extracted style feature data to the quirk label identification unit 13 and the quirk-removed illustration generation unit 15.

[0029] The content feature extraction unit 123 extracts content feature data from the illustration data received from the data acquisition unit 11. For example, the content feature extraction unit 123 extracts content feature data from the illustration data using CNN. Here, the content feature data may be data that characterizes the illustration data. For example, if the illustration is a face image or the like, the content feature data may be the position of the five senses or the like. Note that a specific method for extracting content feature data will be described later. Furthermore, the style feature extraction unit 122 outputs the extracted style feature data to the quirk label identification unit 13 and the quirk-removed illustration generation unit 15.

[0030] The habit label identifying unit 13 identifies a habit label from the habit feature data using a neural network. The habit label identifying unit 13 outputs the identified habit label to the identification result processing unit 14 or the output control unit 17.

[0031] The identification result processing unit 14 calculates a loss function based on the habit label (first habit label) acquired from the data acquisition unit 11 and the habit label (second habit label) identified by the habit label identification unit 13. Here, the loss function may be calculated using, for example, cross entropy. Furthermore, the identification result processing unit 14 trains parameters used in the neural network used in the feature extraction unit 12 and the neural network used in the habit label identification unit 13 by back propagation based on the calculated function.

[0032] The quirk removal illustration generation unit 15 generates the quirk removal illustration based on the style feature information extracted by the style feature extraction unit 122 and the content extraction information extracted by the content feature extraction unit 123. For example, the quirk removal illustration generation unit 15 may create the quirk removal illustration data using Adaptive Instance Normalization (Adain). The quirk removal illustration generation unit 15 outputs the quirk removal illustration to the generated illustration refinement unit 16 or the output control unit 17.

[0033] The generated illustration refinement unit 16 functions as a discriminator for a generative adversarial network (GAN) with the quirk feature extraction unit 121, style feature extraction unit 122, content feature extraction unit 123, and quirk removal illustration generation unit 15 as generators. For example, the generated illustration refinement unit 16 compares a fake image generated by the generator with the correct illustration data received from the data acquisition unit 11 as a real image, and uses backpropagation to learn parameters used by the quirk feature extraction unit 121, style feature extraction unit 122, content feature extraction unit 123, and quirk removal illustration generation unit 15 that constitute the generator. Furthermore, the generated illustration refinement unit 16 compares the fake image with a real image and uses backpropagation to learn parameters used by the generated illustration refinement unit 16 itself. The generated illustration refinement unit 16 then stores the learned results in the illustration refinement parameter storage unit 22.

[0034] The output control unit 17 outputs a display message based on the refined illustration data and the identification label to the output device 5 via the input / output interface 3.

[0035] (Learning behavior) FIG. 4 is a flowchart showing an example of a learning operation for illustration conversion by the illustration converting device 100. The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0036] First, in this embodiment, it is assumed that an illustration has a three-layer structure (first layer: content, second layer: style, and third layer: quirks). In this embodiment, illustration conversion device 100 extracts information from the first and second layers and trains parameters used by each unit of illustration conversion device 100 using a neural network to enable the generation of an illustration with clean lines.

[0037] The operation is executed to convert an illustration, i.e., to learn to output an illustration with clean lines from an illustration with a peculiar feature. The operation starts when the input device 4 receives learning input data including illustration pattern data including correct answer illustration data, a plurality of peculiar illustration data, and peculiar labels corresponding to the correct answer illustration data and the plurality of peculiar illustration data, and learning illustration data drawn by a plurality of artists (including the user) based on the correct answer illustration.

[0038] The data acquisition unit 11 acquires input data from the input device 4 through the input / output interface 3 (step S101). The data acquisition unit 11 outputs the received learning input data to the feature extraction unit 12. The data acquisition unit 11 also outputs a habit label included in the illustration pattern data of the input data to the identification result processing unit 14. Furthermore, the data acquisition unit 11 outputs correct illustration data included in the illustration pattern data of the input data to the generated illustration refinement unit 16.

[0039] FIG. 5 is a diagram showing an example of illustration pattern data. 5, the illustration pattern data includes, for example, correct illustration data data0 and a quirk label (pat0. beautiful) associated with the correct illustration data data0, and multiple quirk illustration data data1 and quirk labels (pat1. rough, pat2. casual, pat3. multiple lines, pat4. neutral, pat5. wrinkled) associated with each of the multiple quirk illustration data data1. Here, in this embodiment, an example is shown in which an illustration is classified into five quirks, but the number of quirks is not limited to five and may, of course, be more or less than five.

[0040] The learning input data may include illustration data drawn by multiple authors with reference to the correct illustration data. Furthermore, the illustration pattern data may include illustration pattern data of a human face different from the human face shown in Fig. 5 and learning illustration data drawn by multiple authors based on the correct illustration data included in the illustration pattern data. In other words, the input data may include one or more sets, each set consisting of illustration pattern data and learning illustration data based on the illustration pattern data.

[0041] FIG. 6 is a diagram illustrating an example of learning input data. As shown in FIG. 6(a), for example, the learning illustration data indicates that the size is 1024*1024, the resolution is 600 dpi, the mode is grayscale, and the save format is png.

[0042] As shown in Figure 6(b), the genre of the learning input data is assumed to be illustrations of human faces from the front up to the neck, with no hidden faces, no blackouts, and no collars. In other words, the illustrations shown in Figure 5 fall into the same genre. Furthermore, there are ten artists, each with their own unique style. The illustrations are assumed to include ten characters (referred to as "ten characters" in Figure 6(b)), including five males and five females, and the ten artists draw the ten characters. These illustrations become the learning illustration data. The correct illustration data is one piece, and the peculiar illustration data is five pieces. For example, in the example shown in Figure 6(b), the learning illustration data includes illustration data of ten characters drawn by ten artists, i.e., 100 pieces of illustration data.

[0043] 6 is merely an example, and it goes without saying that any number of genres, artists, illustration types, etc. may be used. Also, while an example is described in which there are five pieces of peculiar illustration data, it goes without saying that there may be more or fewer pieces.

[0044] The feature extraction unit 12 acquires the learning results from the feature extraction parameter storage unit 21 (step S102). The habit feature extraction unit 121, style feature extraction unit 122, and content feature extraction unit 123 of the feature extraction unit 12 each acquire the learning results from the feature extraction parameter storage unit 21. The learning results may be those stored in the feature extraction parameter storage unit 21 in step S105, which will be described later, in a learning process that was performed in the past. Note that if there are no learning results, the process may skip step S102. The habit feature extraction unit 121 learns the habit feature network (step S103).

[0045] FIG. 7 is a flowchart showing an example of the operation of step S103 in more detail. First, the quirk feature extraction unit 121 extracts quirk feature data from the training illustration data (step S201). The quirk feature extraction unit 121 extracts quirk feature data from the training illustration data of the input image using a convolutional neural network. For example, the quirk feature extraction unit 121 extracts quirk feature data from the training illustration data using a VGG19 feature extraction network. As described above, the quirk feature data may be included in the style feature data. Therefore, the style feature extraction unit 122 extracts the style feature data from the training illustration data using the VGG19 feature extraction network. Then, the quirk feature extraction unit 121 may extract quirk feature data from the extracted data using the VGG19 feature extraction network. The quirk feature extraction unit 121 outputs the extracted quirk feature data to the quirk label identification unit 13. Here, the VGG19 feature extraction network has, for example, 16 convolutional layers and 5 pooling layers.

[0046] FIG. 8 is a diagram illustrating an example of a feature extraction network of VGG19. 8, Cn is an n×n convolution layer, R is a ReLU which is a normalized linear function, P is a MaxPooling layer, and RF is a ReflectionPad2d layer. For example, the habit feature extraction unit 121 extracts habit data of the image illustration data in the image input layer using a feature space provided from the convolution layers up to the ninth layer and the three pooling layers counting from the image input layer.

[0047] In addition, in this embodiment, an example is described in which the habit feature extraction unit 121 extracts habit data using VGG19, but this is not limited to this, and it goes without saying that habit feature data may be extracted using other models, such as GoogLenet, ResNet, etc.

[0048] The habit label identifying unit 13 identifies the habit label from the habit feature data (step S202).

[0049] FIG. 9 is a diagram showing an example of a neural network used by the habit label identifying unit 13. As shown in FIG. As shown in FIG. 9, the neural network used by the habit label identification unit 13 is composed of two convolutional layers and three fully connected layers. Here, Cn in FIG. 9 is an n×n convolutional layer, R is a ReLU layer, P is a MaxPooling layer, and FC is a Linear fully connected layer. The habit label identification unit 13 then inputs habit feature data to this neural network and identifies habit labels. For example, when habit feature data is input to the neural network, an estimated probability of each pattern is output. Then, the habit label identification unit 13 identifies the pattern with the highest estimated probability as the habit label. The habit label identification unit 13 then outputs the identified habit label to the identification result processing unit 14.

[0050] The identification result processing unit 14 compares the identified habit label with the habit label received from the data acquiring unit 11 and calculates a loss function (step S203). For example, the identification result processing unit 14 may calculate the loss function of the identified habit label and the habit label received from the data acquiring unit 11 using cross entropy.

[0051] The identification result processing unit 14 trains network parameters for habit feature extraction (step S204). For example, the identification result processing unit 14 trains the parameters of each layer of the habit feature extraction unit 121 and the habit label identification unit 13 by back propagation so as to minimize the calculated loss function. By training the parameters used by the habit feature extraction unit 121 and the habit label identification unit 13 so as to minimize the loss function in this way, the habit label identification unit 13 can use the trained parameters to identify the correct habit label from the habit feature data.

[0052] Here, if there is no need to output the habit label, step S103, that is, steps S201 to S204, may be omitted.

[0053] Referring to FIG. 4, the style feature extraction unit 122, the content feature extraction unit 123, the habit label identification unit 13, and the identification result processing unit 14 of the control unit 1 learn a network of style features and content features (step S104).

[0054] FIG. 10 is a flowchart showing an example of the operation of step S104 in more detail.

[0055] The style feature extraction unit 122 and the content feature extraction unit 123 respectively extract style feature data and content feature data from the training illustration data (step S301). The style feature extraction unit 122 and the content feature extraction unit 123 may respectively extract style feature data and content feature data using a VGG19 feature extraction network, as in step S201 described with reference to FIG. 7 . Of course, as in step S201, the style feature extraction unit 122 and the content feature extraction unit 123 may respectively extract style feature data and content feature data using a convolutional neural network model other than VGG19. The style feature extraction unit 122 then outputs the extracted style feature data to the quirk label identification unit 13. Furthermore, the content feature extraction unit 123 further outputs the extracted style feature data and content feature data to the quirk-removed illustration generation unit 15.

[0056] The habit label identification unit 13 identifies habit labels from the style feature data (step S302). As in step S202 described with reference to FIG. 7, the habit label identification unit 13 inputs the style feature data into the neural network shown in FIG. 9 to identify habit labels. For example, when the style feature data is input into the neural network, an estimated probability of each pattern is output. Then, the habit label identification unit 13 identifies the pattern with the highest estimated probability as the habit label. Then, the habit label identification unit 13 outputs the identified habit label to the identification result processing unit 14.

[0057] The identification result processing unit 14 compares the identified habit label with the received habit label and calculates a loss function (step S303). The identification result processing unit 14 may calculate the loss function using cross entropy using the identified habit label and the received habit label, similar to step S203 described with reference to FIG. 7.

[0058] The identification result processing unit 14 trains network parameters for style feature extraction (step S304). For example, the identification result processing unit 14 trains the parameters of each layer of the style feature extraction unit 122 by back propagation so as to maximize the calculated loss function. For example, the identification result processing unit 14 may train the parameters of each layer of the style feature extraction unit 122 by back propagation so as to maximize the calculated loss function. Furthermore, the identification result processing unit 14 trains the parameters of each layer of the habit label identification unit 13 by back propagation so as to minimize the calculated loss function. In this way, the parameters used by the style feature extraction unit 122 are trained so as to maximize the loss function, and the parameters used by the habit label identification unit 13 are trained so as to minimize the loss function. By training the parameters in this way, the habit label identification unit 13 will no longer be able to identify the correct habit label from the style feature data using the trained parameters.

[0059] Here, if only the habit label is to be output, steps S303 and S304 may be omitted.

[0060] The quirk removal illustration generation unit 15 generates quirk removal illustration data from which the quirk features have been removed (step S305). For example, the quirk removal illustration generation unit 15 generates the quirk removal illustration data using an algorithm that inputs the style feature data received from the style feature extraction unit 122 and the content feature data received from the content feature extraction unit 123. The quirk removal illustration generation unit 15 also outputs the quirk removal illustration data to the generated illustration refinement unit 16.

[0061] The habit-removed illustration generation unit 15 compares the habit-removed illustration data with the correct illustration data received from the data acquisition unit 11 and calculates a loss function (step S306). The habit-removed illustration generation unit 15 may calculate the loss function using, for example, mean squared error (MSE).

[0062] The peculiar feature removal illustration generation unit 15 trains the network parameters for style / content feature extraction (step S307). The peculiar feature removal illustration generation unit 15 trains the parameters of each layer of the style feature extraction unit 122 and the content feature extraction unit 123 by back propagation so as to minimize the loss function.

[0063] The identification result processing unit 14 and the quirk removal illustration generation unit 15 store the learning results of each network (step S105). The identification result processing unit 14 stores the learning results, for example, parameters, of each layer of the quirk feature extraction unit 121, the style feature extraction unit 122, and the quirk label identification unit 13 in the feature extraction parameter storage unit 21. Furthermore, the quirk removal illustration generation unit 15 stores the learning results of each layer of the style feature extraction unit 122 and the content feature extraction unit 123 in the feature extraction parameter storage unit 21.

[0064] The generated illustration refinement unit 16 acquires the learning result from the feature extraction parameter storage unit 21 (step S106). The generated illustration refinement unit 16 acquires each parameter stored in the feature extraction parameter storage unit 21 in step S105.

[0065] The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22 (step S107). The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22. The learning results may be those stored in the illustration refinement parameter storage unit 22 in step S110 (described later) during a previous learning process. Note that if there are no learning results, the process may skip step S107.

[0066] The generated illustration refinement unit 16 trains the network for generating illustrations (step S108).

[0067] FIG. 11 is a flowchart showing an example of the process of step S108 in more detail.

[0068] The generated illustration refinement unit 16 sets the quirk-removed illustration data as a fake image and the correct illustration data as a real image (step S401). The generated illustration refinement unit 16 sets the quirk-removed illustration data generated by the generator as a fake image, and the correct illustration data received from the data acquisition unit 11 as a real image. The generated illustration refinement unit 16 uses the quirk-removed illustration data generated by the generator as a fake image, and the correct illustration data received from the data acquisition unit 11 as a real image.

[0069] The generated illustration refinement unit 16 trains the parameters of each layer of each part constituting the generator by backpropagation (step S402). First, the generated illustration refinement unit 16 trains the parameters of each layer of each part constituting the generator by evaluating using mean squared error (MSE) in a loss function that inputs fake images generated by the generator and real images. Next, the generated illustration refinement unit 16 evaluates using cross entropy error (BCE) whether the judgment result by the discriminator of the fake image data generated by the above MSE training matches the judgment result by the discriminator of the real image data, and trains the parameters of each layer of each part constituting the generator by backpropagation so as to minimize MSE + BCE.

[0070] The generated illustration refinement unit 16 learns the parameters of each layer of the generated illustration refinement unit 16, which is a discriminator (step S403). Next, the generated illustration refinement unit 16 uses the judgment result of the discriminator of the real image data (hereinafter referred to as BCE real ) and the judgment result of the fake image data by the discriminator (hereafter referred to as BCE fake and BCE real +BCE fake The parameters of each layer of the generator / illustrator refiner 16 are trained by back propagation so as to minimize the above.

[0071] 4, the generated illustration refinement unit 16 stores the learning result of the generator network in the feature extraction parameter storage unit 21 (step S109). The generated illustration refinement unit 16 stores the learning result of step S402, which was described with reference to FIG. 11, in the feature extraction parameter storage unit 21.

[0072] The generated illustration refinement unit 16 stores the learning results of the discriminator network in the illustration refinement parameter storage unit 22 (step S110). The generated illustration refinement unit 16 stores the learning results of step S403, which was described with reference to FIG. 11, in the illustration refinement parameter storage unit 22.

[0073] Through the above process, the parameters of each layer of each unit of the control unit 1 are learned, enabling the illustration conversion device 100 to output an illustration without quirks instead of an illustration with quirks. Learning step 1 and learning step 2 shown in FIG. 4 do not need to be performed simultaneously; it is of course possible to perform at least one of them. When learning step 2 is performed, the quirk-free illustration generation unit 15 may of course generate quirk-free illustration data in the same way as described for learning step 1.

[0074] (illustration generation without quirks) FIG. 12 is a flowchart showing an example of a quirk-free illustration generating operation for generating an illustration in which the quirk has been corrected.

[0075] The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0076] The operation starts when the input device 4 receives data of a user-drawn habitual illustration (habitual illustration data). The habitual illustration data may be any data learned through a learning operation. For example, it may be an illustration of a person's face from the front up to the neck.

[0077] The data acquisition unit 11 acquires peculiar illustration data from the input device 4 via the input / output interface 3 (step S501). The data acquisition unit 11 outputs the acquired peculiar illustration data to the feature extraction unit 12 and the peculiar illustration generation unit 15.

[0078] The quirk label identification unit 13 identifies the quirk label (step S502). The quirk feature extraction unit 121, the style feature extraction unit 122, and the content feature extraction unit 123 of the feature extraction unit 12 respectively extract quirk feature data, style feature data, and content feature data from the quirk illustration data in the same manner as the operations described with reference to FIG. 4 and the like. Here, the style feature extraction unit 122 and the content feature extraction unit 123 output the extracted style feature data and content feature data to the quirk removal illustration generation unit 15. Then, at least one of the quirk feature extraction unit 121 or the style feature extraction unit 122 outputs at least one of the extracted quirk feature data or style feature data to the quirk label identification unit 13. Then, the quirk label identification unit 13 identifies the quirk label in the same manner as the operations described with reference to FIG. 4 and the like. The quirk label identified by the quirk label identification unit 13 is output to the output control unit 17.

[0079] The quirk-removed illustration generation unit 15 generates refined illustration data by removing the quirks from the quirk illustration data (step S503). The quirk-removed illustration generation unit 15 creates refined illustration data by removing the quirks from the quirk illustration data based on the style feature data and the content feature data, similar to the operation described with reference to Figure 4 etc. The quirk-removed illustration generation unit 15 outputs the refined illustration data to the output control unit 17.

[0080] The output control unit 17 outputs the display image data including the refined illustration data and the display message based on the habit label to the output device 5 via the input / output interface 3 (step S504).

[0081] FIG. 13 is a diagram showing an example of a display associated with a habit label. For example, if the habit label identification unit 13 identifies that an illustration drawn by a user corresponds to habit label 1, the output control unit 17 outputs to the output device 5 a display message saying, "There are parts where the lines are rough."

[0082] FIG. 14 is a diagram showing an example of display image data. For example, in the example of FIG. 14, the output control unit 17 outputs, as display image data, to the output device 5, image data obtained by superimposing the quirky pre-correction image, i.e., image data included in the quirk illustration data, and the corrected image, i.e., the refined illustration data generated by the quirk removal illustration generation unit 15. The output control unit 17 may also change the refined illustration data so that the corrected image is displayed in a color different from that of the quirk illustration data. The example of FIG. 16 shows an example in which the refined illustration data includes only the changed portions of the quirk illustration data, i.e., only a portion of the data, in the display data. However, the display data may of course include the entire refined illustration data, i.e., the entire facial image.

[0083] The control unit 1 may also assign identification information to each user, and store the date and time when the habit illustration data was received and the habit label in association with the identification information in the storage unit 2. Then, when habit illustration data is received again from a user, the output control unit 17 may output the corrected habit and the uncorrected habit to the output device 5, i.e., display them to the user.

[0084] FIG. 15 shows an example of a table of input dates and times and habit labels stored in the storage unit 2. As shown in Figure 15, the habit illustration data received from the user was initially determined to have habit label 1 (December 1, 2021), but was eventually changed (December 9, 2021) to illustration data without a habit. Furthermore, this table may be obtained by the control unit 1 from the memory unit 2 in response to a user request, output to the output device 5 using the output control unit 17, and presented to the user.

[0085] (Operation and effect of the first embodiment) According to the first embodiment, the illustration conversion device 100 can identify and analyze multiple patterns from an illustration using multiple structures (e.g., patterns, style, and content).The illustration conversion device 100 stores the analysis results and can identify the patterns of an illustration created by a user and automatically display an illustration with beautiful lines that corresponds to the patterns.

[0086] [Second embodiment] (composition) The configuration of the illustration conversion apparatus 100 in the second embodiment is the same as the hardware configuration and software configuration described with reference to FIGS. 1 to 3, and therefore a description thereof will be omitted here.

[0087] (Learning behavior) In the second embodiment, a method will be described in which training illustration data, which is input data, is divided into a plurality of blocks, a quirk label is identified for each block, and refined illustration data is generated for each block.

[0088] FIG. 16 is a flowchart showing an example of a learning operation for illustration conversion by the illustration converting device 100.

[0089] The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0090] The operation is executed to convert an illustration, i.e., to learn to output an illustration with clean lines from an illustration with a peculiar feature. The operation starts when the input device 4 receives learning input data including illustration pattern data including correct answer illustration data, a plurality of peculiar illustration data, and peculiar labels corresponding to the correct answer illustration data and the plurality of peculiar illustration data, and learning illustration data drawn by a plurality of artists (including the user) based on the correct answer illustration.

[0091] The data acquisition unit 11 acquires learning input data from the input device 4 via the input / output interface 3 (step S601). Here, the learning input data, habit labels, and correct illustration data may be the same data and labels as those in the first embodiment.

[0092] The data acquiring unit 11 divides the training input data into N blocks (step S602). Here, N may be a positive integer. For example, the data acquiring unit 11 divides each of the correct illustration data, the plurality of quirk illustration data, and the training illustration data included in the training input data into N blocks. The data acquiring unit 11 may output information about the divided blocks of the training input data and quirk labels to the generated illustration data.

[0093] FIG. 17 is a diagram showing an example in which learning input data is divided into N blocks. When N=16, the data acquiring unit 11 divides the learning input data into blocks I1 to I16. When N=32, the data acquiring unit 11 divides the learning input data into blocks I1 to I32. FIG. 17 shows an example in which the learning illustration data in the learning input data is divided into N blocks. The data acquiring unit 11 may also use a common method to identify blocks that have no lines, i.e., no content, from the divided blocks.

[0094] The data acquisition unit 11 selects an unselected block (step S603). The selection method may be to select in order starting from I1, or randomly. The data acquisition unit 11 outputs the selected blocks of learning input data (i.e., blocks of habit illustration data and correct illustration data) and habit labels to the feature extraction unit 12. The data acquisition unit 11 also outputs the habit labels included in the illustration pattern data of the input data to the classification result processing unit 14.

[0095] Steps S604 to S612 after the block is selected are the same as steps S102 to S110 described with reference to FIG. 4 and other figures in the first embodiment, and therefore will not be described here.

[0096] The feature extraction unit 12 determines whether all blocks have been selected (step S613). After completing network learning for one block through steps S604 to S612, the feature extraction unit 12 determines whether all blocks have been selected. If there are unselected blocks, the process returns to step S603. That is, the data acquisition unit 11 selects a block from the unselected blocks. On the other hand, if it is determined that all blocks have been selected, that is, that network learning for all blocks has been completed, the process ends. Note that blocks with no content do not need to be learned and may therefore be excluded.

[0097] (illustration generation without quirks) FIG. 18 is a flowchart showing an example of a quirk-free illustration generating operation for generating an illustration in which the quirk has been corrected. The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0098] The operation starts when the input device 4 receives data of a user-drawn habitual illustration (habitual illustration data). The habitual illustration data may be any data learned through a learning operation. For example, it may be an illustration of a person's face from the front up to the neck.

[0099] The data acquisition unit 11 acquires peculiar illustration data from the input device 4 via the input / output interface 3 (step S701). The data acquisition unit 11 outputs the acquired peculiar illustration data to the feature extraction unit 12 and the peculiar illustration generation unit 15.

[0100] The data acquiring unit 11 divides the habit illustration data into N blocks (step S702). The data acquiring unit 11 divides the acquired habit illustration data into N blocks, similar to the learning operation described above.

[0101] The data acquisition unit 11 selects an unselected block (step S703). The data acquisition unit selects a block of quirk illustration data that has not yet been selected, and outputs the selected block to the feature extraction unit 12 and the quirk-removed illustration generation unit 15.

[0102] Steps S704 and S705 after the block is selected are the same as steps S502 and S503 described with reference to FIG. 12 in the first embodiment, and therefore will not be described here.

[0103] The data acquisition unit 11 determines whether there are any unselected blocks (step S706). After generating the habit label and generated illustration data for one block through steps S704 to S705, the data acquisition unit 11 determines whether there are any unselected blocks. If there are any unselected blocks, the process returns to step S703. If all blocks have been selected, that is, if the habit label and generated illustration data for all blocks have been generated, the process proceeds to step S707.

[0104] The output control unit 17 outputs the generated illustration data and the habit label to the output device 5 via the input / output interface 3 (step S707).

[0105] FIG. 19 is a diagram showing an example of the habit label identification results and generated illustration data displayed on the output device 5.

[0106] In steps S701 to S706, the output control unit 17 receives the habit label and generated illustration data for each block. These may then be combined to form the original image, which may then be output to the output device 5. In this case, the habit label and generated illustration data may be displayed separately, as indicated by the arrows in FIG. 19.

[0107] (Effects of the second embodiment) According to the second embodiment, the illustration conversion device 100 can identify and analyze multiple quirks from an illustration based on multiple structures (e.g., quirks, style, and content). It is also possible to divide an illustration into multiple blocks and analyze the quirks of each block. The illustration conversion device 100 then stores the analysis results and can identify the quirks of an illustration created by a user, automatically displaying an illustration with clean lines that corresponds to the quirks.

[0108] [Third embodiment] (composition) The hardware configuration of the illustration conversion apparatus 100 in the third embodiment may be the same as the configuration shown in FIG. 1 in the first embodiment, so a duplicated description will be omitted.

[0109] FIG. 20 is a block diagram showing the software configuration of the illustration conversion apparatus 100 according to the embodiment in relation to the hardware configuration shown in FIG. FIG. 21 is a block diagram showing feedback for performing parameter learning of the neural network used by each unit of the illustration conversion apparatus 100 according to the embodiment.

[0110] The third embodiment differs from the first embodiment in that the habit feature extraction unit 121 is not provided.

[0111] The data acquisition unit 11 acquires various data from the input device 4 via the input / output interface 3. For example, when the illustration conversion device 100 learns parameters to be used in a neural network, the acquired various data may be, for example, image data created by the user (habit illustration data), correct image data, and learning input data including habit labels that explain the habit of the image. Furthermore, when the illustration conversion device 100 corrects an illustration, the acquired data may include illustration data drawn by the user (habit illustration data).

[0112] The data acquisition unit 11 divides each of the acquired peculiar illustration data and correct answer image data into N blocks. Then, the data acquisition unit 11 selects one block and outputs it to the feature extraction unit 12.

[0113] The style feature extraction unit 122 of the feature extraction unit 12 extracts an idiosyncratic style from the illustration data received from the data acquisition unit 11. For example, the style feature extraction unit 122 extracts idiosyncratic style feature data from the block illustration data using CNN. Here, the idiosyncratic style feature data is data that characterizes the individuality of the user's drawing habit. For example, the idiosyncratic style feature data may be data that combines the idiosyncratic feature data and style feature data of the first embodiment. The style feature extraction unit 122 outputs the extracted idiosyncratic style feature data to the idiosyncratic label identification unit 13 and the idiosyncratic illustration generation unit 15.

[0114] The content feature extraction unit 123 may have the same function as in the first embodiment, and therefore a duplicated description will be omitted here.

[0115] The habit label identifying unit 13 identifies habit labels from the habit style data using a neural network, and outputs the identified habit labels to the identification result processing unit 14 or the output control unit 17.

[0116] The identification result processing unit 14 may have the same function as in the first embodiment, and therefore a duplicated description will be omitted here.

[0117] The quirk removal illustration generation unit 15 generates the quirk removal illustration based on the quirk style feature information extracted by the style feature extraction unit 122 and the content extraction information extracted by the content feature extraction unit 123. For example, the quirk removal illustration generation unit 15 may create the quirk removal illustration data using Adaptive Instance Normalization (Adain). The quirk removal illustration generation unit 15 outputs the quirk removal illustration to the generated illustration refinement unit 16 or the output control unit 17.

[0118] The generated illustration refinement unit 16 functions as a discriminator for a generative adversarial network (GAN) with the style feature extraction unit 122, content feature extraction unit 123, quirk label identification unit 13, and quirk removal illustration generation unit 15 as generators. For example, the generated illustration refinement unit 16 compares a fake image generated by the generator with the correct illustration data received from the data acquisition unit 11 as a real image, and uses backpropagation to learn parameters used by the style feature extraction unit 122, content feature extraction unit 123, quirk label identification unit 13, and quirk removal illustration generation unit 15 that constitute the generator. Furthermore, the generated illustration refinement unit 16 compares the fake image with a real image and uses backpropagation to learn parameters used by the generated illustration refinement unit 16 itself. The generated illustration refinement unit 16 then stores the learned results in the illustration refinement parameter storage unit 22.

[0119] The output control unit 17 outputs a display message based on the refined illustration data and the identification label to the output device 5 via the input / output interface 3.

[0120] (Learning behavior) FIG. 22 is a flowchart showing an example of a learning operation for illustration conversion by the illustration converting device 100. The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0121] First, in this embodiment, it is assumed that an illustration has a two-layer structure (first layer: content, second layer: distinctive style). In this embodiment, illustration conversion device 100 extracts information from the second layer and trains parameters used by each unit of illustration conversion device 100 using a neural network to enable the generation of an illustration with clean lines.

[0122] The operation is executed to convert an illustration, i.e., to learn to output an illustration with clean lines from an illustration with a peculiar feature. The operation starts when the input device 4 receives learning input data including illustration pattern data including correct answer illustration data, a plurality of peculiar illustration data, and peculiar labels corresponding to the correct answer illustration data and the plurality of peculiar illustration data, and learning illustration data drawn by a plurality of artists (including the user) based on the correct answer illustration.

[0123] The data acquisition unit 11 acquires learning input data from the input device 4 through the input / output interface 3 (step S801). Here, the learning input data, habit label, and correct illustration data may be the same data and labels as those in the first embodiment. Furthermore, the habit illustration data may be data to which a habit label 1 is assigned.

[0124] The data acquiring unit 11 divides the training input data into N blocks (step S802). Here, N may be a positive integer. For example, the data acquiring unit 11 divides each of the correct illustration data, the plurality of peculiar illustration data, and the training illustration data included in the training input data into N blocks. The data acquiring unit 11 may output information about the divided blocks of the training input data and peculiar labels to the generated illustration data.

[0125] FIG. 23 is a diagram showing an example of the learning input data divided into N blocks by the data acquiring unit 11. When N=16, the data acquiring unit 11 divides the learning input data into blocks I1 to I16. When N=32, the data acquiring unit 11 divides the learning input data into blocks I1 to I32. Fig. 23 shows an example in which the learning illustration data is divided into N blocks.

[0126] The label block shows an example in which the habit label is 1. After dividing the training illustration data into, for example, N=16, i.e., 16 blocks, the data acquisition unit 11 may assign the habit label 1 to each of the divided training input data. Alternatively, for example, if the input image data has been assigned the habit label 1, the data acquisition unit 11 may assign the habit label 1 to each block. In this case, the data acquisition unit 11 may determine whether each block contains data, i.e., a line, using a general method. Then, the data acquisition unit 11 may not assign a habit label to a block without a line; that is, as shown in FIG. 23, blocks without content may be represented by blank spaces.

[0127] FIG. 24 is a diagram showing an example of a method for dividing input image data into blocks when the input image data includes eyes. As shown in Fig. 24, when the input image data includes eyes, blocks may be allocated so that each eye fits into one block. The lines used to draw eyes are thicker and more complex than the lines used to draw other parts. Therefore, when the input image data includes eyes, the data acquisition unit 11 may detect the eyes using a normal method and divide the blocks so that each eye fits into one block.

[0128] The data acquisition unit 11 selects a block (step S803). The data acquisition unit 11 randomly selects one block from among blocks with content. The data acquisition unit 11 outputs the selected block of learning input data and its habit label to the feature extraction unit 12. The data acquisition unit 11 also outputs the habit label included in the illustration pattern data of the input data to the identification result processing unit 14.

[0129] The feature extraction unit 12 acquires the learning results from the feature extraction parameter storage unit 21 (step S804). The style feature extraction unit 122 and the content feature extraction unit 123 of the feature extraction unit 12 each acquire the learning results from the feature extraction parameter storage unit 21. The learning results may be those stored in the feature extraction parameter storage unit 21 in step S807, which will be described later, in a previous learning process. Note that if there are no learning results, the process may skip step S804. The style feature extraction unit 122 trains the habit feature network (step S805).

[0130] FIG. 25 is a flowchart showing an example of the operation of step S805 in more detail. First, the style feature extraction unit 122 extracts quirk feature data from the training illustration data (step S901). The style feature extraction unit 122 uses a convolutional neural network to extract quirk feature data of a block selected by the data acquisition unit 11 of the training illustration data. For example, the style feature extraction unit 122 extracts quirk feature data of the training illustration data using a VGG19 feature extraction network. The style feature extraction unit 122 outputs the extracted quirk feature data to the quirk label identification unit 13. Here, the VGG19 feature extraction network may be the same as the feature extraction network described with reference to FIG. 8.

[0131] The habit label identification unit 13 identifies habit labels from the habit style feature data (step S902). The neural network used by the habit label identification unit 13 may be the same as the neural network described with reference to FIG. 9, so a duplicated description will be omitted here. The habit label identification unit 13 inputs the habit style feature data into this neural network and identifies habit labels. For example, when the habit style feature data is input into the neural network, an estimated probability of each pattern is output. Then, the habit label identification unit 13 identifies the pattern with the highest estimated probability as the habit label. Then, the habit label identification unit 13 outputs the identified habit label to the identification result processing unit 14.

[0132] The identification result processing unit 14 compares the identified habit label with the habit label received from the data acquiring unit 11 and calculates a loss function (step S903). For example, the identification result processing unit 14 may calculate a loss function of the identified habit label and the habit label received from the data acquiring unit 11 using cross entropy.

[0133] The classification result processing unit 14 trains the network parameters of the style features (step S904). For example, the classification result processing unit 14 trains the parameters of each layer of the style feature extraction unit 122 and the habit label identification unit 13 by backpropagation so as to minimize the calculated loss function. By training the parameters used by the style feature extraction unit 122 and the habit label identification unit 13 so as to minimize the loss function in this manner, the habit label identification unit 13 can identify the correct habit label from the habit feature data using the trained parameters. Here, if the illustration data has eyes, the data acquisition unit 11 divides each data item of the training input data so that one eye is contained in one block. Then, by learning the eye portion from the above-described training, the control unit 1 can generate a dedicated network for identifying eye habit patterns. As a result, the habit label identification unit 13 can identify the correct habit label for the eye habit using the network parameters trained by the classification result processing unit 14.

[0134] Here, if there is no need to output the habit label, step S805, that is, steps S901 to S904, may be omitted.

[0135] Referring to FIG. 22, the content feature extraction unit 123, the habit label identification unit 13, and the identification result processing unit 14 of the control unit 1 learn a network of content features (step S806).

[0136] FIG. 26 is a flowchart showing an example of the operation of step S806 in more detail.

[0137] The style feature extraction unit 122 and the content feature extraction unit 123 respectively extract style feature data and content feature data from the selected block of the learning illustration data (step S1001). Step S1001 may be the same operation as step S301 described with reference to Fig. 10. Therefore, a duplicated description will be omitted here.

[0138] The habit label identification unit 13 identifies habit labels from the content feature data (step S1002). As in step S202 described with reference to FIG. 7, the habit label identification unit 13 inputs the content feature data into the neural network shown in FIG. 9 to identify habit labels. For example, when the content feature data is input into the neural network, an estimated probability of each pattern is output. Then, the habit label identification unit 13 identifies the pattern with the highest estimated probability as the habit label. Then, the habit label identification unit 13 outputs the identified habit label to the identification result processing unit 14.

[0139] The identification result processing unit 14 compares the identified habit label with the received habit label and calculates a loss function (step S1003). The identification result processing unit 14 may calculate the loss function using cross entropy using the identified habit label and the received habit label, similar to step S203 described with reference to FIG. 7.

[0140] The identification result processing unit 14 trains the network parameters of the content feature extraction unit 123 (step S1004). For example, the identification result processing unit 14 trains the parameters of each layer of the content feature extraction unit 123 by back propagation so as to maximize the calculated loss function. For example, the identification result processing unit 14 may train the parameters of each layer of the content feature extraction unit 123 by back propagation so as to maximize the calculated loss function. Furthermore, the identification result processing unit 14 trains the parameters of each layer of the habit label identification unit 13 by back propagation so as to minimize the calculated loss function. In this way, the parameters used by the content feature extraction unit 123 are trained so as to maximize the loss function, and the parameters used by the habit label identification unit 13 are trained so as to minimize the loss function. By training the parameters in this way, the habit label identification unit 13 will no longer be able to identify the correct habit label from the content feature data using the trained parameters. Here, if only the habit label is to be output, steps S1003 and S1004 may be omitted.

[0141] The quirk removal illustration generation unit 15 generates quirk removal illustration data from which the quirk features have been removed (step S1005). For example, the quirk removal illustration generation unit 15 generates the quirk removal illustration data using AdAin, which inputs the content feature data received from the content feature extraction unit 123. The quirk removal illustration generation unit 15 also outputs the quirk removal illustration data to the generated illustration refinement unit 16.

[0142] The quirk-removed illustration generation unit 15 compares the quirk-removed illustration data with the correct illustration data corresponding to the selected block received from the data acquisition unit 11, and calculates a loss function (step S1006). The quirk-removed illustration generation unit 15 may calculate the loss function using, for example, mean squared error (MSE).

[0143] The quirk-free illustration generation unit 15 trains the network parameters for content feature extraction (step S1007). The quirk-free illustration generation unit 15 trains the parameters of each layer of the content feature extraction unit 123 by back propagation so as to minimize the loss function.

[0144] The classification result processing unit 14 and the quirk removal illustration generation unit 15 store the learning results of each network (step S807). The classification result processing unit 14 stores the learning results, for example, parameters, of each layer of the style feature extraction unit 122 and the quirk label identification unit 13 in the feature extraction parameter storage unit 21. Furthermore, the quirk removal illustration generation unit 15 stores the learning results of each layer of the content feature extraction unit 123 in the feature extraction parameter storage unit 21.

[0145] The generated illustration refinement unit 16 acquires the learning result from the feature extraction parameter storage unit 21 (step S808). The generated illustration refinement unit 16 acquires each parameter stored in the feature extraction parameter storage unit 21 in step S807.

[0146] The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22 (step S809). The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22. The learning results may be those stored in the illustration refinement parameter storage unit 22 in step S812 (described later) during a previous learning process. Note that if there are no learning results, the process may skip step S809.

[0147] The generated illustration refinement unit 16 trains the network for generating illustrations (step S810).

[0148] FIG. 27 is a flowchart showing an example of the process of step S810 in more detail.

[0149] The generated illustration refinement unit 16 sets the quirk-removed illustration data as a fake image and the correct illustration data as a real image (step S1101). The generated illustration refinement unit 16 sets the style feature extraction unit 122, content feature extraction unit 123, quirk label identification unit 13, and quirk-removed illustration generation unit 15 as generators of the adversarial network, and sets the generated illustration refinement unit 16 itself as a discriminator of the adversarial network. Then, the generated illustration refinement unit 16 sets the quirk-removed illustration data generated by the generator as a fake image, and sets the correct illustration data received from the data acquisition unit 11 as a real image.

[0150] Steps S1102 and S1103 may be the same as steps S402 and S403 described with reference to FIG. 11, and therefore a redundant description will be omitted here. Here, if the illustration data includes eyes, the data acquisition unit 11 divides each data item in the training input data so that one eye is contained in one block. Then, by learning the eye portion from the above-described training, the control unit 1 can generate a dedicated network for removing eye quirks. As a result, the quirk-removed illustration generation unit 15 can generate an illustration in which the quirk has been removed from the quirk-free drawing of the eyes, using the network parameters learned by the generated illustration refinement unit 16.

[0151] 22, the generated illustration refinement unit 16 stores the learning result of the generator network in the feature extraction parameter storage unit 21 (step S811). The generated illustration refinement unit 16 stores the learning result of step S1102 described with reference to FIG. 27 in the feature extraction parameter storage unit 21.

[0152] The generated illustration refinement unit 16 stores the learning results of the discriminator network in the illustration refinement parameter storage unit 22 (step S812). The generated illustration refinement unit 16 stores the learning results of step S1103 described with reference to FIG. 27 in the illustration refinement parameter storage unit 22.

[0153] The data acquiring unit 11 determines whether there are any unselected blocks (step S813). After network learning for one block is completed through steps S804 to S812, the data acquiring unit 11 determines whether there are any unselected blocks. If there are any unselected blocks, the process returns to step S803. If all blocks have been selected, that is, if network learning for all blocks has been completed, the process ends. Note that blocks with no content do not need to be learned and may therefore be excluded.

[0154] Through the above process, the parameters of each layer of each unit of the control unit 1 are learned, enabling the illustration conversion device 100 to output an illustration without quirks instead of an illustration with quirks. Learning step 1 and learning step 2 shown in FIG. 22 do not need to be performed simultaneously; it is of course possible to perform at least one of them. When learning step 2 is performed, the quirk-free illustration generation unit 15 may of course generate quirk-free illustration data in the same way as described for learning step 1.

[0155] (illustration creation without quirks) FIG. 28 is a flowchart showing an example of a quirk-free illustration creation operation for generating an illustration in which the quirk has been corrected.

[0156] The control unit 1 of the illustration converting device 100 reads and executes the program stored in the storage unit 2, thereby realizing the operation of this flowchart.

[0157] The operation starts when the input device 4 receives data of a user-drawn habitual illustration (habitual illustration data). The habitual illustration data may be any data learned through a learning operation. For example, it may be an illustration of a person's face from the front up to the neck.

[0158] The data acquisition unit 11 acquires quirk illustration data from the input device 4 through the input / output interface 3 (step S1201).

[0159] The data acquiring unit 11 divides the habit illustration data into N blocks (step S1202). As in the learning operation described above, the data acquiring unit 11 divides the acquired habit illustration data into N blocks.

[0160] The data acquisition unit 11 selects an unselected block (step S1203). The data acquisition unit selects a block of quirk illustration data that has not yet been selected, and outputs the selected block to the feature extraction unit 12 and the quirk-removed illustration generation unit 15.

[0161] The quirk label identification unit 13 identifies the quirk label (step S1204). The style feature extraction unit 122 and content feature extraction unit 123 of the feature extraction unit 12 respectively extract quirk style feature data and content feature data from the quirk illustration data in the same manner as the operations described using FIG. 22 etc. The style feature extraction unit 122 and content feature extraction unit 123 output the extracted quirk style feature data and content feature data to the quirk removal illustration generation unit 15. Then, the style feature extraction unit 122 outputs the extracted quirk style feature data to the quirk label identification unit 13. Then, the quirk label identification unit 13 identifies the quirk label in the same manner as the operations described using FIG. 4 etc. The quirk label identified by the quirk label identification unit 13 is output to the output control unit 17.

[0162] The quirk-removed illustration generation unit 15 generates refined illustration data by removing the quirks from the quirk illustration data (step S1205). The quirk-removed illustration generation unit 15 creates refined illustration data by removing the quirks from the quirk illustration data based on the quirky style feature data and the content feature data, similar to the operation described with reference to Figure 4 etc. The quirk-removed illustration generation unit 15 outputs the refined illustration data to the output control unit 17.

[0163] The acquired data 11 determines whether there are any unselected blocks (step S1206). After generating the habit label and generated illustration data for one block through steps S1204 to S1205, the acquired data 11 determines whether there are any unselected blocks. If there are any unselected blocks, the process returns to step S1203. If all blocks have been selected, that is, if the habit label and generated illustration data for all blocks have been generated, the process proceeds to step S1207.

[0164] The output control unit 17 outputs the refined illustration data and the quirk label to the output device 5 via the input / output interface 3 (step S1207).

[0165] FIG. 29 is a diagram showing an example of the habit label identification result and generated illustration data displayed on the output device 5.

[0166] In steps S1201 to S1206, the output control unit 17 receives the habit label and generated illustration data for each block. These may then be combined to form the original image, which may then be output to the output device 5. In this case, the habit label and generated illustration data may be displayed separately, as indicated by the arrows in FIG. 29.

[0167] FIG. 30 is a diagram showing another example of the habit label identification result and generated illustration data displayed on the output device 5. As shown in FIG. 30, the output control unit 17 may display the identified habit label and the generated illustration data in an overlapping manner.

[0168] (Operation and effect of the third embodiment) According to the third embodiment, the illustration conversion device 100 divides an illustration into multiple blocks, and identifies and analyzes multiple quirks from the illustration using multiple structures (e.g., quirky styles and content) for one of the blocks. The illustration conversion device 100 stores the analysis results and outputs quirks that are aligned with the positions of the quirks in the illustration created by the user, and can automatically display them as a picture.

[0169] [Modification of the third embodiment] (composition) The configuration of the modified example of the third embodiment is the same as that of the third embodiment, so a duplicated description will be omitted here.

[0170] (Learning behavior and habit-free illustration generation behavior) In the modified example of the third embodiment, these operations may be the same as those in the third embodiment, and therefore a duplicated description will be omitted here.

[0171] In a modification of the third embodiment, when the input device 4 receives peculiar illustration data, the illustration converting device 100 identifies the peculiar label. Then, the output control unit 17 outputs only the peculiar label to the output device.

[0172] FIG. 31 is a diagram showing an example of the habit label identification result displayed on the output device 5. As shown in FIG. 31(a), the results of identifying habit labels may be displayed for each block. Alternatively, as shown in FIG. 31(b), the habit labels, the number of blocks corresponding to the habit labels, and the display messages corresponding to the habit labels may be displayed in a table. When creating the table, the numbers of blocks may be arranged in descending order. By displaying the data in this manner, the user can visually see which habit patterns frequently appear in the illustrations.

[0173] The control unit 1 may also assign identification information to each user, and store the date and time when the habit illustration data was received and the identified habit label in association with the identification information in the storage unit 2. That is, the control unit 1 may store the table (habit ranking data) and date and time shown in FIG. 31 in association with the user's identification information in the storage unit 2.

[0174] FIG. 32 is a diagram showing an example of ranking data of habit labels displayed on the output device 5. As shown in FIG. 32, the output control unit 17 can, in accordance with a user's instruction, cause the output device 5 to display in chronological order the habit ranking data acquired when the habit illustration data is input. In this way, the user's habits are displayed in chronological order, and the user is shown the habits that have been corrected and the habits that have not been corrected.

[0175] In the modified example of the third embodiment, since only the habit labels need to be displayed, it is sufficient to simply execute steps S801 to S807 described with reference to Fig. 22. In other words, learning step 1 is executed. Here, steps S1003 and S1004 described with reference to Fig. 26 may be omitted. Even when operating in this manner, the illustration converting device 100 can identify habit labels when habit illustration data is input and output only the identified habit labels.

[0176] (Operations and Effects of the Modification of the Third Embodiment) In a modification of the third embodiment, the habit ranking is stored in the storage unit 2 together with the date and time when the habit illustration data was input. This makes it possible to visually display corrected and uncorrected habits to the user.

[0177] [Other embodiments] In the third embodiment, an example is shown in which input illustration data is divided into 16 or 32 blocks. Here, N is a positive integer, so the third embodiment naturally also includes the case where N=1, i.e., no division into blocks.

[0178] Furthermore, in the above-described embodiment, the learning operation is performed on all blocks with content among the N blocks divided, but the learning operation may also be performed on any number of blocks with content among the divided blocks.

[0179] In the modified example of the third embodiment, an example has been described in which only the quirk label is output as the display image data, but it is also possible to output only the refined illustration data as the image display data.

[0180] The techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be distributed by transmitting the program via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that implements this device loads the program stored on the storage medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by controlling the operation of the software means. The term "storage medium" as used herein is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0181] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in combination as appropriate as possible, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements. [Explanation of symbols]

[0182] 100...Illustration conversion device 1...Control unit 2...Storage section 21...Feature extraction parameter storage unit 22...Illustration refinement parameter memory section 3... Input / output interface 4...Input device 5...Output device 10...Control unit 11...Data acquisition section 12...Feature extraction unit 121... Habit feature extraction unit 122...Style feature extraction unit 123...Content feature extraction unit 13...Flat label identification section 14...Identification result processing unit 15...Humanity removal illustration generation section 16...Generated Illustration Refinement Department 17...Output control section

Claims

1. A learning model generation method executed by an illustration conversion device, comprising: a data acquisition unit of the illustration conversion device acquiring input data including habit illustration data, a first habit label corresponding to the habit illustration data, training illustration data, and correct illustration data; a feature extraction unit of the illustration conversion device extracting style feature data characterizing the individuality of the user, including their drawing style, and content feature data characterizing the training illustration data using a first neural network; a habit label identifying unit of the illustration converting device identifying a second habit label from the style feature data using a second neural network; calculating a first loss function based on the first habit label and the second habit label in a classification result processing unit of the illustration conversion device; the identification result processing unit causes the first neural network and the second neural network to learn parameters used by the first neural network and the second neural network by back propagation based on the first loss function; generating quirk-removed illustration data from at least the content feature data by a quirk-removed illustration generating unit of the illustration converting device; the quirk-removed illustration generating unit calculates a second loss function based on the quirk-removed illustration data and the correct illustration data; the quirk removal illustration generation unit learns parameters used by the first neural network by back propagation based on the second loss function; storing the parameters used by the first neural network and the second neural network in a storage unit of the illustration conversion device; A learning model generation method comprising:

2. The learning model generation method according to claim 1 , wherein the style feature data includes habit feature data that characterizes the user's drawing habits.

3. the feature extraction unit further comprises extracting, using the first neural network, habit feature data that characterizes a user's drawing habit, which is included in the style feature data; the habit label identifying unit identifies the second habit label from the habit feature data included in the style feature data using the first neural network; the quirk-removed illustration generation unit generates the quirk-removed illustration data from the style feature data and the content feature data. The learning model generation method according to claim 1 .

4. A generated illustration refinement unit of the illustration conversion device, which operates as a discriminator of the adversarial network, compares the quirk-removed illustration data with the correct illustration data, thereby learning parameters used by the feature extraction unit and the quirk-removed illustration generation unit, which constitute a generator of the adversarial network, through back propagation. The generated illustration refinement unit compares the quirk-removed illustration data with the correct illustration data, thereby learning parameters used by the generated illustration refinement unit through back propagation. storing parameters used by the generated illustration refinement unit in the storage unit; The learning model generation method according to claim 2 or 3, further comprising:

5. The acquiring step includes acquiring quirk illustration data; the extracting step includes extracting at least one of quirk feature data or content feature data of the quirk illustration data using the trained first neural network; the identifying step includes identifying a habit label from the habit feature data or the content feature data of the extracted habit illustration data using the trained second neural network; The learning model generating method according to claim 4 , further comprising outputting, by an output control unit of the illustration converting device, display information corresponding to the extracted habit label.

6. The data acquisition unit divides the training illustration data and the correct illustration data into a plurality of blocks, The learning model generation method according to claim 1, wherein, for each block having content among the divided blocks, the extraction, the identification of the second habit label, the calculation of the first loss function, the training of the parameters by backpropagation, the generation of the habit-removed illustration data, the calculation of the second loss function, the training of the parameters by backpropagation, and the storage of the parameters.

7. The extracting step further comprises extracting content feature data of the peculiar illustration data; the generating step further comprises creating refined illustration data based on illustration feature data of the quirky illustration data and the content feature data; The learning model generation method according to claim 5 , wherein the outputting step further comprises outputting the refined illustration data.

8. The learning model generating method according to claim 7 , wherein the outputting step further comprises outputting display image data for displaying the peculiar illustration data and the refined illustration data in an overlapping manner.

9. The storage unit further stores a quirk label corresponding to the quirk illustration data, When the acquiring step acquires second habit illustration data after the input of the habit illustration data, the identifying step further includes identifying a habit label corresponding to the second habit illustration data; The outputting step further includes outputting a habit label corresponding to the habit illustration data and a habit label corresponding to the second habit illustration data in time series. The learning model generation method according to claim 5 .

10. An illustration conversion method executed by an illustration conversion device, acquiring quirk illustration data by a data acquisition unit of the illustration conversion device; a feature extraction unit of the illustration conversion device extracting, using a trained neural network, a habit label corresponding to the habit illustration data, style feature data characterizing the user's individuality, and content feature data characterizing the habit illustration data; generating refined illustration data based on the style feature data and the content feature data in a quirk-removed illustration generation unit of the illustration conversion device; an output control unit of the illustration conversion device outputting the refined illustration data and a display message based on the habit label; An illustration conversion method comprising:

11. a data acquisition unit that acquires input data including habit illustration data, a first habit label corresponding to the habit illustration data, training illustration data, and correct illustration data; a feature extraction unit that uses a first neural network to extract style feature data that characterizes the individuality and drawing habits of a user of the training illustration data, and content feature data that characterizes the training illustration data; a habit label identifying unit that identifies a second habit label from the style feature data using a second neural network; a classification result processing unit that calculates a first loss function based on the first habit label and the second habit label, and that learns parameters used by the first neural network and the second neural network by back propagation based on the first loss function; a quirk-removed illustration generation unit that generates quirk-removed illustration data from the style feature data and the content feature data, calculates a second loss function based on the quirk-removed illustration data and the correct illustration data, and learns parameters used by the first neural network by back propagation based on the second loss function; a storage unit that stores the parameters used by the first neural network and the parameters used by the second neural network; A learning model generation device comprising:

12. a data acquisition unit that acquires input data including habit illustration data, a first habit label corresponding to the habit illustration data, training illustration data, and correct illustration data; a feature extraction unit that uses a first neural network to extract habit feature data that characterizes the drawing habits of a user of the training illustration data, style feature data that characterizes the individuality of the user, and content feature data that characterizes the training illustration data; a habit label identifying unit that identifies a second habit label from at least one of the habit feature data and the style feature data using a second neural network; a classification result processing unit that calculates a first loss function based on the first habit label and the second habit label, and that learns parameters used by the first neural network and the second neural network by back propagation based on the first loss function; a quirk-removed illustration generation unit that generates quirk-removed illustration data from the style feature data and the content feature data, calculates a second loss function based on the quirk-removed illustration data and the correct illustration data, and learns parameters used by the first neural network by back propagation based on the second loss function; a storage unit that stores the parameters used by the first neural network and the parameters used by the second neural network; A learning model generation device comprising:

13. a data acquisition unit for acquiring quirk illustration data; a feature extraction unit that uses the trained neural network to extract a quirk label corresponding to the quirk illustration data, style feature data that characterizes the user's individuality, and content feature data that characterizes the quirk illustration data; a quirk-removed illustration generation unit that generates refined illustration data based on the style feature data and the content feature data; an output control unit that outputs a display message based on the refined illustration data and the habit label; An illustration conversion device comprising:

14. A learning model generation program that causes a processor of the illustration conversion device to execute the learning model generation method according to claim 1.

15. An illustration conversion program that causes a processor of the illustration conversion device to execute the illustration conversion method according to claim 10.

Images