Illustration conversion device, illustration conversion method, and illustration conversion program
The illustration conversion device addresses the challenge of correcting quirks in line drawings by using a neural network to identify and refine user-created illustrations, ensuring the preservation of line strength and weakness, thus aiding beginner artists in improving their skills without expert evaluation.
Patent Information
- Application Number
- PCT/JP2023/020380
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Existing drawing support technologies fail to effectively correct quirks in line drawings created by beginners, often simplifying illustrations and lacking the ability to evaluate subjective qualities like line strength and sharpness, making it difficult for users to advance without expert feedback.
An illustration conversion device utilizing a neural network with a data acquisition unit, feature extraction units, and a quirk removal illustration generation unit to identify and remove quirks from user-created illustrations, employing a noise estimation network for denoising processing to refine illustrations while preserving line strength and weakness.
Enables the correction of multiple quirks in illustrations, identifying their type and location, and generating refined illustrations that maintain the original line drawing's strength and weakness, facilitating user improvement without expert feedback.
Smart Images

Figure JP2023020380_05122024_PF_FP_ABST
Abstract
Description
Illustration conversion device, illustration conversion method, and illustration conversion program
[0001] The present invention relates to an illustration conversion device, an illustration conversion method, and an illustration conversion program.
[0002] Beginner users who have just begun learning how to create illustrations may create line drawings. Such users often have little or no desire to receive feedback (evaluation) from experts on their illustrations. Without expert feedback, the illustrations created by the user often end up being quirky. Even if the user continues to practice with quirky illustrations, it is difficult for the user to reach an advanced level. Therefore, there is a technology that automatically corrects illustrations created by users.
[0003] Existing drawing assistance technologies tend to aim for simplification of line drawings and eliminate details in the line drawings.
[0004] For example, Non-Patent Document 1 proposes a tool called Smart Inker that applies deep learning to enable interactive inking. Here, Smart Inker is a tool that can naturally connect broken lines and efficiently erase unnecessary lines.
[0005] Edgar Simo-Serra et al. “Real-Time Data-Driven Interactive Rough Sketch Inking”, ACM Transactions on Graphics, 2018
[0006] Furthermore, existing drawing assistance technologies have the problem that they cannot display multiple types of habits, and users must select the position of the illustration to be corrected.
[0007] The illustrations output using the technique disclosed in Non-Patent Document 1 have the problem that they are not drawn in detail and the generated lines have no strength or weakness.
[0008] Furthermore, illustrations are often evaluated subjectively based on the shape, breaks, and sharpness of the lines, and no clean line modeling is performed. Therefore, there is a problem that the quirks of illustrations created by users can only be evaluated by experts and cannot be evaluated by applications that edit user illustrations.
[0009] This invention was made in light of the above circumstances, and its purpose is to generate a corrected image without losing the strength and weakness of the line drawing by using a diffusion model in an image generation network.
[0010] Furthermore, an object of the present invention is to deal with a plurality of quirks in illustrations created by a user, and to present the type of quirk and its location.
[0011] In order to solve the above problem, one aspect of the present invention is an illustration conversion device comprising: a data acquisition unit that acquires quirk illustration data; a feature extraction unit that uses a first neural network to extract content feature data that characterizes the quirk illustration data from the quirk illustration data; a quirk-removed illustration generation unit that generates quirk-removed illustration data by removing the quirks from the content feature data; a generated illustration refinement unit that acquires a predetermined noise level and Gaussian noise corresponding to the quirk illustration data, and performs denoising on the quirk-removed illustration data in accordance with a predetermined algorithm using a noise estimation network based on the noise level and Gaussian noise; and an output control unit that outputs the denoised quirk-removed illustration data.
[0012] According to one aspect of the present invention, it is possible to handle a plurality of quirks in an illustration created by a user, and to present the type of quirk and its location.
[0013] FIG. 1 is a block diagram showing an example of the hardware configuration of an illustration conversion apparatus according to an embodiment. FIG. 2 is a block diagram showing the software configuration of the illustration conversion apparatus according to an embodiment in association with the hardware configuration shown in FIG. 1. FIG. 3 is a block diagram showing feedback for parameter learning of a neural network used by each unit of the illustration conversion apparatus according to an embodiment. FIG. 4 is a flowchart showing an example of a learning operation for illustration conversion by the illustration conversion apparatus. FIG. 5 is a diagram showing an example of illustration pattern data. FIG. 6 is a diagram showing an example of learning input data. FIG. 7 is a diagram showing an example of the data acquisition unit dividing the learning input data into N blocks. FIG. 8 is a flowchart showing an example of the operation of step S105 in more detail. FIG. 9 is a flowchart showing an example of the operation of step S106 in more detail. FIG. 10 is a flowchart showing an example of the process of step S110 in more detail. FIG. 11 is a diagram showing an example of the processing operation of the generated illustration refinement unit in the process of step S110. FIG. 12 is a flowchart showing an example of a quirk-free illustration generation operation for generating an illustration with quirks corrected. FIG. 13 is a diagram showing an example of the processing operation of the generated illustration refinement unit in the quirk-free illustration generation process. Fig. 14 is a diagram showing an example of the habit label identification result and generated illustration data displayed on the output device. Fig. 15 is a diagram showing another example of the habit label identification result and generated illustration data displayed on the output device. Fig. 16 is a diagram showing an example of the habit label identification result displayed on the output device. Fig. 17 is a diagram showing an example of habit label ranking data displayed on the output device.
[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that, hereinafter, elements that are the same as or similar to elements that have already been described will be designated by the same or similar reference numerals, and duplicated descriptions will basically be omitted.
[0015] [Embodiment] (Configuration) FIG. 1 is a block diagram showing an example of the hardware configuration of an illustration conversion device 100 according to an embodiment. The illustration conversion device 100 may be, for example, a learning model generation device. The illustration conversion device 100 may be, for example, a user terminal used by a user who creates illustrations. Here, the user terminal may be any computer that is generally usable by a user, such as a personal computer (PC), smartphone, tablet device, or wearable device. The illustration conversion device 100 may also be a server to which the user terminal is connected via a network. The server may also be any computer that can be used as a server.
[0016] The illustration conversion device 100 includes a control unit 1, a storage unit 2, and an input / output interface 3. The control unit 1, the storage unit 2, and the input / output interface 3 are communicatively connected to one another via a bus. The input / output interface 3 is also communicatively connected to an input device 4 and an output device 5.
[0017] The control unit 1 controls the illustration conversion device 100. The control unit 1 includes a hardware processor such as a central processing unit (CPU).
[0018] The storage unit 2 is configured, for example, by combining a nonvolatile memory such as a solid state drive (SSD) that can be written to and read from at any time with a nonvolatile memory such as a read-only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs necessary for executing various control processes according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "programs." The storage unit 2 may further include a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from at any time with a volatile memory such as a random access memory (RAM).
[0019] The input / output interface 3 is connected to the input device 4, the output device 5, etc. The input / output interface 3 is an interface that enables transmission and reception of information between the input device 4 and the output device 5. The input / output interface 3 may also include a communication interface. For example, the illustration conversion device 100 and at least one of the input device 4 or the output device 5 may be wirelessly connected using short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology. The communication interface may also include, for example, a communication module that provides a wired or wireless connection to a device or server used by another user via a network.
[0020] The input device 4 includes, for example, a keyboard, a pointing device, etc., which are used by the owner (e.g., a user) of the illustration conversion device 100 to input instructions to the illustration conversion device 100. The input device 4 may also include a reader for reading data to be stored in the storage unit 2 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.
[0021] The output device 5 includes a display that displays output data to be presented to the user from the illustration conversion device 100, a printer that prints the output data, and the like.
[0022] Fig. 2 is a block diagram showing the software configuration of the illustration conversion apparatus 100 according to the embodiment in relation to the hardware configuration shown in Fig. 1. Fig. 3 is a block diagram showing feedback for performing parameter learning of the neural network used by each unit of the illustration conversion apparatus 100 according to the embodiment. The storage unit 2 includes a feature extraction parameter storage unit 21 and an illustration refinement parameter storage unit 22.
[0023] The feature extraction parameter storage unit 21 is used to store the learning results of the feature extraction unit 12, habit label identification unit 13, and habit removal illustration generation unit 15 of the control unit 1, which will be described later.
[0024] The illustration refinement parameter storage unit 22 is used to store the learning results of the generated illustration refinement unit 16 (described later) of the control unit 1.
[0025] The control unit 1 includes a data acquisition unit 11 , a feature extraction unit 12 , a habit label identification unit 13 , an identification result processing unit 14 , a habit removal illustration generation unit 15 , a generated illustration refinement unit 16 , and an output control unit 17 .
[0026] The data acquisition unit 11 acquires various data from the input device 4 via the input / output interface 3. For example, when the illustration conversion device 100 learns parameters to be used in a neural network, the acquired various data may be, for example, image data created by the user (habit illustration data), correct image data, and training input data including habit labels that explain the habit of the image. Furthermore, when the illustration conversion device 100 corrects an illustration, the acquired data may include illustration data drawn by the user (habit illustration data). The data acquisition unit 11 outputs at least a portion of the acquired data to the feature extraction unit 12, the classification result processing unit 14, and the generated illustration refinement unit 16.
[0027] The feature extraction unit 12 includes a style feature extraction unit 121 and a content feature extraction unit 122 .
[0028] The style feature extraction unit 121 extracts style feature data from the illustration data received from the data acquisition unit 11. For example, the style feature extraction unit 121 extracts style feature data from the illustration data using a CNN. Here, the style feature data may be data that characterizes the individuality of the user, such as their drawing style. The style feature data may also include quirk feature data. A specific method for extracting style feature data will be described later. The style feature extraction unit 121 outputs the extracted style feature data to the quirk label identification unit 13 and the quirk-removed illustration generation unit 15.
[0029] The content feature extraction unit 122 extracts content feature data from the illustration data received from the data acquisition unit 11. For example, the content feature extraction unit 122 extracts content feature data from the illustration data using a CNN. Here, the content feature data may be data that characterizes the illustration data. For example, if the illustration is a facial image, the content feature data may be the position of the five senses. A specific method for extracting content feature data will be described later. Furthermore, the content feature extraction unit 122 outputs the extracted content feature data to the quirk-removed illustration generation unit 15.
[0030] The habit label identifying unit 13 identifies habit labels from the habit feature data using a neural network, and outputs the identified habit labels to the identification result processing unit 14 or the output control unit 17.
[0031] The identification result processing unit 14 calculates a loss function based on the habit labels (first habit labels) acquired from the data acquisition unit 11 and the habit labels (second habit labels) identified by the habit label identification unit 13. Here, the loss function may be calculated using cross entropy, for example. Furthermore, the identification result processing unit 14 trains parameters used in the neural network used in the feature extraction unit 12 and the neural network used in the habit label identification unit 13 by back propagation based on the calculated function.
[0032] The quirk removal illustration generation unit 15 generates the quirk removal illustration based on the style feature information extracted by the style feature extraction unit 121 and the content extraction information extracted by the content feature extraction unit 122. For example, the quirk removal illustration generation unit 15 may generate quirk removal illustration data using AdaiN (Adaptive InSance Normalization). The quirk removal illustration generation unit 15 outputs the quirk removal illustration to the generated illustration refinement unit 16 or the output control unit 17.
[0033] The generated illustration refinement unit 16 includes an input noise calculation unit 161 , a noise estimation unit 162 , and an output image calculation unit 163 .
[0034] The input noise calculation unit 161 calculates the noise level γ, the Gaussian noise ε, the quirk-free illustration data x, and the correct illustration data y 0 Here, the noise level γ is a value indicating the noise level in the learning illustration data, and may be a value obtained by a general method. In other words, the noise level γ may be a value defined by the user. For example, the noise level γ is [1*10 -6 , 0.01]. Gaussian noise ε is data of the same size as the image data, generated using random numbers from a standard normal distribution N(0,I) with a mean of 0 and a variance of 1. Here, I is the unit matrix. The standard normal distribution is expressed as f(x) = 1 / (√(2π)) exp(-x 2 / 2).
[0035] In addition, the input noise calculation unit 161 calculates the correct illustration data y 0 The noise image y is obtained by adding noise level γ and Gaussian noise ε to ~ It should be noted that the noise image y ~ The calculation method will be described later.
[0036] The noise estimation unit 162 estimates the noise image y ~ , quirk-free illustration data x, noise level γ, and Gaussian noise ε. The noise estimation unit 162 estimates the estimated noise ε ~ and the Gaussian noise ε, where the loss function is calculated as ~ The loss function will be described in detail later. Furthermore, the noise estimation unit 162 trains the network of the noise estimation unit 162 so as to minimize the loss function according to a predetermined algorithm. The algorithm will be described in detail later.
[0037] Furthermore, when outputting refined illustration data, the noise estimation unit 162 acquires a noise estimation network, a noise level ε, a count t, input noise (Gaussian noise ε), and quirk-free illustration data x. Then, the noise estimation unit 162 refers to the quirk-free illustration data x and calculates estimated noise ε from the Gaussian noise ε using the noise estimation network. ~ Calculate.
[0038] The output image calculation unit 163 performs denoising processing using the estimated noise from the input noise. Details of the denoising processing will be described later.
[0039] The output control unit 17 outputs to the output device 5 via the input / output interface 3 a display image based on the refined illustration data and the identification label, and a message corresponding to the display image.
[0040] 4 is a flowchart showing an example of a learning operation for illustration conversion by illustration conversion device 100. The operation of this flowchart is realized by control unit 1 of illustration conversion device 100 reading and executing a program stored in storage unit 2.
[0041] First, in this embodiment, it is assumed that an illustration has a two-layer structure (first layer: content, second layer: unusual style). In this embodiment, the illustration conversion device 100 extracts information from the second layer and trains parameters used by each unit of the illustration conversion device 100 using a neural network to generate an illustration with clean lines. Here, the unusual style is defined as a style of drawing lines that should not be drawn.
[0042] The operation is executed to convert an illustration, i.e., to learn to output an illustration with clean lines from an illustration with a peculiar feature, and the operation starts when the input device 4 receives learning input data including illustration pattern data including correct answer illustration data, a plurality of peculiar illustration data, and peculiar labels corresponding to the correct answer illustration data and the plurality of peculiar illustration data, and learning illustration data drawn by a plurality of artists (including the user) based on the correct answer illustration data.
[0043] The data acquisition unit 11 acquires input data from the input device 4 via the input / output interface 3 (step S101). The data acquisition unit 11 outputs the received learning input data to the feature extraction unit 12. The data acquisition unit 11 also outputs a habit label included in the illustration pattern data of the input data to the identification result processing unit 14. Furthermore, the data acquisition unit 11 outputs correct illustration data included in the illustration pattern data of the input data to the generated illustration refinement unit 16.
[0044] 5 is a diagram showing an example of illustration pattern data. As shown in FIG. 5, the illustration pattern data includes, for example, correct illustration data data0 and a quirk label (pat0. Beautiful) associated with the correct illustration data data0, and multiple quirk illustration data data1 and quirk labels (pat1. Rough, pat2. Random, pat3. Multiple lines, pat4. Neutral, pat5. Wrinkled) associated with each of the multiple quirk illustration data data1. Here, in this embodiment, an example is shown in which an illustration is classified into five quirks, but the number of quirks is not limited to five and may be more or less than five.
[0045] The learning input data may include illustration data drawn by multiple authors with reference to the correct illustration data. Furthermore, the illustration pattern data may include illustration pattern data of a human face different from the human face shown in Fig. 5 and learning illustration data drawn by multiple authors based on the correct illustration data included in the illustration pattern data. In other words, the input data may include one or more sets, each set consisting of illustration pattern data and learning illustration data based on the illustration pattern data.
[0046] 6A and 6B are diagrams showing examples of learning input data. As shown in Fig. 6A, for example, the learning illustration data has a size of 1024*1024, a resolution of 600 dpi, a grayscale mode, and a PNG storage format.
[0047] As shown in FIG. 6( b), the genre of the learning input data is assumed to be a human face illustration showing the front of the neck, with no hidden face, no blackout, and no collar. In other words, the illustration shown in FIG. 5 falls into the same genre. Furthermore, it is assumed that there are ten artists, each with their own unique style. The illustrations are assumed to include ten characters (referred to as "ten characters" in FIG. 6( b)), including five males and five females, and the ten artists draw the ten characters. These drawn illustrations become the learning illustration data. The correct illustration data is one piece, and the peculiar illustration data is five pieces. For example, in the example shown in FIG. 6( b), the learning illustration data includes illustration data of ten characters drawn by ten artists, i.e., 100 pieces of illustration data.
[0048] 6 is merely an example, and it goes without saying that any number of genres, artists, illustration types, etc. may be used. Also, although an example is described in which there are five pieces of peculiar illustration data, it goes without saying that there may be more or fewer pieces.
[0049] The data acquiring unit 11 divides the training input data into N blocks (step S102), where N may be a positive integer. For example, the data acquiring unit 11 divides each of the correct answer illustration data, the plurality of peculiar illustration data, and the training illustration data included in the training input data into N blocks. The data acquiring unit 11 may output information about the divided blocks of the training input data and peculiar labels to the generated illustration data.
[0050] 7 is a diagram showing an example in which the data acquisition unit 11 divides the learning input data into N blocks. When N=16, the data acquisition unit 11 divides the learning input data into blocks I1 to I16. When N=32, the data acquisition unit 11 divides the learning input data into blocks I1 to I32. FIG. 7 shows an example in which the learning illustration data is divided into N blocks.
[0051] The label block shows an example in which the habit label is 1. After dividing the training illustration data into, for example, N=16, i.e., 16 blocks, the data acquisition unit 11 may assign habit label 1 to each of the divided training input data. Alternatively, for example, if habit label 1 has been assigned to the input image data, the data acquisition unit 11 may assign habit label 1 to each block. In this case, the data acquisition unit 11 may determine whether each block contains data, i.e., a line, using a general method. Then, the data acquisition unit 11 may not assign habit labels to blocks without lines; that is, as shown in FIG. 7, blocks without content may be represented by blank spaces.
[0052] The data acquisition unit 11 selects an unselected block (step S103). The data acquisition unit 11 randomly selects one block from among the blocks with content. However, the data acquisition unit 11 does not select a block that has already been selected. The data acquisition unit 11 outputs the selected block of learning input data and its habit label to the feature extraction unit 12. The data acquisition unit 11 also outputs the habit label included in the illustration pattern data of the input data to the identification result processing unit 14.
[0053] The feature extraction unit 12 acquires the learning results from the feature extraction parameter storage unit 21 (step S104). The style feature extraction unit 121 and the content feature extraction unit 122 of the feature extraction unit 12 each acquire the learning results from the feature extraction parameter storage unit 21. The learning results may be those stored in the feature extraction parameter storage unit 21 in step S107 (described later) during a previous learning process. Note that if there are no learning results, the process may skip step S104.
[0054] The style feature extraction unit 121 trains a network of style features (step S105).
[0055] 8 is a flowchart illustrating an example of the operation of step S105 in more detail. First, the style feature extraction unit 121 extracts quirky style feature data from the training illustration data (step S201). The style feature extraction unit 121 uses a convolutional neural network to extract quirky feature data of blocks selected by the data acquisition unit 11 of the training illustration data. For example, the style feature extraction unit 121 extracts quirky style feature data from the training illustration data using a VGG19 feature extraction network. The style feature extraction unit 121 outputs the extracted quirky style feature data to the quirk label identification unit 13. Here, the VGG19 feature extraction network may be a general feature extraction network.
[0056] In addition, in this embodiment, the style feature extraction unit 121 has been described as an example of extracting quirky style feature data using VGG19, but this is not limited to this, and it is of course possible to extract quirky style feature data using other models, such as GoogleLeneNet and ResNet.
[0057] The habit label identification unit 13 identifies habit labels from the habit style feature data (step S202). The habit label identification unit 13 inputs the habit style feature data into a neural network and identifies habit labels. Here, the neural network used by the habit label identification unit 13 may be a general neural network consisting of two convolutional layers and three fully connected layers. For example, when the habit style feature data is input into the neural network, an estimated probability of each pattern is output. Then, the habit label identification unit 13 identifies the pattern with the highest estimated probability as the habit label. Then, the habit label identification unit 13 outputs the identified habit label to the identification result processing unit 14.
[0058] The identification result processing unit 14 compares the identified habit label with the habit label received from the data acquisition unit 11 and calculates a loss function (step S203). For example, the identification result processing unit 14 calculates a loss function of the identified habit label and the habit label received from the data acquisition unit 11 using cross entropy.
[0059] The identification result processing unit 14 trains the network parameters of the style features (step S204). For example, the identification result processing unit 14 trains the parameters of each layer of the style feature extraction unit 121 and the habit label identification unit 13 by backpropagation so as to minimize the calculated loss function. By training the parameters used by the style feature extraction unit 121 and the habit label identification unit 13 so as to minimize the loss function in this way, the habit label identification unit 13 can identify the correct habit label from the habit feature data using the trained parameters.
[0060] Here, if there is no need to output habit labels, step S105, that is, steps S201 to S204, may be omitted.
[0061] Referring again to FIG. 4, the style feature extraction unit 121, content feature extraction unit 122, habit label identification unit 13, and identification result processing unit 14 of the control unit 1 learn a network of style features and content features (step S106).
[0062] FIG. 9 is a flowchart showing an example of the operation of step S106 in more detail.
[0063] The style feature extraction unit 121 and the content feature extraction unit 122 respectively extract style feature data and content feature data from selected blocks of the training illustration data (step S301). As in step S201 described with reference to FIG. 8 , the style feature extraction unit 121 and the content feature extraction unit 122 may respectively extract style feature data and content feature data from selected blocks of the training illustration data using a VGG19 feature extraction network. Of course, as in step S201, the style feature extraction unit 121 and the content feature extraction unit 122 may respectively extract style feature data and content feature data using a convolutional neural network of a model other than VGG19. The style feature extraction unit 121 then outputs the extracted style feature data to the quirk label identification unit 13. Furthermore, the content feature extraction unit 122 further outputs the extracted style feature data and content feature data to the quirk-removed illustration generation unit 15.
[0064] The habit label identification unit 13 identifies habit labels from the content feature data (step S302). Similar to step S202 described with reference to FIG. 8, the habit label identification unit 13 inputs the content feature data into a neural network to identify habit labels. For example, when the content feature data is input into the neural network, an estimated probability of each pattern is output. The habit label identification unit 13 then identifies the pattern with the highest estimated probability as the habit label. The habit label identification unit 13 then outputs the identified habit label to the identification result processing unit 14.
[0065] The identification result processing unit 14 compares the identified habit label with the received habit label and calculates a loss function (step S303). Similar to step S203 described with reference to FIG. 8, the identification result processing unit 14 calculates the loss function using the cross entropy of the identified habit label and the received habit label.
[0066] The identification result processing unit 14 trains network parameters for content feature extraction (step S304). For example, the identification result processing unit 14 trains the parameters of each layer of the content feature extraction unit 122 by backpropagation so as to maximize the calculated loss function. For example, the identification result processing unit 14 may train the parameters of each layer of the content feature extraction unit 122 by backpropagation so as to maximize the calculated loss function. Furthermore, the identification result processing unit 14 trains the parameters of each layer of the habit label identification unit 13 by backpropagation so as to minimize the calculated loss function. In this way, the parameters used by the content feature extraction unit 122 are trained so as to maximize the loss function, and the parameters used by the habit label identification unit 13 are trained so as to minimize the loss function. By training the parameters in this way, the habit label identification unit 13 will no longer be able to identify the correct habit label from the content feature data using the trained parameters.
[0067] Here, if only the habit label is output, steps S303 and S304 may be omitted.
[0068] The quirk removal illustration generation unit 15 generates quirk removal illustration data from which the quirk features have been removed (step S305). For example, the quirk removal illustration generation unit 15 generates the quirk removal illustration data using Adain, which inputs the content feature data received from the content feature extraction unit 122. The quirk removal illustration generation unit 15 also outputs the quirk removal illustration data to the generated illustration refinement unit 16.
[0069] The quirk-removed illustration generation unit 15 compares the quirk-removed illustration data with the correct illustration data corresponding to the selected block received from the data acquisition unit 11, and calculates a loss function (step S306). The quirk-removed illustration generation unit 15 may calculate the loss function using, for example, mean squared error (MSE).
[0070] The quirk-free illustration generation unit 15 trains the network parameters for content feature extraction (step S307). The quirk-free illustration generation unit 15 trains the parameters of each layer of the content feature extraction unit 122 by backpropagation so as to minimize the loss function.
[0071] 4 , the identification result processing unit 14 and the quirk-removed illustration generation unit 15 store the learning results of each network (step S107). The identification result processing unit 14 stores the learning results, e.g., parameters, of each layer of the style feature extraction unit 121 and the quirk label identification unit 13 in the feature extraction parameter storage unit 21. Furthermore, the quirk-removed illustration generation unit 15 stores the learning results of each layer of the content feature extraction unit 122 in the feature extraction parameter storage unit 21.
[0072] The generated illustration refinement unit 16 acquires the learning result from the feature extraction parameter storage unit 21 (step S108). The generated illustration refinement unit 16 acquires each parameter stored in the feature extraction parameter storage unit 21 in step S107.
[0073] The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22 (step S109). The generated illustration refinement unit 16 acquires the learning results from the illustration refinement parameter storage unit 22. The learning results may be those stored in the illustration refinement parameter storage unit 22 in step S110 (described later) during a previous learning process. Note that if there are no learning results, the process may skip step S109.
[0074] The generated illustration refinement unit 16 trains the network for generating illustrations (step S110).
[0075] 10 is a flowchart illustrating an example of the process of step S110 in more detail. FIG. 11 is a diagram illustrating an example of the processing operation of the generated illustration refinement unit 16 in the process of step S110. The input noise calculation unit 161 of the generated illustration refinement unit 16, which operates as an acquisition unit, calculates the noise level γ, the Gaussian noise ε, the quirk-free illustration data x, and the correct illustration data y. 0Here, the noise level γ is a value indicating the noise level in the learning illustration data of the selected block, and may be a value obtained by a general method as described above. In this embodiment, -6 ,0.01].
[0076] The input noise calculation unit 161 calculates the correct illustration data y 0 The noise image y is obtained by adding noise level γ and Gaussian noise ε to ~ is calculated (step S402).
[0077]
[0078] Here, N represents a normal distribution and I is a unit matrix. 0 This means that Gaussian noise ε is added γ times to the -6 ,0.01] across all values in the range [1*10 -6 ,0.01] means all values in the range [1*10 -6 , 0.01] into a predetermined number (for example, 1000 or 2000) of equal distances. ~ to the noise estimation unit 162.
[0079] The noise estimation unit 162 estimates the noise image y ~ , quirk-removed illustration data x, noise level γ, and Gaussian noise ε are acquired (step S403). The image diagram of Fig. 11 shows an example in which the noise level γ, Gaussian noise ε, and quirk-removed illustration data x are acquired directly by the noise estimation unit 162. However, for example, the input noise calculation unit 161 may input this information included in the input information to the noise estimation unit 162.
[0080] The noise estimation unit 162 estimates the estimated noise ε ~ A loss function indicating the difference between the Gaussian noise ε and the eigenvalue ε is calculated (step S404). Here, the loss function is expressed by the following formula:
[0081]
[0082] Here, as shown in equation (2), the estimated noise ε ~ is a neural network model f θ (x, y ~ , γ), where p represents the probability distribution.
[0083] The noise estimation unit 162 estimates the estimated noise ε ~ (Step S405). The noise estimation unit 162 outputs the calculated estimated noise ε ~ For example, the noise estimation unit 162 outputs the output estimated noise ε ~ is used according to the following algorithm:
[0084] The noise estimation unit 162 calculates the estimated noise ε ~ The network of the noise estimation unit 162 is trained so as to minimize a loss function indicating the difference between the input noise (Gaussian noise ε) and the input noise (Gaussian noise ε) (step S406).
[0085]
[0086] 4, generated illustration refinement unit 16 stores the learning results of the network (first network) of noise estimation unit 162, which is configured to minimize the loss function, in feature extraction parameter storage unit 21 (step S111). Generated illustration refinement unit 16 stores the learning results of step S406, which was described with reference to FIG. 10, in feature extraction parameter storage unit 21.
[0087] The generated illustration refinement unit 16 stores the learning results of the network (second network) of the noise estimation unit 162 that minimizes the loss function in the illustration refinement parameter storage unit 22 (step S112). The generated illustration refinement unit 16 stores the learning results of step S406 described with reference to FIG. 10 in the illustration refinement parameter storage unit 22.
[0088] The data acquisition unit 11 determines whether there are any unselected blocks (step S113). After completing network learning for one block through steps S104 to S112, the data acquisition unit 11 determines whether there are any unselected blocks. If there are any unselected blocks, the process returns to step S103. If all blocks have been selected, that is, if network learning for all blocks has been completed, the process ends. Note that blocks with no content do not need to be learned and may therefore be excluded.
[0089] Through the above process, the parameters of each layer of each unit of the control unit 1 are learned, enabling the illustration conversion device 100 to output an illustration without quirks instead of an illustration with quirks. Learning Step 1 and Learning Step 2 shown in Figure 4 do not need to be performed simultaneously; it goes without saying that at least one of them may be performed. When learning Step 2 is performed, the quirk-free illustration generation unit 15 may of course generate quirk-free illustration data in the same way as described for learning Step 1.
[0090] 12 is a flowchart showing an example of a quirk-free illustration generation operation for generating an illustration with corrected quirks. The operation of this flowchart is realized by the control unit 1 of the illustration conversion device 100 reading and executing a program stored in the storage unit 2.
[0091] The operation starts when the input device 4 receives data of a user-drawn habitual illustration (habitual illustration data). The habitual illustration data may be any data learned through a learning operation. For example, it may be an illustration of a person's face from the front up to the neck.
[0092] The data acquisition unit 11 acquires quirk illustration data from the input device 4 through the input / output interface 3 (step S501).
[0093] The data acquisition unit 11 divides the peculiar illustration data into N blocks (step S502). As in the learning operation described above, the data acquisition unit 11 divides the acquired peculiar illustration data into N blocks.
[0094] The data acquisition unit 11 selects an unselected block (step S503). The data acquisition unit 11 selects a block of quirk illustration data that has not yet been selected, and outputs the selected block to the feature extraction unit 12 and the quirk-removed illustration generation unit 15.
[0095] The quirk label identification unit 13 identifies quirk labels (step S504). The style feature extraction unit 121 and content feature extraction unit 122 of the feature extraction unit 12 respectively extract quirk style feature data and content feature data from the quirk illustration data, similar to the operations described with reference to FIG. 4 and other figures. That is, the style feature extraction unit 121 extracts quirk feature data characterizing the user's writing quirks, and the content feature extraction unit 122 extracts content feature data characterizing the quirk illustration data. The style feature extraction unit 121 and content feature extraction unit 122 output the extracted quirk style feature data and content feature data to the quirk removal illustration generation unit 15. The style feature extraction unit 121 then outputs the extracted quirk style feature data to the quirk label identification unit 13. The quirk label identification unit 13 then identifies quirk labels, similar to the operations described with reference to FIG. 4 and other figures. The quirk labels identified by the quirk label identification unit 13 are output to the output control unit 17.
[0096] The quirk-removed illustration generation unit 15 generates quirk-removed illustration data by removing the quirks from the quirk illustration data (step S505). The quirk-removed illustration generation unit 15 generates quirk-removed illustration data by removing the quirks from the quirk illustration data based on the quirky style feature data and the content feature data, in the same manner as the operation described with reference to FIG.
[0097] The quirk-removed illustration generating unit 15 outputs the quirk-removed illustration data to the generated illustration refining unit 16 (step S506). The quirk-removed illustration generating unit 15 outputs the generated quirk-removed illustration to the generated illustration refining unit 16.
[0098] 13 is a diagram showing an example of an image of the processing operation of the generated illustration refinement unit 16 in the quirk-free illustration generation process. As shown in FIG. 13 , the noise estimation unit 162 of the generated illustration refinement unit 16 acquires a noise estimation network, a noise level γ, a count t, input noise (Gaussian noise ε), and quirk-removed illustration data x (step S507). The noise estimation unit 162 acquires the noise estimation network stored in the illustration refinement parameter storage unit 22. The noise estimation unit 162 further acquires the noise level γ, the count t, and the Gaussian noise ε stored in the storage unit 2. The noise estimation unit 162 then acquires the quirk-removed illustration output from the quirk-removed illustration generation unit 15.
[0099] The noise estimation unit 162 outputs the estimated noise (step S508). The noise estimation unit 162 references the quirk-free illustration data x and calculates the estimated noise ε from the input noise (Gaussian noise ε) using the noise estimation network. ~ where the estimated noise is f θ (x, y t , γ t ) and y t Let N(0,I) be the estimated noise ε ~ is calculated by calculating the difference from the input noise (Gaussian noise ε) using a loss function (mean square estimation error) using the parameters stored in the illustration refinement parameter 22 (i.e., using a noise estimation network). Then, the noise estimation unit 162 outputs the estimated noise to the output image calculation unit 163.
[0100] The output image calculation unit 163 performs denoising processing using the estimated noise from the input noise (step S509). For example, the output image calculation unit 163 performs denoising processing according to the following algorithm.
[0101]
[0102] In addition, the noise level γ t is expressed by the following formula:
[0103]
[0104] where α t is a hyperparameter of the noise schedule.
[0105] The output control unit 17 outputs the denoised refined illustration data calculated by equation (3) as an output image (step S510). The output image calculation unit 163 outputs the denoised refined illustration data as an output image to the output control unit 17. The output control unit 17 outputs the received refined illustration data to the output device 5, causing it to be displayed on a display or the like.
[0106] The output image calculation unit 163 determines whether noise removal has been performed t times (step S511). The output image calculation unit 163 determines whether denoising has been performed t times, the number of times being received as input. If it is determined that denoising has been performed t times, the process proceeds to step S512. On the other hand, if it is determined that denoising has not been performed t times, the process returns to step S508.
[0107] The above steps ST508 to ST511 can also be expressed as the following algorithm: That is, the generated illustration refinement unit performs denoising on the quirk-free illustration data in accordance with a predetermined algorithm using a noise estimation network based on Gaussian noise of a predetermined noise level.
[0108]
[0109] The output control unit 17 outputs the refined illustration data that has been denoised using equation (3) t times as an output image (step S512). The output image calculation unit 163 outputs the refined illustration data that has been denoised t times as an output image to the output control unit 17. The output control unit 17 outputs the received refined illustration data to the output device 5, causing it to be displayed on a display or the like.
[0110] The data acquisition unit 11 determines whether there are any unselected blocks (step S513). After generating the habit label and refined illustration data for one block through steps S504 to S512, the data acquisition unit 11 determines whether there are any unselected blocks. If there are any unselected blocks, the process returns to step S503. If all blocks have been selected, i.e., if habit labels and refined illustration data have been generated for all blocks, the process proceeds to step S514.
[0111] The output control unit 17 outputs the refined illustration data and the quirk label to the output device 5 via the input / output interface 3 (step S514).
[0112] 14 is a diagram showing an example of the habit label identification results and generated illustration data displayed on the output device 5. In steps S501 to S514, the output control unit 17 receives the habit label and generated illustration data for each block. These may then be combined to form the original image, which may then be output to the output device 5. In this case, the habit label and generated illustration data may be displayed separately, as indicated by the arrows in FIG. 14.
[0113] 15 is a diagram showing another example of the habit label identification result and the generated illustration data displayed on the output device 5. As shown in FIG. 15, the output control unit 17 may display the identified habit label and the generated illustration data in an overlapping manner.
[0114] FIG. 16 is a diagram showing an example of the habit label identification results displayed on the output device 5. As shown in FIG. 16(a), the habit label identification results may be displayed for each block. Alternatively, as shown in FIG. 16(b), the habit label, the number of blocks corresponding to the habit label, and the display message corresponding to the habit label may be displayed in a table. When creating the table, the numbers of blocks may be arranged in descending order. By displaying in this manner, the user can visually see which habit patterns appear most frequently in the illustrations.
[0115] The control unit 1 may also assign identification information to each user, and store the date and time when the habit illustration data was received and the identified habit label in association with the identification information in the storage unit 2. That is, the control unit 1 may store the table shown in Fig. 31 (habit ranking data) and the date and time in association with the user's identification information in the storage unit 2.
[0116] 17 is a diagram showing an example of ranking data of habit labels displayed on the output device 5. As shown in Fig. 17, the output control unit 17 can display, in chronological order, the habit ranking data acquired when habit illustration data is input in accordance with a user's instruction on the output device 5. In this way, the user's habits are shown in chronological order, and the habit that has been corrected and the habit that has not been corrected are displayed for the user.
[0117] In this embodiment, if it is sufficient to only display the habit labels, it is sufficient to simply execute steps S501 to S504 and steps S513 to S514 described with reference to Fig. 12. That is, it is sufficient to execute learning step 1 described with reference to Fig. 4, and then execute steps S501 to S507 and steps S513 to S514. Conversely, if it is sufficient to only display the refined illustration data, it is sufficient to simply execute steps S501 to S502 and steps S505 to S514 described with reference to Fig. 12. That is, it is sufficient to execute learning step 2 described with reference to Fig. 3, and then execute steps S501 to S502 and steps S505 to S514.
[0118] (Effects of the embodiment) In this embodiment, the quirks of an illustration are identified for each block, the image content is separated from the identified quirky style, and the image generation network is trained to generate beautiful illustrations (refined illustration data) without quirks.
[0119] Furthermore, in this embodiment, by using a diffusion model in the image generation network, it is possible to generate a corrected drawing (refined illustration data) without losing the strength and weakness of the line drawing.
[0120] [Other Embodiments] In the above embodiment, an example was shown in which input illustration data was divided into N blocks. Here, N is a positive integer, so the above embodiment can also be implemented with N=1, i.e., without dividing into blocks.
[0121] Furthermore, in the above-described embodiment, the learning operation is performed on all blocks with content among the N blocks divided, but the learning operation may also be performed on any number of blocks with content among the divided blocks.
[0122] Furthermore, the techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device loads the program stored on the storage medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. Note that the term "storage medium" as used herein is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.
[0123] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in combination as appropriate as possible, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements.
[0124] 100...Illustration conversion device 1...Control unit 2...Memory unit 21...Feature extraction parameter memory unit 22...Illustration refinement parameter memory unit 3...Input / output interface 4...Input device 5...Output device 11...Data acquisition unit 12...Feature extraction unit 121...Style feature extraction unit 122...Content feature extraction unit 13...Habit label identification unit 14...Identification result processing unit 15...Habit removal illustration generation unit 16...Generated illustration refinement unit 161...Input noise calculation unit 162...Noise estimation unit 163...Output image calculation unit 17...Output control unit
Claims
1. An illustration conversion device comprising: a data acquisition unit that acquires quirk illustration data; a feature extraction unit that uses a first neural network to extract content feature data that characterizes the quirk illustration data from the quirk illustration data; a quirk-removed illustration generation unit that generates quirk-removed illustration data by removing the quirk from the content feature data; a generated illustration refinement unit that acquires a predetermined noise level and Gaussian noise corresponding to the quirk illustration data, and performs denoising on the quirk-removed illustration data in accordance with a predetermined algorithm using a noise estimation network based on the noise level and Gaussian noise; and an output control unit that outputs the denoised quirk-removed illustration data.
2. The illustration conversion device of claim 1, wherein the generated illustration refinement unit refers to the quirk-free illustration data, uses the noise estimation network to output estimated noise from the Gaussian noise, and performs denoising processing on the quirk-free illustration data based on the estimated noise and Gaussian noise.
3. The illustration conversion device of claim 1, wherein the data acquisition unit divides the quirk illustration data into a plurality of blocks, the quirk-removed illustration generation unit generates the quirk-removed illustration data for each block, the generated illustration refinement unit performs the denoising process on the quirk-removed illustration data for each block, and the output control unit collectively outputs the quirk-removed illustration data for each block that has been denoised.
4. The illustration conversion device of claim 3, wherein the feature extraction unit uses the first neural network to extract style feature data that characterizes the user's individuality including their writing style for each block, the illustration conversion device further includes a quirk label identification unit that identifies a quirk label from the style feature data for each block, and the output control unit outputs the quirk label for each block and quirk-removed illustration data that has been denoised by superimposing them on each other.
5. The illustration conversion device of claim 1, wherein the data acquisition unit further acquires input data including training illustration data and correct illustration data; the feature extraction unit uses the first neural network to extract training content feature data characterizing the training illustration data; the quirk-removed illustration generation unit generates training quirk-removed illustration data by removing quirks from the training illustration data, calculates a first loss function based on the training quirk-removed illustration data and the correct illustration data, and learns parameters to be used by the first neural network by backpropagation based on the first loss function; the generated illustration refinement unit acquires a noise level and Gaussian noise corresponding to the training illustration data, calculates a second loss function based on the training quirk-removed illustration data, the noise level, the Gaussian noise, and the correct illustration data, and trains a network to minimize the second loss function in accordance with a predetermined algorithm; and the illustration conversion device further comprises a memory unit for storing a result of the learning.
6. The illustration conversion device of claim 5, wherein the generated illustration refinement unit further comprises an input noise calculation unit that generates a noise image from the correct illustration data based on the Gaussian noise and the noise level, and a noise estimation unit that trains the network further based on the noise image.
7. An illustration conversion method executed by a processor of an illustration conversion device, comprising: acquiring peculiar illustration data; extracting from the peculiar illustration data using a first neural network content feature data that characterizes the peculiar illustration data; generating peculiar-removed illustration data by removing the peculiar features from the content feature data; acquiring a predetermined noise level and Gaussian noise corresponding to the peculiar illustration data; performing denoising on the peculiar illustration data in accordance with a predetermined algorithm using a noise estimation network based on the noise level and Gaussian noise; and outputting the denoised peculiar illustration data.
8. An illustration conversion program having instructions to be executed by a processor of an illustration conversion device, the instructions comprising: acquiring quirk illustration data; extracting from the quirk illustration data using a first neural network content feature data that characterizes the quirk illustration data; generating quirk-free illustration data by removing the quirk from the content feature data; acquiring a predetermined noise level and Gaussian noise corresponding to the quirk illustration data; performing denoising on the quirk-free illustration data in accordance with a predetermined algorithm using a noise estimation network based on the noise level and Gaussian noise; and outputting the denoised quirk-free illustration data.
Citation Information
Patent Citations
Image conversion device and image conversion program
JP2003203221A
Image generating device and image generating system
JP2004318187A
Adaptive Gaussian Noise Level Estimation Model Using Optimization Technique and Shallow CNN
KR102313870B1
Automatic illustrator guides
US20200320756A1
