Method, program, and device for training generative artificial intelligence model to ensure consistency of output data

Abstract

According to an embodiment of the present invention, a method for training a generative artificial intelligence model is disclosed, the method comprising the steps of obtaining outline data, obtaining coloring data, and generating a drawing style matrix on the basis of the outline data and the coloring data. As a result, the present invention ensures the consistency of data output by the generative artificial intelligence model and enables obtaining output that matches the intention of a user and, by using different input data depending on the purpose of training the generative artificial intelligence model, significantly reduces the time and cost required for prompting the generative artificial intelligence model and enhances the quality of output data.

Description

Method, program, and device for training a generative artificial intelligence model to ensure consistency of output data

[0001] The present invention relates to a method, a program, and a device for training a generative artificial intelligence model, and more particularly, to a method, a program, and a device for training a generative artificial intelligence model to ensure consistency of data output by the generative artificial intelligence model and obtain output that conforms to the user's intention.

[0002] Recently, many people have been generating images using diffusion models, a generative AI model, without actually drawing them. These diffusion models possess a vast amount of training data, and open-source policies allow for additional training data from a wide range of individuals. Consequently, the quality of images generated through diffusion models is steadily improving.

[0003] However, these generative AI models only randomly generate a variety of images, failing to consistently produce images as intended by the user. Consequently, users must make numerous attempts to obtain the desired image.

[0004] Meanwhile, companies are increasingly seeking to replace some tasks or personnel with generative AI models, particularly in automation of tasks like translation and repetitive daily routines. This is because, even if generative AI models produce random results, human corrections are relatively easy for these tasks. However, there are limitations to using generative AI models in tasks that require consistent output data or where the resulting output is highly dependent on a single individual. These tasks include the creative work of artists such as composers, lyricists, painters, novelists, and cartoonists.

[0005] In particular, the comics market is expanding due to the growing demand for webtoons, which are digital comics serialized online as well as offline comics. Typically, webtoon artists create webtoons on a regular basis (e.g., once a week, twice a week, etc.) and upload them online. As the webtoon market has grown, so have the general public's expectations for the quality of webtoons. Consequently, it is difficult to produce webtoons that meet these expectations on a regular basis. Efforts are being made to create comics using generative AI models. However, these models only output randomly generated images and cannot produce consistent images. This makes it difficult to maintain the essential requirement of comics, which is to produce multiple consistent images that match the artist's style.

[0006] Accordingly, when generating images using a generative artificial intelligence model, the need for a method of training a generative artificial intelligence model to generate consistent images that reflect the user's intention is emerging.

[0007] The present invention has been proposed to solve the above problems, and its purpose is to provide a method, program, and device for training a generative artificial intelligence model to ensure consistency of data output by the generative artificial intelligence model and obtain output that matches the user's intention.

[0008] The technical problem to be solved by the present invention is not limited to the technical problem mentioned above, and may include various technical problems within a scope obvious to those skilled in the art from the contents described below.

[0009] According to one embodiment of the present invention, a method for training a generative artificial intelligence model by a processor of a device includes the steps of: obtaining outline data; obtaining coloring data; and generating a drawing style matrix based on the line drawing data and the coloring data.

[0010] In one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include the steps of: acquiring initial data; and extracting the line drawing data and the coloring data from the initial data.

[0011] The step of extracting the line drawing data and the coloring data from the initial data may include the step of classifying the initial data into a plurality of preset categories; and the step of extracting the line drawing data and the coloring data based on the categories.

[0012] The step of extracting the line drawing data and the coloring data from the initial data may further include a step of removing error data from the initial data.

[0013] In addition, the step of extracting the line drawing data and the coloring data from the initial data may further include a step of additionally generating the line drawing data or the coloring data based on the initial data when the number of the line drawing data or the coloring data is less than a preset number.

[0014] The step of generating a painting style matrix based on the line drawing data and the coloring data may include: obtaining a first artificial neural network model including a weight matrix; training the first artificial neural network model with the line drawing data and the coloring data; and generating the painting style matrix based on a change in the weight matrix after training.

[0015] In one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include the steps of: obtaining character data; and generating a character matrix based on the character data.

[0016] The step of generating a character matrix based on the character data may include: obtaining a second artificial neural network model including a weight matrix; training the character data to the second artificial neural network model; and generating the character matrix based on changes in the weight matrix after training.

[0017] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include a step of generating a merge matrix based on the painting style matrix and the character matrix.

[0018] The step of generating a merge matrix based on the above-mentioned style matrix and the above-mentioned character matrix may include a step of obtaining a merge ratio; and a step of merging the style matrix and the character matrix based on the merge ratio.

[0019] The above merging ratio may be a ratio of the style matrix 0.8: the character matrix 0.2, the style matrix 0.5: the character matrix 0.5, or a ratio in which the ratio of the character matrix does not exceed the style matrix.

[0020] In one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include a step of merging the painting matrix or the merging matrix into a previously stored generative artificial intelligence model.

[0021] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include: obtaining output data from the generative artificial intelligence model; modifying the merging ratio; regenerating a merging matrix based on the modified merging ratio; and merging the regenerated merging matrix with a previously stored generative artificial intelligence model.

[0022] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include: obtaining output data from the generative artificial intelligence model; obtaining additional data; regenerating a painting style matrix or a merge matrix based on the additional data; and merging the regenerated painting style matrix or merge matrix with a previously stored generative artificial intelligence model. Alternatively, the method may further include: visually displaying the output data.

[0023] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include the steps of: obtaining background data; and generating a background matrix based on the background data.

[0024] The step of generating a background matrix based on the above background data may include: obtaining a third artificial neural network model including a weight matrix; training the third artificial neural network model with the background data; and generating the background matrix based on a change in the weight matrix included in the third artificial neural network model after training.

[0025] The step of generating a merge matrix based on the above-mentioned style matrix and the above-mentioned character matrix can be performed by merging the above-mentioned style matrix, the above-mentioned character matrix, and the above-mentioned background matrix.

[0026] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include a step of acquiring initial data; and a step of extracting the background data from the initial data.

[0027] The step of extracting the background data from the initial data may include the step of classifying the initial data into a plurality of preset categories; and the step of extracting the background data based on the categories.

[0028] The step of extracting the background data from the initial data may further include a step of removing error data from the initial data.

[0029] In addition, the step of extracting the background data from the initial data may further include a step of additionally generating the line drawing data or the coloring data based on the initial data when the number of the line drawing data or the coloring data is less than a preset number.

[0030] In addition, in one embodiment, a method for training a generative artificial intelligence model by a processor of a device may further include a step of acquiring initial data; and a step of extracting the character data from the initial data.

[0031] The step of extracting the character data from the initial data may include a step of classifying the initial data into a plurality of preset categories; and a step of extracting the character data based on the categories.

[0032] The step of extracting the character data from the initial data may further include a step of removing error data from the initial data.

[0033] In addition, the step of extracting the character data from the initial data may further include a step of additionally generating the character data based on the initial data when the character data is less than a preset number.

[0034] It is preferable that the above preset number be 100 or 90 to 110.

[0035] The above weight matrix may include a first weight matrix and a second weight matrix, and the step of training the artificial neural network model with the painting style data, the character data, or the background data may be to change the second weight matrix without changing the first weight matrix.

[0036] The above-mentioned painting matrix, the above-mentioned character matrix, or the above-mentioned background matrix may be generated based on changes in the above-mentioned second weight matrix.

[0037] According to one embodiment of the present invention, input data can be varied depending on the purpose of training a generative artificial intelligence model to obtain output that better matches the user's intent. For example, when obtaining graphic data such as webtoons through a generative artificial intelligence model that has learned a large amount of data indiscriminately, all of the character's appearance, drawing style, background, situation, etc. must be input. However, if a generative artificial intelligence model learned through a method according to one embodiment of the present invention is used, output that basically reflects the drawing style, character, and background information can be generated, so that output results that match the user's intent can be obtained simply by entering the character name and situation description. Therefore, not only is the time and cost required for prompting a generative artificial intelligence model drastically reduced, but the quality of the output data can also be improved.

[0038] FIG. 1 is a diagram for explaining the structure of an artificial intelligence model according to one embodiment of the present invention.

[0039] FIG. 2 is a diagram showing a network environment of a generative artificial intelligence model learning system related to one embodiment of the present invention.

[0040] FIG. 3 is a block diagram illustrating the configuration of a device used to train a generative artificial intelligence model according to one embodiment of the present invention.

[0041] FIG. 4 is a flowchart illustrating a method for training a generative artificial intelligence model and obtaining output values ​​according to one embodiment of the present invention.

[0042] FIG. 5 is a flowchart illustrating a process of obtaining character data, line drawing data, color data, or background data from initial data according to one embodiment of the present invention.

[0043] FIG. 6 is a flowchart illustrating a process for generating output data according to one embodiment of the present invention.

[0044] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0045] In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, a detailed description thereof will be omitted.

[0046] Additionally, the following examples may be modified in various other forms, and the scope of the technical concepts of the present invention is not limited to the following examples. Rather, these examples are provided to further faithfully and completely convey the technical concepts of the present invention to those skilled in the art.

[0047] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0048] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0049] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0050] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0051] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0052] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between the component and the other component.

[0053] Singular expressions include plural expressions unless the context clearly indicates otherwise. That is, unless otherwise specified or the context clearly indicates a singular form, the singular should generally be construed as meaning "one or more" in this disclosure and claims.

[0054] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0055] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0056] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0057] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0058] The terms "information" and "data" used herein may be used interchangeably. Similarly, the terms "picture" and "image" used herein may be used interchangeably. "Content" used herein refers to various types of information or content provided via the Internet or computer communications, and may include pictures, images, videos, text, and the like.

[0059] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0060] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0061] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary skill in the art to which the present disclosure pertains can easily implement the present disclosure.

[0062] FIG. 1 is a diagram for explaining the structure of an artificial intelligence model according to one embodiment of the present invention.

[0063] Referring to Figure 1, an artificial intelligence model or a generative artificial intelligence model may include an input layer, an output layer (or decision layer), and at least one hidden layer.

[0064] Each layer has multiple weight values, and performs layer operations through the operation results of the previous layer and the operation of multiple weights. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and deep Q-networks. The neural networks in the present disclosure are not limited to the examples described above, unless otherwise specified.

[0065] A model is a set that includes an algorithm that defines the process from input to output when performing a calculation based on input data and outputting at least one result value, and rules for modifying or executing the algorithm. An artificial neural network (ANN) model or artificial intelligence model refers to a mathematical model in the form of a graph designed to imitate the operating principles of biological neurons.

[0066] A neuron in a hidden layer or hidden layer receives at least one input from other neurons in the model and generates at least one output value. The function that generates an output value from the input value within the neuron is called an activation function. Activation functions may include, but are not limited to, the sigmoid function, the rectified linear unit function (ReLU), and the softmax function, and the activation functions used may differ between neurons that make up a single model or layer.

[0067] Learning is performed to improve the performance of a model. In a narrow sense, it means determining the value of a weight that is multiplied by each input value input to the activation function of a neuron in the model and a bias that is set for each neuron and determines the degree of activation of the neuron. In a broader sense, it means all actions that can affect the model, such as creating or collecting a dataset that will be the input value of the model, preprocessing the created or collected data, selecting or designing a model to be used, evaluating, improving, and optimizing the model based on the determined weights and biases, in addition to determining the values ​​of the weights and biases. Some of the learning process may be performed repeatedly to improve the performance of the model. Examples of algorithms by which an artificial intelligence model performs learning include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithms in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0068] The model can store weights in a form that can be converted into a matrix, such as a list of matrices or vectors, and can store the weight matrix before learning and the weight change matrix after learning together. The matrices representing the weights or the weight change can be stored as two or more matrices. For example, the change in the matrix W representing the weights shown in Figure 1 can be the inner product of lower-dimensional matrices A and B. In this case, when performing learning, the model can change only A and B while fixing W, where the weights of the existing model are stored.

[0069] When two or more matrices are used to represent a single matrix, the learning process has the advantages of faster learning speed, lower hardware requirements such as memory, and high scalability as the low-rank matrix can be applied to other models or tasks. In the present disclosure, the style matrix, the character matrix, and the background matrix may be at least one of two or more matrices representing the weight matrix of an artificial neural network model. All matrices include vectors.

[0070] FIG. 2 is a diagram showing a network environment of a generative artificial intelligence model learning system related to one embodiment of the present invention.

[0071] Referring to FIG. 2, a device (hereinafter referred to as a learning device) used to train a generative artificial intelligence model can exchange information or electrical signals with a user terminal (100) or a generative artificial intelligence device (300) including a generative artificial intelligence model, directly or through a network. At this time, the network can include the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), 3G, 4G, LTE (long term evolution), VoLTE (voice over LTE), 5G NR (new radio), Wi-Fi (wireless-fidelity), Bluetooth, NFC, RFID (radio frequency identification), a home network, the Internet of Things (IoT), etc.

[0072] The user terminal (100) may include at least one input device, output device, memory, communication device, and processor, and may be an electronic device in the form of a computer, tablet, or smartphone.

[0073] The input device can receive commands or data to be used in a component (e.g., a processor) of the user terminal (100) from outside the user terminal (e.g., a user).

[0074] The input device may include, for example, a microphone, a mouse, a keyboard, a digital pen (e.g., a stylus pen), a touch panel, etc. The touch panel may be formed of a capacitive touch panel that detects changes in electrical signals, a pressure-sensitive touch panel that detects physical pressure, an infrared touch panel that forms an infrared grid and detects changes in the infrared grid, a surface ultrasonic touch panel that forms an ultrasonic grid and detects changes in the ultrasonic grid, etc.

[0075] The output device may include a light, a speaker, a text display device, and a display panel. In particular, the display panel may include various types of display panels such as a liquid crystal display (LCD) panel, an organic light emitting diode (OLED) panel, an active-matrix organic light-emitting diode (AM-OLED), a liquid crystal on silicon (LCoS), a quantum dot light-emitting diode (QLED), a digital light processing (DLP), a plasma display panel (PDP) panel, an inorganic LED panel, and a micro LED panel, but is not limited thereto. Meanwhile, the display panel may also form a touch screen together with the above-described touch panel, and may be formed as a flexible panel.

[0076] Memory temporarily or non-temporarily stores various programs and data and transmits the stored information to the processor upon request. Furthermore, memory can store various information required for the processor's operations, processing, or control operations in electronic format.

[0077] The memory may include, for example, at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM. The ROM may include, for example, conventional ROM, EPROM, EEPROM, and / or MASK-ROM. The RAM may include, for example, DRAM and / or SRAM. The auxiliary memory may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a flash memory device, a Secure Digital (SD) card, a solid state drive (SSD), a hard disk drive (HDD), an optical media such as a magnetic drum, a compact disc (CD), a DVD, or a laser disc, a magnetic tape, a magneto-optical disc, and / or a floppy disk.

[0078] The communication device may be a wired communication device or a wireless communication device, and may include both, but it is preferable to include a wireless communication device. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared data association (IrDA), or near field communication (NFC). Mobile communication technologies may include Wi-Max, LTE (Long Term Evolution), 5G, 6G, and subsequent communication standard technologies. The wireless communication interface may be implemented using an antenna, a communication chip, a substrate, or the like that can transmit electromagnetic waves to the outside or receive electromagnetic waves transmitted from the outside.

[0079] The processor is connected to the configuration of a user terminal (100) including a memory, and can control an operation performed in a learning device (200) or a generative artificial intelligence device (300) by executing at least one instruction stored in the memory as described above. In particular, the processor may be implemented not only as one processor but also as multiple processors. In this case, one or more processors may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a graphics-only processor such as a GPU (Graphics Processing Unit), a VPU (Vision processing unit), or an artificial intelligence-only processor such as an NPU (Neural Processing Unit), a TPU (Tensor Processing Unit).

[0080] The user terminal (100) can transmit data including various pictures drawn by one artist to the learning device (200) directly or through a network. Thereafter, the learning device (200) can train the generative artificial intelligence device (300) based on the data transmitted from the user terminal (100). The generative artificial intelligence device (300) can transmit output data to the learning device (200) or the user terminal (100) directly or through a network. At least two of the user terminal (100), the learning device (200), and the generative artificial intelligence device (300) can be implemented by being physically coupled to each other.

[0081] FIG. 3 is a block diagram illustrating the configuration of a device used to train a generative artificial intelligence model according to one embodiment of the present invention.

[0082] Referring to FIG. 3, the learning device (200) may include a memory (220) in which at least one program command is stored and at least one processor (210) that executes at least one program command.

[0083] The memory (220) and the auxiliary memory device (260) can temporarily or non-temporarily store various programs or data, and transmit the stored information to the processor (210) according to a call of the processor. The various programs or data stored in the memory (220) and the auxiliary memory device (260) may include a generative artificial intelligence model or an artificial neural network model for generating a weight matrix. The generative artificial intelligence model and the artificial neural network model for generating a weight matrix may be the same. It is preferable that the artificial neural network model is one that has pre-learned information for distinguishing each object within graphic data such as an image. When training the artificial neural network model with input data (e.g., line drawing data, coloring data, background data, and person data), some of the data stored within the artificial neural network model may be deleted and then new data may be generated. That is, training may be performed in an overwrite manner.

[0084] In addition, the memory (220) can store various information required for the calculation, processing, or control operations of the processor (210) in an electronic format. The user can store data created or collected by the user in the memory (220) through the user terminal (100).

[0085] The memory (220) or the auxiliary memory (260) may include, for example, at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM. The ROM may include, for example, a conventional ROM, EPROM, EEPROM, and / or MASK-ROM. The RAM may include, for example, DRAM and / or SRAM. The auxiliary memory may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a flash memory device, an SD (Secure Digital) card, a solid state drive (SSD), a hard disk drive (HDD), an optical media such as a magnetic drum, a compact disc (CD), a DVD, or a laser disc, a magnetic tape, a magneto-optical disc, and / or a floppy disk. Additionally, the memory (220) or auxiliary memory device (260) may be implemented in a form that supports linkage with a storage physically located outside the learning device (200), or may be configured as a logical combination of a storage located outside the learning device (200) and a storage located inside the learning device (200).

[0086] The initial data stored in the memory (220) or auxiliary storage device (260) may be data in a tagged or labeled form, including not only graphic data but also text or delimiters. In one embodiment, the initial data may be entirely generated by a single author. In this case, the initial data has the advantage of directly reflecting the author's habits and characteristics, thereby enabling more consistent output data.

[0087] The processor (210) is connected to the configuration of the learning device (200) including the memory (220), and can control the overall operation performed in the learning device (200) by executing at least one instruction stored in the memory (220) as described above. In particular, the processor (210) may be implemented not only as one processor but also as multiple processors. In this case, one or more processors may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a GPU (Graphics Processing Unit), a VPU (Vision processing unit), or a AI-specific processor such as an NPU (Neural Processing Unit), a TPU (Tensor Processing Unit). When the memory (220) or the auxiliary memory (260) includes an artificial neural network model, the processor (210) can perform learning of the artificial neural network model.

[0088] The transceiver (230) may be a wired communication device or a wireless communication device, and may include both. However, it is preferable to include a wireless communication device so that information can be exchanged with the above-described network, user terminal (100), or generative artificial intelligence device (300). When the learning device (200) includes the transceiver (230), the learning device (200) may be a server. The transceiver (230) may obtain initial data from a user and collect initial data from an external source (e.g., the web).

[0089] The input device (240) may include, for example, a microphone, a mouse, a keyboard, a digital pen (e.g., a stylus pen), a touch panel, etc. The touch panel may be formed of a capacitive touch panel that detects changes in electrical signals, a pressure-sensitive touch panel that detects physical pressure, an infrared touch panel that forms an infrared grid to detect changes in the infrared grid, a surface ultrasonic touch panel that forms an ultrasonic grid to detect changes in the ultrasonic grid, etc. The device (200) may determine whether or not to proceed with a process related to model learning, such as matrix regeneration, or the order thereof, based on input through the input device (240).

[0090] The output device (250) may include a light, a speaker, a text display device, and a display panel. In particular, the display panel may include various types of display panels such as an LCD (Liquid Crystal Display) panel, an OLED (Organic Light Emitting Diodes) panel, an AM-OLED (Active-Matrix Organic Light-Emitting Diode), an LcoS (Liquid Crystal on Silicon), a QLED (Quantum dot Light-Emitting Diode), a DLP (Digital Light Processing), a PDP (Plasma Display Panel) panel, an inorganic LED panel, and a micro LED panel, but is not limited thereto. Meanwhile, the display panel may configure a touch screen together with the above-described touch panel, and may be formed of a flexible panel. Output data generated by the generative artificial intelligence model may be visually expressed through the output device (250).

[0091] FIG. 4 is a flowchart illustrating a method for training a generative artificial intelligence model and obtaining output values ​​according to one embodiment of the present invention.

[0092] Referring to Fig. 4, initial data must be acquired in order to train a generative artificial intelligence model or obtain output values. The initial data may be creative data directly generated by the author or a person. The device (200) may acquire the initial data by receiving the initial data from an external source related to the author or by collecting the author's initial data from the web. In this case, the device (200) may collect a preset quantity of initial data (e.g., approximately 100 or more). The initial data may be converted into data (e.g., line drawing data, coloring data, background data, character data) that is input to the model for matrix generation through classification or processing. This conversion process may be referred to as an extraction process.

[0093] A device (200) for training a generative artificial intelligence model can obtain line drawing data and coloring data, and generate a drawing style matrix based on the line drawing data and the coloring data. At this time, the device (200) can obtain an artificial neural network model, train the artificial neural network model on the line drawing data to generate a line drawing matrix, train the artificial neural network model on the coloring data to generate a coloring matrix, and then merge the line drawing matrix and the coloring matrix to generate a drawing style matrix. The merging between matrices is preferably performed based on addition between two matrices and multiplication between two matrices or one matrix and one number. The method for obtaining a matrix through training an artificial neural network model can be based on the amount of change in a weight matrix before and after training the model, as described above. The process of generating a background matrix from background data is also preferably performed similarly. The background data can be referred to as material data. The process of generating a character matrix from character data may include a process of collecting or learning gender data and clothing data in addition to the above-described process. Alternatively, character data and clothing data can be separated to create character matrices and clothing matrices, respectively.

[0094] In one embodiment, the outline data may be graphic data that includes only black and white, a single color, or limited colors, and is preferably data indicating information about the outline of an object in the graphic data. The coloring data may be data indicating information about the colors used in the graphic data, and is preferably data that includes saturation, brightness, etc. of the graphic data. The character data may be data indicating the appearance of a character in a webtoon, and in this case, the user of the generative artificial intelligence model can generate the intended output data by using only a delimiter such as the character's name without having to prompt the character's appearance at length every time. The background data is preferably data indicating the appearance of an object other than a character. The background data may be referred to as material data, post-processing data, or other data depending on the usage environment.

[0095] The artificial neural network models used to generate the line drawing matrix, the color matrix, the style matrix, the background matrix, and the character matrix may be the same, and may differ depending on the characteristics of each data. In addition, the artificial neural network models used to generate the matrices may be stored in the memory (220) of the device, but may not be stored in the memory (220) but may be located externally, and after receiving learning data from the device (200), data for generating the line drawing matrix, the color matrix, the style matrix, the background matrix, and the character matrix, such as weight matrices, may be returned.

[0096] The device (200) can generate at least two of a painting matrix, a background matrix, and a character matrix, and then merge the generated multiple matrices to generate a merge matrix. The device (200) that has obtained the merge matrix can select an engine, i.e., a generative artificial intelligence model, and then obtain output data through an input including the merge matrix. The data input to the generative artificial intelligence model can be a merge matrix and a text-based prompt, or can be input through a user interface (Graphical User Interface; GUI) including the merge matrix.

[0097] FIG. 5 is a flowchart illustrating a process of obtaining character data, line drawing data, color data, or background data from initial data according to one embodiment of the present invention.

[0098] The device (200) can collect or acquire initial data. The initial data may be graphic data generated by the artist. The device (200) can obtain the artist's original data by receiving the original data directly from the artist or by collecting the artist's original data from the web. In this case, the artist includes a composer, lyricist, painter, novelist, cartoonist, etc., and while the artist basically refers to an individual (one person), it can also include an organization.

[0099] In step S510, the device (200) that acquired the initial data can process the initial data into a form that is easy to process by an artificial intelligence model or an artificial neural network model. The method by which the device (200) processes the data may include error or missing value processing, scaling and normalization, dimensionality reduction, data augmentation, data cleansing, tokenization, embedding, verification, etc. The normalization process may include a step of calculating a quality index of each initial data, and if the quality index includes the size of the data, a step of distinguishing the minimum size of the graphic data and removing data below a certain size may be included. For example, data of about 200 KB or 262,144 bytes or less in the data source or data converted to a specific format (e.g., extension) may be removed in step S510. Alternatively, all of a certain amount of data may be converted to the same size (e.g., 512,512 bytes).

[0100] Additionally, a process for refining or modifying images of a specific form may be included. For example, graphic data intentionally created by an artist in a different style may be removed, or graphic data containing special effects may be modified to prevent them from affecting model performance. Furthermore, the process of processing data by the device (200) may include a tagging process, which adds text describing the characteristics of the graphic data. This tagging process can emphasize the style, material, and characteristics of the graphic data.

[0101] In step S520, the device (200) can classify the processed initial data into multiple categories (or categories). The classification performed by the device (200) may be at least one of binary classification, multi-class classification, and multi-label classification. Binary classification is a classification method that assigns an input to one of two mutually exclusive categories, and multi-class classification is a problem that assigns an input to one of three or more categories. Binary classification and multi-class classification have the characteristic that each input belongs to only one category. Multi-label classification is a classification method in which each input can belong to more than one category. The initial data or the processed initial data may be data labeled with a category to be classified or unlabeled data, or may include both. When the device (200) acquires labeled data, it is preferable to perform the classification based on each label of the data. Alternatively, the classification may be performed based on a tag included in the graphic data during the tagging process. In one embodiment, step S520 may precede step S510, i.e., initial data classified into multiple categories may be processed within each category.

[0102] Among the above multiple categories, at least two are preferably related to line drawing data and coloring data, respectively, and the other categories may be related to background data, character data, etc. Alternatively, the multiple categories may include at least one of a painting style category, a background category, and a character category. When the multiple categories include a painting style category, the line drawing data and the coloring data may be acquired based only on initial data classified into the painting style category among the initial data. Similarly, the background data may be acquired based on initial data classified into the background category, and the character data may be acquired based on initial data classified into the character category. The background data may be background and material data including information related to the background and the material, and a category may include background data and material data respectively, and may also include special effect data, etc. The categories are not limited to the above examples and may be defined differently according to various properties related to the graphic data.

[0103] At step S530, the device (200) can verify whether the number of data in each category is greater than or equal to a preset number. Preferably, the preset number is 100 or less, and may vary by category. For example, the device (200) can verify whether the number of line drawing data is greater than or equal to 100, and the number of coloring data is greater than or equal to 150.

[0104] If the number of data in each category exceeds a preset number, the device (200) may perform error data removal. Error data removal performed by the device (200) may include noise removal. Specifically, the device (200) may remove initial data containing noise from among a plurality of initial data, or may remove only the portion corresponding to noise from the initial data containing noise. In this case, noise may refer to elements that interfere with learning. For example, it may refer to special lighting effects and blur processing effects included by the artist in the painting.

[0105] If there is a category with less than a preset number of data in step S530, the device (200) can acquire additional data. In this case, the additional data may include similar data generated from a separate model after training at least a portion of the initial data in the separate model. For example, if the number of line drawing data is insufficient, data may be generated by extracting only the line drawing (outline) from the initial data that is not classified as line drawing data, and this may be utilized as line drawing data. In this case, the device (200) may transmit the initial data and a request to an external source to extract only the line drawing from the initial data, and receive the line drawing data as a result. In addition, initial data removed in step S510 due to reasons such as size inadequacy may be utilized as additional data.

[0106] The device (200) that has acquired additional data can repeat step S520, and the additional data acquisition process can be repeated two or more times until the number of data in each category exceeds a preset number. Finally, the device (200) can acquire at least one of character data, line drawing and coloring data, and background data from the initial data.

[0107] FIG. 6 is a flowchart illustrating a process for generating output data according to one embodiment of the present invention.

[0108] Referring to FIG. 6, the device (200) can generate a merge matrix from at least two of a merge ratio, a character matrix, a style matrix, and a background matrix. The merge ratio can be set differently for each block or node of the artificial neural network, and thus can be set differently for each layer of the artificial neural network. Generally, the merge ratio is set for each of the input layer (IL), the hidden layer (HL), and the output layer (OL). The merge ratio can also further designate the reflection ratio of the existing weight matrix of the artificial neural network model that has not learned character data, line drawing data, coloring data, or background data through the device (200). In addition, the device (200) can further set a learning rate for generating the merge matrix internally or based on external input. The learning rate can be different for each model or data, and can be set and input to the artificial neural network model in the form of a learning rate scheduler to prevent the same learning rate from being applied during repeated learning. The learning rate scheduler can be modified during training.

[0109] Additionally, a merge matrix can be generated through two or more merges between matrices. For example, after merging the style matrix and the background matrix, a character matrix can be additionally merged. In this case, the merge ratio includes a first merge ratio and a second merge ratio, and it is preferable that the first and second merge ratios are independent of each other.

[0110] At step S620, the device (200) may select an engine for generating output data. The engine may be referred to as a generative artificial intelligence model, and may be referred to as a checkpoint, particularly when used in a diffusion model. At step S620, the device (200) may select an engine based on similarity with the initial data. At this time, the engine may be a generative artificial intelligence model existing in the memory (220) of the device or a part of a generative artificial intelligence model. After selecting the engine, the device (200) may generate at least one piece of output data through a matrix (e.g., a painting matrix, a merge matrix) and a prompt.

[0111] At step S630, the device (200) can determine whether to confirm output data. Step S630 can be determined based on information received from an external source (e.g., a writer or a user) by outputting the output data, and the device (200) can directly measure the similarity between the output data and the initial data. If the similarity is greater than a preset threshold, the device (200) can confirm the output data and terminate the model learning process. The preset threshold can be approximately 0.7.

[0112] If the device (200) determines to modify the output data at step S630, the device (200) may modify the merge matrix or engine. To modify the merge matrix, the merge ratio may be adjusted, or the matrix may be regenerated by adding or deleting at least one of the character data, line drawing data, coloring data, and background data. This process may be repeated two or more times until the output data is confirmed.

[0113] The process described above effectively reproduces the artist's artwork, not only creating new images in the artist's style, but also learning the artist's habits and characteristics to generate new forms of art that did not exist in the training data (i.e., the initial data). Furthermore, it can generate finished manuscripts resembling the artist's artwork from just rough sketches and storyboards, and it can provide a system capable of automated coloring. Specifically, when applied to webtoon production, it can offer advantages such as increased production speed, reduced dependence on the artist's labor, and automated coloring, thereby enhancing production speed and efficiency.

[0114] Although the preferred embodiments of the present invention have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present invention as claimed in the claims. Furthermore, such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. A method for training a generative artificial intelligence model by a processor of a device, Step of acquiring outline data; Step of acquiring coloring data; and A method comprising: a step of generating a painting style matrix based on the line drawing data and the coloring data.

2. In paragraph 1, Step of acquiring initial data; and A method further comprising a step of extracting the line drawing data and the coloring data from the initial data.

3. In paragraph 2, The step of extracting the line drawing data and the coloring data from the initial data is as follows: A step of classifying the above initial data into a plurality of preset categories; and A method comprising: a step of extracting the line drawing data and the coloring data based on the above category.

4. In paragraph 3, The step of extracting the line drawing data and the coloring data from the initial data is as follows: A method further comprising the step of removing error data from the initial data.

5. In paragraph 4, The step of extracting the line drawing data and the coloring data from the initial data is as follows: A method further comprising: a step of additionally generating the line drawing data or the coloring data based on the initial data when the line drawing data or the coloring data is less than a preset number.

6. In paragraph 1, The step of generating a painting style matrix based on the above line drawing data and the above coloring data is: A step of obtaining a first artificial neural network model including a weight matrix; A step of training the line drawing data and the coloring data to the artificial neural network model; and A method comprising: a step of generating the painting style matrix based on a change in the weight matrix included in the first artificial neural network model after learning; 7. In paragraph 1, Step of acquiring character data; and A method further comprising: a step of generating a character matrix based on the character data; 8. In paragraph 7, The step of generating a character matrix based on the above character data is: A step of obtaining a second artificial neural network model including a second weight matrix; A step of training the second artificial neural network model with the person data; and A method comprising: a step of generating the character matrix based on changes in the weight matrix included in the second artificial neural network model after learning; 9. In paragraph 8, A method further comprising: a step of generating a merge matrix based on the above-mentioned painting matrix and the above-mentioned character matrix.

10. In paragraph 9, The step of generating a merge matrix based on the above-mentioned painting matrix and the above-mentioned character matrix is ​​as follows: Step of obtaining a merge ratio; and A method comprising: a step of merging the painting style matrix and the character matrix based on the merging ratio.

11. In paragraph 10, A method further comprising a step of merging the painting matrix or the merging matrix into a previously stored generative artificial intelligence model.

12. In paragraph 11, A step of obtaining output data from the above generative artificial intelligence model; A step of modifying the above merge ratio; A step of regenerating a merge matrix based on the above modified merge ratio; and A method further comprising the step of merging a re-generated merge matrix into a previously stored generative artificial intelligence model.

13. In paragraph 11, A step of obtaining output data from the above generative artificial intelligence model; Step of obtaining additional data; A step of regenerating a wind matrix or a merge matrix based on the above additional data; and A method further comprising a step of merging a re-generated painting matrix or a merging matrix into a previously stored generative artificial intelligence model.

14. In paragraph 11, A step of obtaining output data from the above generative artificial intelligence model; and A method further comprising: a step of visually displaying the output data; 15. In paragraph 9, Step of acquiring background data; and A method further comprising: a step of generating a background matrix based on the above background data.

16. In paragraph 15, A method wherein the step of generating a merge matrix based on the above-mentioned painting style matrix and the above-mentioned character matrix is ​​a step of generating a merge matrix based on the above-mentioned painting style matrix, the above-mentioned character matrix, and the above-mentioned background matrix.

17. In paragraph 15, Step of acquiring initial data; and A method further comprising the step of extracting the background data from the initial data.

18. In paragraph 7, Step of acquiring initial data; and A method further comprising a step of extracting the character data from the initial data.

19. A device used to train a generative artificial intelligence model, memory; and Processor; including; The above processor, A device that acquires outline data and coloring data, stores them in the memory, and generates a drawing style matrix based on the outline data and the coloring data.

20. A computer program stored in a computer-readable storage medium, The above computer program, when executed on a processor of the device, performs steps for training a generative artificial intelligence model, the steps comprising: Step of acquiring outline data; Step of acquiring coloring data; and A program comprising a step of generating a painting style matrix based on the line drawing data and the coloring data.

Images