Program, image identification method, and image identification system

Abstract

To provide an image identification system capable of obtaining a target image from a large quantity of images generated with a generation model, and using the image for creation of animation.SOLUTION: An image identification system 10 which identifies a generation image generated by a diffusion model 50 includes: a diffusion model 50 in which the diffusion model 50 using an additional learning model having learnt a learning image as an adapter generates for generating a generation image so as to interpolate a prescribed frame period on the basis of instruction information; and an image identifier 56 which captures the generation image, identifies if the generation image generated by the diffusion model 50 is suitable or unsuitable, disposes a generation image which is identified unsuitable, and allows the diffusion model 50 to generate again a generation image corresponding to the disposed frame.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a program, an image recognition method, and an image recognition system. [Background technology]

[0002] Traditionally, animation production has required generating a large number of different poses, expressions, and motions for specific characters, strictly following the character's design code. However, even professional animators can only produce around 10 to 20 images per day during inbetween frames, the process of drawing the intermediate frames between key animations. This number corresponds to about 2.5 seconds of the finished work. For this reason, technology was being considered that would use a program to generate images that could reproduce character movements.

[0003] In recent years, there has been remarkable progress in image generation technology using image generation AI (Artificial Intelligence), and the use of image generation AI to create animations is being considered. Image generation AI is capable of generating a large number of images in a short period of time by inputting certain generation conditions. One example of a generative model that makes up image generation AI is the diffusion model, which uses random numbers to generate noise, which is the starting point for image generation. It is known that the resulting image changes significantly each time the random numbers are changed. This mechanism that utilizes random numbers is inherently beneficial because it adds diversity to the images output by the generative model, making it possible to generate a wide variety of variations.

[0004] Another known image generation technology is the technology disclosed in Patent Document 1. Patent Document 1 states, "When a user inputs a change in index value related to movement, the standard skeleton movement is changed according to the axis parameters. The changed skeleton movement is displayed with the changed parts from the standard skeleton movement identified. This makes it possible to change the skeleton (animation) obtained by measuring the movement into a desired movement." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-026149 Summary of the Invention [Problem to be solved by the invention]

[0006] Even if an image generation AI generates multiple images of a character, some images will look unnatural when animated. This requires an animator to review each image generated by the image generation AI, discard inappropriate images, and then retry the process of trial and error. Since trial and error is a bottleneck when actually using image generation AI, a solution was needed. Therefore, a mechanism called CLIP was expected to allow users to specify prompts to control parameters within the diffusion model. However, the number of combinations of prompts and random numbers was enormous, and trial and error often relied on luck. In particular, as mentioned above, the initial values ​​of the diffusion model were random numbers, and the results changed each time the diffusion model generated an image. Therefore, multiple iterations were required to generate the desired image.

[0007] However, when using the technology described in Patent Document 1, an animator needs to input detailed index data for each character. Also, it is difficult to reproduce the facial expressions of characters using only the technology described in Patent Document 1. For this reason, it is difficult to use the technology described in Patent Document 1 in the production of animation.

[0008] The present invention has been made in view of the above circumstances, and aims to use images generated by a diffusion model in the production of animation. [Means for solving the problem]

[0009] The program of the present invention identifies generated images generated by a diffusion model, and causes a computer to execute the following steps: an image classifier imports generated images generated by a diffusion model that uses an additional learning model that has learned training images as an adapter to interpolate a predetermined frame period based on instruction information; an image classifier classifies the generated images generated by the diffusion model as suitable or unsuitable; and an image classifier discards generated images that the image classifier has identified as unsuitable, and causes the diffusion model to regenerate generated images corresponding to the discarded frames. [Effects of the Invention]

[0010] According to the present invention, images generated by a diffusion model can be used to create animations. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is an overall configuration diagram showing an overview of an image identification system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram illustrating an example of a hardware configuration of an image identification system according to an embodiment of the present invention. [Figure 3] 1 is a block diagram illustrating an example of a functional configuration of an image identification system according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram illustrating an example of the internal functional configuration of an image generating unit according to an embodiment of the present invention. [Figure 5] 1 is a flowchart illustrating an example of a LoRA learning process according to an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a learning process of an image classifier according to an embodiment of the present invention. [Figure 7] 10A and 10B are diagrams showing examples of learning images according to an embodiment of the present invention and examples of classification results of suitable or unsuitable in-between images. [Figure 8] 10 is a flowchart illustrating an example of an image generation process using a diffusion model and an in-between image classification process using an image classifier according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] In the following embodiments, the program that generates images is referred to as "image generation AI." The image generation AI of this embodiment has the function of automatically selecting only preferred images from images generated by a generative model that has been fine-tuned specifically to generate a specific character, and is capable of feedback learning without updating the generative model. The image generation AI's automatic image selection process automatically finds images that satisfy the character's consistency and design code from a group of images generated through a huge number of trial results, eliminating trial and error from the image generation AI workflow. Note that the image generation AI of this embodiment uses a diffusion model as its generative model.

[0014] Here, we will explain the design code. A design code is a guideline for character creation. Traditionally, design codes serve as guidelines for animators to draw consistent objects (such as characters, buildings, and structures) and to determine the consistency of objects across multiple images. Animators are assumed to be those who draw characters or those responsible for drawing characters, such as those who determine whether the characters are drawn in accordance with the design code. By referring to the design code during drawing, animators can maintain consistency for characters whose details vary depending on the scene or image they are drawn in throughout a segment of animation (e.g., an entire scene, an entire episode, or an entire work). Note that the design code may be any information that can be used to determine whether an object drawn in an image is correct. For example, the design code may be a combination of text data about the object (information about the position and size of each component) and image data of the object (e.g., training images A1 to A3 shown in FIG. 7, which will be described later) or may be information extracted from such information or training data obtained using the text data and image data.

[0015] The image generation AI of this embodiment has a function to automate OK judgment, which can determine whether the results generated by the diffusion model meet a specific purpose. Therefore, the image generation AI of this embodiment can automatically select suitable images, i.e., more desirable images, from the vast amount of trial and error results of the image generation AI. The image generation AI can automatically select more desirable images by using an image classifier, which is a model fine-tuned through incremental learning such as LoRA. Note that incremental learning is a method that enables the generation of images with features not possessed by the original model by adding training data.

[0016] Here, we will briefly explain LoRA (Low Rank Adaptation). LoRA is a method for fine-tuning additional weights using a very small amount of additional data, and is used as an incremental learning method to customize image generation AI for specific applications. In LoRA, the weights of the existing generative model are locked, i.e., additional weights are learned without updating the weights. For this reason, LoRA is very commonly used in image generation AI as a method that allows incremental learning without damaging the characteristics of the existing generative model. In a generative model that has been fine-tuned through incremental learning such as LoRA, the purpose of image generation is narrowed down to the training target of LoRA. This makes it possible to train an image classifier that can distinguish whether or not the image generated by the generative model is appropriate as an output result. Below, we will explain a system that can realize the image generation AI of this embodiment.

[0017] [One embodiment] <Overall configuration example of an image recognition system> First, a configuration example of an image recognition system according to an embodiment of the present invention will be described. This image recognition system is configured by combining an image generation server that generates images with an information processing terminal. The image recognition system recognizes images generated by the image generation server.

[0018] <Image Recognition System Overview> FIG. 1 is a diagram showing the overall configuration of an image identification system 10 according to one embodiment of the present invention. The image identification system 10 includes an image generation server 1, a smartphone 2A, and a PC (Personal Computer) 2B. The smartphone 2A and the PC 2B can be connected to the image generation server 1 via a network N such as the Internet. In the following description, the smartphone 2A and the PC 2B are collectively referred to as an information processing terminal 2.

[0019] The image generation server 1 is an example of a first information processing device constituting the image recognition system 10. The image generation server 1 manages a program used as the image generation AI, a program for identifying whether an image generated by the image generation AI is suitable or unsuitable, and various data. A diffusion model 50 shown in FIG. 4, which will be described later, is used as the image generation AI. The image generation server 1 starts generating an image based on a request from the authenticated information processing terminal 2, and transmits image data of the generated image to the information processing terminal 2.

[0020] The information processing terminal 2 processes image data received from the image generation server 1. The image generation program according to this embodiment generates an image that interpolates a predetermined frame period based on instruction information input via an input device 26 (an example of an input unit). Of the images generated by the image generation program, an image that is determined to be suitable is displayed on an output device 27 (an example of a display unit).

[0021] The smartphone 2A uses a touch panel display device in which the input device 26 and the output device 27 are integrated. The smartphone 2A may be a tablet terminal. The input device 26 and the output device 27 are separate devices in the PC 2B. The PC 2B may be a desktop PC, with the input device 26 and the output device 27 separately connected to the desktop PC.

[0022] The information processing terminal 2 is an example of a second information processing device constituting the image recognition system 10. The information processing terminal 2 can store data including an image generation program from the image generation server 1 in a storage device 22 shown in FIG. 2, which will be described later. Here, the information processing terminal 2 is described as being operated by a user. The user is assumed to be a person who operates the information processing terminal 2 to instruct the image generation AI to generate an image. The user may also include the animators mentioned above.

[0023] The information processing terminal 2 can also read data including an image generation program from a memory, an optical disk, etc. The information processing terminal 2 then executes the image generation program read from the storage device 22 to realize image generation processing. Furthermore, when image data generated by the image generation server 1 is transmitted to the information processing terminal 2, the information processing terminal 2 can also display the image using, for example, an internet browser.

[0024] The information processing terminal 2 selects a program based on an operation signal input from the input device 26 in response to an operation performed by the user, and outputs a video signal to the output device 27 that matches the screen of the output device 27. The output device 27 displays an image based on the video signal. The operation signal input from the input device 26 is, for example, a signal corresponding to each operation button on a keyboard. The user can input instructions and operate the image generation program through the input device 26.

[0025] The information processing terminal 2 stores the image data downloaded from the image generation server 1 in a storage device 22 shown in Fig. 2, which will be described later. The information processing terminal 2 also performs processes such as reading the image data from the storage device 22 and executing a program, drawing a screen in accordance with an operation signal input from the input device 26, and displaying a screen by the output device 27. For example, in response to an operation by a user via the input device 26, the information processing terminal 2 displays on the output device 27 a screen on which an image based on the image data read from the storage device 22 is drawn.

[0026] Operations for the image generation program input from the input device 26 include various command inputs, such as specifying a frame period, specifying a training image for LoRA, etc. Another example of an operation performed from the input device 26 is a tap operation in which the screen of the output device 27 is touched with a finger or a pen.

[0027] <Example of hardware configuration for image recognition system> Next, an example of the hardware configuration of the image identification system 10 according to an embodiment will be described. FIG. 2 is a block diagram showing an example of the hardware configuration of the image identification system 10. As shown in FIG.

[0028] (Image generation server configuration example) The image generation server 1 is an example of a computer that operates as a computer capable of executing various programs. The image generation server 1 includes a processing device 11, a storage device 12, and a network interface 14, all of which are connected to a bus 13.

[0029] The processing device 11 is configured with at least one of, for example, a CPU (Central Processing Unit), an MPU (Microprocessor Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The processing device 11 reads program code of software that realizes each function according to the present embodiment from the storage device 12, loads the program code into a temporary storage unit (not shown) provided in the storage device 12, and executes the program code. The processing device 11 performs, for example, arithmetic processing for image generation and processing necessary for drawing objects on the screen of the information processing terminal 2.

[0030] The object is, for example, various computer graphics images (hereinafter abbreviated as "image") displayed on the screen of the information processing terminal 2. In this embodiment, it is assumed that a two-dimensional image object is displayed on the screen.

[0031] The storage device 12 is configured by, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM may be an optical disk, a magneto-optical disk, a DVD (Digital Versatile Disc)-ROM, a CD-ROM, a Blu-ray (registered trademark) disk, or the like. The RAM may be an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), or the like. Variables, parameters, etc. generated during the arithmetic processing of the processing device 11 are temporarily written to the storage device 12, and these variables, parameters, etc. are read out by the processing device 11 as appropriate.

[0032] The storage device 12 is configured with at least one of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a flash memory. The storage device 12 stores the OS (Operating System) of the image generation server 1, various parameters, and programs for operating the image generation server 1. As described above, the storage device 12 stores programs, data, etc. necessary for the operation of the processing device 11, and is used as an example of a computer-readable non-transitory storage medium that stores programs executed by the image generation server 1.

[0033] For example, a network interface card (NIC) or the like is used as the network interface 14. The network interface 14 is capable of transmitting and receiving various data to and from the information processing terminal 2 via a dedicated line or the like connected to a terminal of the NIC and via the network N.

[0034] (Example of information processing terminal configuration) The information processing terminal 2 is an example of a computer that operates as a computer capable of executing various programs. The information processing terminal 2 includes a processing unit 21, a storage unit 22, and a network interface 24, all of which are connected to a bus 23.

[0035] The processing device 21 is configured with at least one of, for example, a CPU, an MPU, a GPU, and an FPGA. The processing device 21 reads out program code of software that realizes each function according to this embodiment from the storage device 22, loads it into a temporary storage unit (not shown) provided in the storage device 22, and executes the program code. The processing device 21 performs, for example, arithmetic processing for image generation and processing necessary for drawing objects on the screen of the information processing terminal 2. The processing device 21 also performs processing such as processing of the OS of the information processing terminal 2 and management of input and output of data performed by each unit in the information processing terminal 2. When processing information related to image generation, the processing device 21 can output an image signal to the output device 27 through the input / output interface 25.

[0036] The storage device 22 is composed of, for example, a ROM and a RAM. The ROM may be an optical disk, a magneto-optical disk, a DVD-ROM, a CD-ROM, a Blu-ray (registered trademark) disk, or the like. The RAM may be an SRAM, a DRAM, or the like. Variables, parameters, and the like generated during the arithmetic processing of the processing device 21 are temporarily written to the storage device 22, and these variables, parameters, and the like are read out by the processing device 21 as appropriate. The processing device 11 also performs processing required to draw, for example, two-dimensional objects that constitute characters or backgrounds on the screen of the output device 27.

[0037] The storage device 22 is configured with at least one of, for example, an HDD, an SSD, and a flash memory. The storage device 22 stores the OS of the information processing terminal 2, various parameters, programs for causing the information processing terminal 2 to function, programs for generating images, etc. As described above, the storage device 22 stores programs, data, etc. necessary for the operation of the processing device 21, and is used as an example of a computer-readable non-transitory storage medium that stores programs executed by the information processing terminal 2.

[0038] For example, a NIC or the like is used as the network interface 24. The network interface 24 is capable of transmitting and receiving various data to and from the image generation server 1 via the network N through a dedicated line or the like connected to a terminal of the NIC, and of communicating with other information processing terminals 2.

[0039] The input / output interface 25 converts operation signals received from the input device 26 into data in a predetermined format and passes the converted data to the processing device 21. The input / output interface 25 also converts screen data drawn by the processing device 21 into video signals and outputs them to the output device 27.

[0040] The input device 26 is a device that accepts input instructions or various types of information from a user. An example of the input device 26 is a pointing device that can input coordinate information of a position designated by the user. This pointing device is a mouse, a touch panel device, or the like. A touch panel device is configured by combining the input device 26 and the output device 27. The input device 26 may also be a keyboard, a mouse, or the like.

[0041] The output device 27 is a device that outputs information processed by the processing device 21. The output device 27 is, for example, a display device (a display device, a touch panel device, etc.). When the output device 27 is a display device, an image based on a video signal received from the input / output interface 25 is displayed on the display device.

[0042] <Example of functional configuration of image recognition system> Next, an example of the functional configuration of the image identification system 10 will be described with reference to FIG. 3 is a block diagram showing an example of the functional configuration of the image identification system 10. In FIG. 3, the communication network N is omitted.

[0043] (Example of functional configuration of image generation server) The image generation server 1 includes a communication unit 31, a pre-learning unit 32, an image generation unit 33, and a storage unit .

[0044] The communication unit 31 controls communication with the information processing terminal 2. For example, when the communication unit 31 receives instruction data or the like transmitted from the information processing terminal 2, the communication unit 31 issues an instruction to the pre-learning unit 32 or the image generation unit 33 to operate them. The communication unit 31 also transmits image data of an image generated by the image generation unit 33 to the information processing terminal 2. The functions of the communication unit 31 are realized by the processing device 11, the network interface 14, etc. shown in FIG. 2 .

[0045] The pre-learning unit 32 performs pre-learning, which is a prerequisite for image generation. The pre-learning unit 32 performs additional learning using, for example, LoRA. The pre-learning unit 32 also performs pre-learning of an image classifier 56 shown in FIG. 4, which will be described later.

[0046] The image generation unit 33 generates an image based on the results of pre-learning by the pre-learning unit 32. The image generation is performed by a diffusion model 50 of the image generation unit 33, as shown in FIG. 4, which will be described later. The image generation unit 33 also classifies the generated image as suitable or unsuitable. The image classification is performed by an image classifier 56 of the image generation unit 33, as shown in FIG. 4, which will be described later. The image generation unit 33 then stores image data of the image classified as suitable in the storage unit 34. This image data is output to the communication unit 31 and transmitted to the information processing terminal 2. A detailed configuration example of the image generation unit 33 is shown in FIG. 4, which will be described later. The functions of the pre-learning unit 32 and the image generating unit 33 are realized by the processing unit 11 and the storage device 12 shown in FIG.

[0047] The storage unit 34 stores image data and an image generation program. The function of the storage unit 34 is realized by the storage device 12 shown in Fig. 2. The storage unit 34 also stores data of a user who instructs the generation of an image. The user data is used for user authentication processing, etc.

[0048] (Example of functional configuration of information processing terminal) The information processing terminal 2 includes a communication unit 41, an input unit 42, an input receiving unit 43, a drawing unit 44, an image output unit 45, an output unit 46, and a storage unit 47.

[0049] The communication unit 41 transmits instruction data including a user's instruction for pre-learning and an instruction for image generation to the image generation server 1. Then, the communication unit 41 writes the image data transmitted from the image generation server 1 to the storage unit 47. The function of the communication unit 41 is realized by the processing device 21 and the network interface 24 shown in FIG. 2.

[0050] The input unit 42 outputs an operation signal generated based on an operation input by the user to the input receiving unit 43. An operation input by the user is, for example, an operation of tapping on the screen of the input device 26, an operation of specifying a specific icon on the screen with the input device 26, etc. The function of the input unit 42 is realized by the input device 26 shown in FIG. 2.

[0051] The input receiving unit 43 receives an operation signal from the input unit 42. The function of the input receiving unit 43 is realized by the input / output interface 25 shown in Fig. 2. The operation signal received by the input receiving unit 43 causes the communication unit 41 to transmit instruction data and the output unit 46 to display an image.

[0052] The drawing unit 44 draws an image to be displayed on the output unit 46 based on the operation signal received by the input receiving unit 43. The drawn image includes, for example, an image generated by the image generating server 1 as well as text and the like.

[0053] The image output unit 45 outputs the image drawn by the drawing unit 44 to the output unit 46. For example, an image with an OK label attached, as shown in FIG. 7 (described later), is output to the output unit 46. The function of the image output unit 45 is realized by the input / output interface 25 shown in FIG. 2.

[0054] The output unit 46 displays the image output by the image output unit 45. The function of the output unit 46 is realized by the output device 27 shown in FIG.

[0055] The storage unit 47 stores image data received by the communication unit 41 from the image generation server 1. The function of the storage unit 47 is realized by the storage device 22 shown in FIG.

[0056] [Technology Overview] Here, we will explain the outline of the technologies of the pre-learning unit 32 and the image generation unit 33 according to this embodiment. The image generation AI according to this embodiment is a technology that automatically selects only desirable results from models that are fine-tuned specifically for generating specific characters, with the aim of improving the productivity of drawing work in the animation production process.

[0057] As mentioned above, in the past, in order to get image generation AI to produce desirable results, it was necessary to manually test a huge number of parameters and manually select the desirable results through trial and error. This trial and error process was a major bottleneck in the practical application of image generation AI. In addition, there were certain restrictions on using image generation AI in commercial content due to the need to check that the image data output by image generation AI did not infringe on the copyrights or design rights of third parties.

[0058] Therefore, in this embodiment, an image discriminator is added to the output layer of the image generation AI, enabling the AI ​​to automatically extract only appropriate results as data for the animation production process for the character to be generated. The image discriminator in this embodiment has the function of automatically extracting only the preferred generated data from the results of automatically and comprehensively trying a huge number of parameter combinations. This image discriminator targets diffusion models, which are currently mainstream image generation AI and require trial and error. For example, an image discriminator that automatically selects only the preferred generated results from multiple images generated by a generative model fine-tuned using LoRA is introduced after the VAE (Variational Auto Encoder) decoder, which is the output layer of the diffusion model.

[0059] A feature of this image classifier is that it uses, for example, a Vision Transformer to train images for additional training by LoRA as suitable results. Furthermore, this image classifier achieves binary classification of generated results by learning using feedback from the animator's classification results of images output by the VAE encoder as either suitable or unsuitable. Binary classification by the image classifier makes it possible to automatically select only images that the image classifier finds suitable from images generated by a diffusion model fine-tuned by LoRA. Therefore, by using the image classifier, it is possible to selectively extract only images that satisfy the character's design intent and design code from character images, etc., generated probabilistically by a diffusion model.

[0060] If an image classifier using Vision Transformer can automatically select the output result of an image generation AI, the diffusion model can perform comprehensive and large-scale image generation trials, which is expected to dramatically improve the productivity of animation production using image generation AI. Furthermore, in the practical application process of image generation AI, the image classifier can learn from feedback obtained from animators. Therefore, the more the image generation AI related to this embodiment is used, the more the image classifier's learning progresses, and the image classifier can automatically extract results suitable for animation production sites.

[0061] FIG. 4 is a block diagram showing an example of the internal functional configuration of the image generating unit 33. As shown in FIG.

[0062] The image generation unit 33 includes a VAE encoder 51, a latent diffusion model (LDM) 52, a noise reduction model 54, a VAE decoder 55, and an image classifier 56. The VAE encoder 51, the latent diffusion model (LDM) 52, the noise reduction model 54, the VAE decoder 55, and the image classifier 56 are collectively referred to as a diffusion model 50. This diffusion model 50 is fine-tuned using LoRA to specialize in generating a specific character.

[0063] For example, a diffusion model 50 using LoRA (an incremental learning model) that has learned a training image Am as an adapter generates a generated image Bn so as to interpolate a predetermined frame period based on input instruction information. Therefore, the diffusion model 50 is used as the image generation AI according to this embodiment. Furthermore, an image recognition program included in the image generation unit 33 recognizes the generated image Bn generated by the diffusion model 50.

[0064] The diffusion model 50 encodes the training image Am in pixel space into latent space, and then learns a diffusion process. The diffusion process is a process in which the latent diffusion model 52 adds random information (e.g., noise) to the encoded training image Am. The diffusion model 50 then generates a generated image Bn by decoding the data, from which the random information (e.g., noise) has been removed using a de-diffusion process based on the diffusion process, into pixel space. The operation of each unit included in the diffusion model 50 will be described in detail below.

[0065] The latent diffusion model 52 has the function of a general diffusion model with the VAE encoder 51 as an input layer, and generates a noise image 53.

[0066] The noise removal model 54 removes noise from the noisy image 53. For example, a U-Net is used as the noise removal model 54. Instruction information for an image to be generated by the noise removal model 54 after removing noise is input from a text instruction unit 57 and an image instruction unit 58 to the noise removal model 54.

[0067] A text encoder such as CLIP is used as the text indicator unit 57. The text indicator unit 57 encodes the input text indicator information into a vector format and outputs the encoded text indicator information to the noise removal model 54 and the image classifier 56. The diffusion model 50 removes noise from the noise image 53 based on the encoded text indicator information. The image classifier 56 classifies the generated image Bn based on the encoded text indicator information.

[0068] The text instruction information in text format input from the text instruction unit 57 is, for example, information obtained by encoding a prompt string. The prompt string is, for example, information such as "1 girl, portrait, black hair, smile" and is used as a positive prompt and a negative prompt. The positive prompt is a string of elements that the diffusion model 50 is desired to generate, and the diffusion model 50 generates an image according to the elements written in the positive prompt. On the other hand, the negative prompt is a string of elements that the diffusion model 50 is desired to exclude from the image generated, and the elements of the image generated by the diffusion model 50 are deleted according to the elements written in the positive prompt. In this way, the diffusion model 50 generates an image according to the text instruction information.

[0069] For example, a ControlNet is used as the image instruction unit 58. The ControlNet can extend the functions of the diffusion model 50 so that the diffusion model 50 can generate an image intended by the animator. Various models such as Canny and Openpose can be selected as the ControlNet. Canny is a model for generating a line drawing from an image and then coloring the line drawing to generate an image. Openpose is a model for generating an image of a character that matches the pose of, for example, a stick figure. Here, a stick figure is a representation of the human body that is as deformed as possible, for example, an illustration in which the head is represented by a circle and the limbs and torso are represented only by lines.

[0070] Therefore, the image instruction information in image format input from the image instruction unit 58 is, for example, information obtained by encoding a stick figure pose image created based on the body structure of the character. The pose image may be, for example, an image of the stick figure looking to the side, an image facing forward, an image facing upward, etc. The diffusion model 50 removes noise based on the input image instruction information, and the image classifier 56 classifies the generated image Bn based on the image instruction information.

[0071] The VAE decoder 55 decodes the noise-removed image to generate an image. The images decoded and generated by the VAE decoder 55 are output as multiple generated images Bn (n is an integer equal to or greater than 2). The image generation unit 33 according to this embodiment uses LoRA. Therefore, the images decoded and generated by the VAE decoder 55 have an atmosphere similar to that of the image shown as the training image Am previously used in the LoRA learning process. Although the training image Am shown in FIG. 4 is one image, in reality, a dozen or so training images Am (m is an integer equal to or greater than 2) are used in the LoRA learning process.

[0072] The generated image Bn is an image that represents at least one of the following changes that occur over time in a frame period: rotation of an object, change in drawing angle, movement, deformation of an object, deletion of some elements of an object, addition of some elements to an object, change in the expression of an object, and change in color of an object. Objects are not only characters such as people, but also parts of images that move over time, such as animals, cars, and landscapes.

[0073] The image classifier 56 is a module unique to this embodiment and is installed after the VAE decoder 55. The image classifier 56 performs image classification processing to classify the generated images Bn generated by the diffusion model 50 as suitable or unsuitable. For example, the image classifier 56 classifies each of the generated images Bn described above and assigns either an OK label indicating that the image is suitable or an NG label indicating that the image is unsuitable to each image. The image surrounded by a dashed line at the top of the generated images Bn in Figure 4 is classified as suitable and assigned an OK label. The other images are classified as unsuitable and assigned an NG label. The image classifier 56 discards the generated images Bn classified as unsuitable and causes the diffusion model 50 to regenerate generated images Bn corresponding to the discarded frames.

[0074] During image classification training, the image classifier 56 uses the images used in LoRA training as initial training data (an example of first training data). Furthermore, during image classification training, the image classifier 56 uses the images output by the latent diffusion model 52 and the results of a human (e.g., an animator) evaluation of the images as training data (an example of second training data). Therefore, the image classifier 56 classifies the generated image Bn as suitable if the generated image Bn is generated in accordance with the object's shape information (also called a design code). For example, the image classifier 56 classifies an image as suitable if the generated image Bn has a similar atmosphere to the image shown in the training image Am used in LoRA pre-training and the character's structure is not flawed.

[0075] Here, the image generation unit 33 will be described in detail by dividing it into three parts: LoRA, which is an existing technology, the latent diffusion model 52, and the image classifier 56, which is a unique module of this embodiment.

[0076] Furthermore, as an application of this embodiment, an image instruction unit 58 can be used, which is a method for arbitrarily controlling the image generated by the noise removal model 54 using a method other than a prompt string. The pose image or reference image input to the image instruction unit 58 may be used as input for learning and inference by the image classifier 56. Therefore, the image classifier 56 collects learning data from the entire image generation unit 33. Because the image generation unit 33 has such characteristics, the following description will not only explain the image classifier 56, but also LoRA and the latent diffusion model 52, which are existing technologies that constitute the image generation unit 33.

[0077] <lora> First, we explain LoRA, which freezes the weights of a pre-trained generative model and injects trainable layers (rank decomposition matrices) into each block, enabling tuning of the generative model with a learning cost of several tens of minutes.

[0078] Originally, LoRA was applied to the LLM Transformer network for natural language processing, but it is also known to function effectively in diffusion models. An intuitive example of LoRA is its ability to mimic existing characters using a diffusion model. For example, simply training LoRA with a few images of popular characters or drawings by popular designers allows it to generate images that closely mimic the patterns and designs of the images it learned. In other words, training a certain image with LoRA allows the diffusion model to learn the image's features. LoRA's feature extraction capabilities are extremely high. Considering the needs of animation production, LoRA's characteristics are expected to be extremely effective in enabling image generation AI to draw large numbers of characters that conform to design codes.

[0079] <Diffusion model> Next, we will explain the diffusion model. The diffusion model is a method for realizing image generation AI that has developed rapidly since 2020, and new papers are published almost every day, so it is difficult to explain the latest technology. Here, we will only explain the behavior of the basic diffusion model.

[0080] The diffusion model consists of two processes: a forward process that adds random noise (Gaussian noise) to the image data being trained, and a reverse process that creates image data by removing noise from the noise distribution. The forward process continues to add random noise (Gaussian noise) to the original image data, learning this addition process (the diffusion process) to obtain the Gaussian distribution of the original image. The reverse process is the de-diffusion process, which removes noise from the Gaussian distribution to create an image. In the diffusion model, the trained noise is trained in association with character information, and during inference, the de-diffusion process activated by this character information is executed. Therefore, because the diffusion model uses noise (random numbers) in both training and inference, which allows it to generate any image associated with characters, the images generated by the diffusion model are strongly influenced by the random number seed.

[0081] As mentioned above, diffusion models generate images internally using random numbers, making it difficult to generate appropriate variations of an input image in a single inference (generation). Specifically, existing diffusion models require several to several dozen trial generation attempts to generate a single image, and a human must then select the desired image from the numerous generated images. In particular, diffusion models have difficulty generating images with facial expressions, angles, poses, etc. that are not included in the diffusion model's training dataset or that are included in only a small number of training datasets. It is not uncommon for dozens of trials to be required before the desired image can be generated. While it is highly likely that the desired image will ultimately be reached, repeated trials like these reduce the productivity of animation production workflows using image generation AI.

[0082] The image generation AI implemented in the image recognition system 10 according to this embodiment is based on a diffusion model. Until now, there has been no example of identifying the output of an image generation AI based on strict criteria, such as whether it is suitable for a specific character in a specific animation. For the first time, the image generation unit 33 according to this embodiment can eliminate the trial-and-error process of image generation that occurs when using a diffusion model image generation AI.

[0083] <Image Classifier 56> Next, the image classifier 56, which is a module unique to this embodiment, will be described. The image classifier 56 can be implemented using any neural network, but as of 2023, it is preferable to implement it as a binary classification module using Vision Transformer (ViT). ViT adopts an approach in which the image to be trained is divided into N patches (P x P), each of which is linearly projected to embed the same dimensions as the latent vector handled by the Transformer, and the positional information of the image is assigned to the patch. This allows the Transformer to learn the image as a kind of sequential data.

[0084] The image classifier 56 uses supervised learning to identify whether the output result of the latent diffusion model 52 satisfies the design code of the target character. More specifically, the image classifier 56 represents the generated image of the latent diffusion model 52 as a d-dimensional ViT latent vector x∈Rd, and finds a target variable (output) y∈{0,1} that indicates whether the image is suitable (1) or unsuitable (0).

[0085] In this embodiment, the classification is performed with the possible values ​​of the explanatory variables limited to two classes, and therefore the technology of the image classifier 56 is positioned as binary classification. The image classifier 56 uses ViT to learn a function for calculating the objective variable from the latent vector from the following two types of training data. Note that the latent vector may include metadata (called tag information or caption) attached to the positive example image and ControlNet parameters (such as OpenPose vector information) used to generate the positive example image.

[0086] The image classifier 56 then learns the training base image data Am obtained when additional training using LoRA is performed as a positive example. The image classifier 56 also learns the results of the animator classifying the generated images Bn generated by the latent diffusion model 52 into positive examples and negative examples.

[0087] The image generation AI according to this embodiment is intended to be used in an environment where an animator can accurately judge the results of a generative model (e.g., generated images). For example, the animator will continuously provide positive and negative examples to the image classifier 56. This allows the image classifier 56 to learn, under the guidance of the animator, so that it can automatically distinguish the character the animator is aiming to generate.

[0088] Furthermore, as an application of this embodiment, a video image classification model such as 3D-CNN (Convolutional Neural Network), SlowFast, or TimeTransformer may be used to classify in-between videos.

[0089] <Example of processing the diffusion model> Here, an example of the processing performed by the diffusion model 50 shown in Fig. 4 will be described with reference to Figs. 5 to 7. The processing performed by the diffusion model 50 includes a learning processing and an inference processing. The learning processing includes a LoRA learning processing and a learning processing of the image classifier 56.

[0090] (LoRA learning process) First, the LoRA learning process will be explained. 5 is a flowchart showing an example of the LoRA learning process. The model that has undergone the LoRA learning process is used as an adapter for the diffusion model 50 as an additional learning model.

[0091] First, a training image A1 is input to the diffusion model 50 (S1). It is desirable to input multiple training images A1. Next, the training process of the training image A1 is repeated a set number of times in the diffusion model 50 (S2). The set number of training processes may be set by the user or may be a default value. When the training process is complete, a LoRA file is created (S3), and the LoRA training process ends. This LoRA file is read when the diffusion model 50 is executed.

[0092] (Image classifier learning process) Next, we will explain the learning process of the image classifier 56. The image classifier 56 uses the training image Am as a positive example and repeats learning based on the training image Am, the generated image Bn, and the result of the animator's judgment of whether the generated image Bn is suitable or unsuitable. FIG. 6 is a flowchart showing an example of the learning process of the image classifier 56.

[0093] First, the image classifier 56 receives first training data (S11). The first training data uses training image A1 trained in the LoRA training process as a positive example.

[0094] Next, the image classifier 56 receives second training data (S12). The second training data uses the images Bn output by the diffusion model 50 and the results of the animator's evaluation of each image Bn. Images that the animator evaluates as suitable are labeled with an OK label as a positive example. On the other hand, images that the animator evaluates as unsuitable are labeled with an NG label as a negative example.

[0095] Next, the image classifier 56 learns the first training data as a positive example (S13). After this learning, the image classifier 56 becomes able to classify, for example, images that have a similar atmosphere to the training image A1 as suitable. As described above, the image classifier 56 may not only learn the first training data as a positive example, but also learn the results that the animator has classified as a negative example.

[0096] Next, the image classifier 56 learns based on the second training data (S14). After this learning, the image classifier 56 can classify the image generated by the diffusion model 50 and automatically classify it as either suitable or unsuitable. When step S14 is completed, this process ends.

[0097] <Automatic inbetween generation> Next, we will explain the function of automatically generating inbetween frames for character motion, which is an application example of the image generation AI according to this embodiment.

[0098] An in-between image is an image that fills the gap between two original images, and drawing this in-between image is a major time and labor cost in animation production. An example in which the diffusion model 50 generates an in-between image and the image classifier 56 according to this embodiment classifies whether the in-between image is suitable or unsuitable will be described with reference to FIG.

[0099] FIG. 7 is a diagram showing an example of a learning image and an example of a classification result of whether an in-between image is suitable or unsuitable.

[0100] As described above, the LoRA learning process uses multiple training images. For example, assume that three training images A1 to A3 shown in the upper part of Fig. 7 are used in the LoRA learning process. Training images A1 to A3 are images in which the character's face is facing forward, left, and right, respectively.

[0101] In animation production, if 24 images per second are required, training image A1 is assigned to the first frame, training image A2 to the 12th frame, and training image A3 to the 24th frame. In the figure, each frame number is represented by "F." Furthermore, the first frame is called the start frame, and the 24th frame is called the end frame. The diffusion model 50 that reads the LoRA file can generate images that reflect the characteristics of training images A1 to A3.

[0102] 7, in addition to the training images A1 to A3, examples of in-between images B11 and B31 generated by the diffusion model 50 are shown. Both of the in-between images B11 and B31 have been classified as suitable images by the image classifier 56 and have been assigned OK labels.

[0103] In-between image B11 is placed between learning images A1 and A2 and is a natural image to use as an in-between image for the 6th frame when the character's face moves from left to front. In-between image B31 is placed between learning images A1 and A3 and is a natural image to use as an in-between image for the 18th frame when the character's face moves from front to right. Although not shown, it is assumed that in-between images have also been generated for frames 2 to 5, 7 to 11, 13 to 17, and 19 to 23.

[0104] Note that the in-between image B31 was not generated in the first image generation process by the diffusion model 50. For example, the diffusion model 50 generated multiple in-between images B21 to B23, as shown in the lower part of FIG. 7. In-between image B21, the character's mouth is positioned too low, and in-between image B22, the character's nose is too large. In-between image B23, one of the character's eyes is too large. For this reason, all of the in-between images B21 to B23 have been classified as inappropriate images by the image classifier 56 and have been labeled as NG.

[0105] Any in-between image that the image classifier 56 classifies as inappropriate is discarded. Then, an in-between image for the 18th frame is repeatedly generated until the image classifier 56 classifies it as suitable. As described above, if the image classifier 56 classifies it as suitable, it is adopted as in-between image B31.

[0106] FIG. 8 is a flowchart showing an example of image generation processing by the diffusion model 50 and classification processing of an in-between image by the image classifier 56.

[0107] First, the diffusion model 50 reads an image and instruction information (S21). The image read by the diffusion model 50 is, for example, the training image An shown in Fig. 4. The instruction information read by the diffusion model 50 is, for example, at least one of the text instruction information of the text instruction unit 57 and the image instruction information of the image instruction unit 58 shown in Fig. 4.

[0108] Next, the VAE encoder 51 of the diffusion model 50 performs encoding processing on the read image (S22), and outputs the encoded image to the latent diffusion model 52.

[0109] The latent diffusion model 52 performs a process of diffusing noise, which is random information, on the encoded image (S23). The image in which noise has been diffused by the process of step S23 is output to the noise removal model 54 as a noise image 53.

[0110] Next, the noise removal model 54 performs a noise removal process (S24) to remove noise from the noise image 53. Next, the VAE decoder 55 performs a decoding process on the image from which noise has been removed (S25). The image after the decoding process is output to the image classifier 56 as the generated image Bn shown in FIG.

[0111] Next, the image classifier 56 performs a classification process for the in-between image (S26). For example, the classification process in step S26 classifies whether the in-between image is an image generated according to the character's design code and whether it is an appropriate image for the set frame number. Then, the image classifier 56 classifies whether the generated image Bn is suitable as an in-between image for the preset frame number (S27).

[0112] Next, if the image classifier 56 determines that the generated image Bn is not suitable as an in-between image (NO in S27), it discards this generated image Bn (S28). Then, the image classifier 56 instructs the diffusion model 50 to regenerate an in-between image for the set frame number (S30). Therefore, the process returns to step S23, and an image is generated again. In this way, the image classifier 56 repeats the procedure of having the diffusion model 50 generate generated images Bn until all generated images Bn to interpolate the frame period by the diffusion model 50 are determined to be suitable.

[0113] On the other hand, if the image classifier 56 has identified the generated image Bn as being suitable as an in-between image (YES in S27), it determines whether or not the number of in-between images identified as being suitable is a specified number (S29). For example, if an in-between image for the set frame number has not been generated and the number of in-between images is not the specified number (NO in S29), the image classifier 56 instructs the diffusion model 50 to regenerate an in-between image for the corresponding frame number (S30). Therefore, the process returns to step S23, and an image is generated again.

[0114] On the other hand, if the number of in-between images that have been identified as suitable is equal to the specified number (YES in S29), the image classifier 56 ends this processing.

[0115] The greatest advantage of the image generation AI according to the present embodiment described above is that it can improve the productivity of the image generation AI simply by additional learning and adding a subsequent workflow, without changing the existing image generation AI. Specifically, it has the following two advantages compared to existing methods for improving image generation AI.

[0116] (compatibility) The image generation AI according to this embodiment is compatible with any existing image generation AI. In particular, since the image generation AI according to this embodiment does not rely on a specific image generation AI model, it can, for example, use different models depending on the title of the animation being produced, and seamlessly accommodate updates to the diffusion model. This is because the image classifier 56 according to this embodiment does not intervene in the image generation process itself, but instead focuses on automating the trial-and-error process essential to image generation AI, selecting only results that satisfy the design code of a specific character. Much of previous image generation AI research has yielded new applications by devising inputs to the latent diffusion model 52. Unlike conventional image generation AI, the image classifier 56 according to this embodiment enables "automated trial-and-error" by using the output of the image generation AI as the learning or inference target.

[0117] (Scalability) The image classifier 56 according to this embodiment can be easily incorporated into the workflow of an existing image generation AI. This is because the interface between the image generation AI and the image classifier 56 relies only on well-known data such as image data, tag information, and ControlNet parameters, and does not require any additional parameters. This allows the image classifier 56 according to this embodiment to be implemented by directly utilizing the workflow of image generation AIs that are being developed as open source software (OSS) around the world. Furthermore, the image classifier 56 according to this embodiment can be easily connected to other image generation AI modules.

[0118] Furthermore, the image recognition system 10 according to this embodiment can automatically select only suitable images from images generated by an image generation AI that has been fine-tuned specifically to generate specific characters. The image recognition system 10 according to this embodiment can automatically and mechanically select, from a vast number of trial results, only generated results that match the character design fine-tuned by LoRA and that match the content specified in the prompt.

[0119] With conventional image generation AI, excessive training with LoRA results in the AI ​​losing its ability to generate various conditions, while relaxing LoRA results in the AI ​​being unable to generate images strictly in accordance with the character's design code. In contrast, the image generation AI of this embodiment uses relatively relaxed LoRA training to allow the AI ​​to undergo a large amount of trial and error, allowing it to extract only those results that satisfy the design code. This makes it possible to entrust the image generation AI with tasks such as video inbetween frames (the process of drawing intermediate frames between original drawings) and the creation of character facial expression variations.

[0120] Furthermore, the image generation AI according to this embodiment can automatically generate the in-between images required for animation in just a few tens of seconds. Therefore, even if a single image requires five minutes of human supervision, an animator assisted by the image generation AI according to this embodiment can produce up to 100 images per day. In this way, the image generation AI according to this embodiment can be positioned as a fundamental technology for transforming animation production sites from labor-intensive environments to knowledge-intensive environments centered around AI.

[0121] Furthermore, by using the diffusion model 50 according to this embodiment, it becomes possible to automatically generate in-between images, which are essential for animation production, simply by subjecting a few character setting images to LoRA learning processing. For this reason, the image generation AI according to this embodiment has an extremely large impact on animation production.

[0122] Furthermore, the image generation AI of this embodiment can use existing diffusion models as they are, and can also automatically curate diffusion models fine-tuned using additive learning methods other than LoRA. Therefore, it can be used as the output layer of image generation AI used in animation studios in the medium to long term. Image generation AI technology is constantly evolving, and it is easy to update the model of this embodiment to keep up with technological advances.

[0123] Note that even if an image is judged to be suitable by the image classifier 56, the animator may judge it to be unsuitable. In this case, the diffusion model 50 regenerates the image of the frame judged to be unsuitable by the animator, and the image classifier 56 classifies the regenerated image. By repeating this process, the image intended by the animator can be obtained in a short time.

[0124] Furthermore, the pre-learning unit 32 and the image generation unit 33 configured in the image generation server 1 can also be configured in the information processing terminal 2. In this case, the information processing terminal 2 generates images offline. This eliminates the risk of pre-release character images being leaked to the outside or of character images being learned by a third party's image generation AI.

[0125] The present invention is not limited to the above-described embodiment, and it goes without saying that various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiment provides a detailed and specific description of the configuration of the image recognition system 10 in order to clearly explain the present invention, and is not necessarily limited to a system that includes all of the described configurations. Furthermore, it is also possible to add, delete, or replace part of the configuration of this embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0126] 1...image generation server, 2...information processing terminal, 10...image recognition system, 32...pre-learning unit, 33...image generation unit, 45...image output unit, 46...output unit, 50...diffusion model, 51...VAE encoder, 52...latent diffusion model, 53...noise image, 54...noise removal model, 55...VAE decoder, 56...image classifier, 57...text indication unit, 58...image indication unit< / lora>

Claims

1. A program for identifying a generated image generated by a diffusion model, a step in which the diffusion model, which uses an additional learning model that has learned learning images as an adapter, generates the generated image so as to interpolate a predetermined frame period based on instruction information, and the image classifier acquires the generated image; a step in which the image classifier classifies whether the generated image generated by the diffusion model is suitable or unsuitable; a step of discarding the generated image that the image classifier has identified as inappropriate, and causing the diffusion model to regenerate the generated image corresponding to the discarded frame; A program for a computer to run.

2. The image classifier uses the training images as positive examples and repeats learning based on the training images, the generated images, and a result of a user classifying the generated images as appropriate or inappropriate. The program according to claim 1.

3. The image classifier repeats the procedure of causing the diffusion model to generate the generated images until all the generated images generated by the diffusion model are suitable. The program according to claim 2.

4. The generated image is an image that represents at least one of the following, which changes over time in the frame period: rotation of an object, change in drawing angle, movement, deformation of the object, deletion of some elements of the object, addition of some elements to the object, change in facial expression of the object, and change in color of the object. The program according to claim 3.

5. The diffusion model encodes the training image in pixel space into a latent space, and then learns a diffusion process in which a latent diffusion model adds random information to the encoded training image. The data from which the random information has been removed by a de-diffusion process based on the diffusion process is decoded into pixel space to generate the generated image. The program according to claim 4.

6. The instruction information is text instruction information in a text format, a text encoder encoding the input text indicating information into a vector format; The diffusion model removes the random information from the training images based on the encoded text indication information; and wherein the image classifier classifies the generated image based on the encoded text indication. The program according to claim 5.

7. The instruction information is image instruction information in an image format, a step in which the diffusion model removes random information added by the latent diffusion model based on the input image indication information; and a step of the image classifier classifying the generated image based on the image indication information. The program according to any one of claims 2 to 6.

8. The image classifier classifies the generated image as suitable when the generated image is generated in accordance with the shape information of the object. The program according to claim 4.

9. An image identification method for identifying a generated image generated by a diffusion model, comprising: a step in which the diffusion model, which uses an additional learning model that has learned learning images as an adapter, generates the generated image so as to interpolate a predetermined frame period based on instruction information, and the image classifier acquires the generated image; A step in which the image classifier classifies whether the generated image generated by the diffusion model is suitable or unsuitable; The image classifier discards the generated image that is identified as inappropriate, and causes the diffusion model to generate the generated image corresponding to the discarded frame again. Image identification methods.

10. An image classification system for classifying generated images generated by a diffusion model, a diffusion model that uses an additional learning model that has learned learning images as an adapter and generates the generated image so as to interpolate a predetermined frame period based on instruction information; an image classifier that takes in the generated images, identifies whether the generated images generated by the diffusion model are suitable or unsuitable, discards the generated images that are identified as unsuitable, and causes the diffusion model to generate the generated images corresponding to the discarded frames again. Image identification system.

Images