System for generating image on basis of input language or sample image

Abstract

[Problem] To provide a system in which digital processing for paying a reward to a right holder, including an original creator, is performed even when it is difficult to determine whether or not unique characteristics of an image used as training data are included in an image to be generated. [Solution] A system for generating an image on the basis of an input language or a sample image, the system comprising: a style identification unit that identifies, on the basis of the input language or the sample image, a generation style, which is the style of an image to be generated; a reward determination unit that determines whether or not the use of the generation style will require a reward; a reward explanation unit that, if the reward determination unit determines that the use of the generation style will require a reward, outputs a chargeable style offer, which is an offer indicating that the use of the generation style will require a reward; and an image generation unit that, if acceptance of the chargeable style offer is input, generates an image on the basis of the input language or the sample image and on the basis of the generation style, and obtains the generated image.

Description

System for generating images based on input language or sample images - Patent Application 20070122997

[0001] The present invention relates to a system for generating images based on input language or sample images.

[0002] Japanese Patent Publication No. 7169027 describes a character image generation device and a learning model generation device. The device includes an image generation unit, a character variable generation unit, and a posture variable generation unit. The image generation unit is an element for generating an image from multiple input variables using an image generation model trained by machine learning using training image data. The character variable generation unit is an element for generating multiple character variables corresponding to the characteristics of each character type. The posture variable generation unit is an element for generating multiple posture variables corresponding to the postures of the character. The character image generation device inputs the multiple character variables generated by the character variable generation unit and the multiple posture variables generated by the posture variable generation unit into the image generation model as the multiple variables, thereby generating a character image of a character of a type specified by the multiple character variables and having a posture specified by the multiple posture variables.

[0003] Patent No. 7169027

[0004] Systems that generate character images based on input information like this are already known. However, if the generated image contains the characteristics of the image used as training data, copyright law issues arise. Even if it is difficult to determine whether the generated image contains the characteristics of the image used as training data, it is desirable to perform digital processing that ensures that rights holders, including the original author, are paid compensation.

[0005] This invention is based on the finding that it is possible to provide a digital system that appropriately pays remuneration to authors and original authors by determining whether or not remuneration will be generated based on input information.

[0006] The first invention relates to a system 1 for generating an image. This system 1 generates an image based on an input language or a sample image. This system 1 has a style identification unit 3, a remuneration determination unit 5, a remuneration explanation unit 7, and an image generation unit 9. The style identification unit 3 is an element for identifying a generated style based on the input language or the sample image. The generated style is the style of an image generated by this system. An example of a style is the style of a certain author. The remuneration determination unit 5 is an element for determining whether a remuneration will be generated when the generated style is used. The remuneration explanation unit 7 is an element for outputting a paid style request when the remuneration determination unit 5 determines that a remuneration will be generated. The paid style request is a request regarding the generation of a remuneration when the used generated style is used. The image generation unit 9 is an element for generating an image based on the generated style based on the input language or the sample image, and obtaining the generated image, when acceptance is input in response to the paid style request.

[0007] The style fee offer may include one or more of the following information: whether the author of the generated style is alive or dead; the remaining copyright term; and whether the generated style is in the public domain.

[0008] The image generation unit 9 may include a first trained model 11 and a second trained model 13. The first trained model 11 is a model for generating an image (first-stage virtual image) based on an input language or a sample image. It is not necessary to actually create an image. The second trained model 13 is a model for additional learning and fine-tuning regarding the generated image style.

[0009] The image generating unit 9 preferably includes a digital watermark or file-embedded copyright information related to the specified style in the generated image. The generated image may include both information about the author of the generated style and information about the author of the generated image.

[0010] An example of the generated image is a cartoon image. The cartoon image preferably includes a frame frame layer related to the frame frames of the cartoon image, a speech bubble layer related to the speech bubbles within the frame frames, a character layer related to the characters within the frame frames, and a background layer related to the background within the frame frames. The image generation unit 9 preferably controls the cartoon image so that when the frame frames change depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frames.

[0011] The second invention relates to a program for causing a computer to function as the above-mentioned system, a program for causing a computer to execute an image generation method, and a non-transitory information recording medium readable by a computer that stores such a program. The image generation process using a computer includes a style identification process (S101), a remuneration determination process (S102), a remuneration explanation process (S103), and an image generation process (S104). The style identification process (S101) is a process for identifying a generated style, which is the style of the image to be generated, based on an input language or a sample image. The remuneration determination process (S102) is a process for determining whether a remuneration will be generated if the generated style is used. The remuneration explanation process (S103) is a process for outputting a style fee offer, which is an offer regarding the generation of a remuneration for using the generated style, if it is determined in the remuneration determination process (S102) that a remuneration will be generated. The image generating step (S104) is a step for generating an image based on the generated style based on the input language or the sample image, when consent is input to the style fee request, to obtain the generated image.

[0012] According to this invention, it is possible to provide a digital system that determines whether or not a fee will be paid based on input information, thereby ensuring that the author or original creator is appropriately paid a fee.

[0013] FIG. 1 is a block diagram illustrating an image generation system. FIG. 2 is a flowchart illustrating an image generation method. FIG. 3 is an example of an input screen. FIG. 4 is a conceptual diagram showing an example of each layer of a generated manga. FIG. 5 is a diagram showing the manga of FIG. 4 converted into a manga for a foreign market.

[0014] The following describes embodiments of the present invention with reference to the drawings. The present invention is not limited to the embodiments described below, but also includes appropriate modifications of the embodiments that are obvious to those skilled in the art.

[0015] FIG. 1 is a block diagram illustrating a system for generating an image (image generation system). As shown in FIG. 1, the image generation system 1 includes a style identification unit 3, a reward determination unit 5, a reward explanation unit 7, and an image generation unit 9. These elements are virtual, and each element may be implemented by a single processor. This system 1 may be implemented by a computer, and the processor may store programs and instructions in a memory unit and implement each process and function based on the instructions. Note that the system 1 may be a server that provides information to a user's client, or applications and programs may be installed on the user's client.

[0016] 2 is a flowchart illustrating the image generation method. As shown in FIG. 2, this method includes a style identification step (S101), a remuneration determination step (S102), a remuneration explanation step (S103), and an image generation step (S104). These steps are virtual, and some steps may be performed simultaneously.

[0017] The computer has an input unit, an output unit, a control unit, a calculation unit, and a memory unit, and each element is connected by a bus or the like to enable the exchange of information. Various information input from the input unit may be converted into digital information such as binary data, and the digital information may be processed on the computer. The memory unit may store a program or various information. When predetermined information is input from the input unit, the control unit reads the program stored in the memory unit. The control unit then reads the information stored in the memory unit as appropriate and transmits it to the calculation unit. The control unit also transmits the input information as appropriate to the calculation unit. The calculation unit performs calculation processing using the received various information based on instructions from the program and stores the results in the memory unit. The control unit reads the calculation results stored in the memory unit and outputs them from the output unit. In this manner, various processes and steps are performed. The various units and means execute these various processes. The computer may have a processor, and the processor may realize various functions and steps. The computer may be standalone. Some of the functions of the computer may be distributed between a server and a terminal. In this case, it is preferable that the server and the terminal are able to exchange information via a network such as the Internet or an intranet. The computer may include a processor and a memory coupled to the processor. The memory may store instructions that, when executed by the processor, cause the computer to perform various processes or function as various elements. The computer may be provided with various training data to construct a learning model and perform various calculations through machine learning. In this case, the computer may perform various analyses using a learning model created by machine learning or deep learning of AI (artificial intelligence).

[0018] Style Identification Step (S101) The style identification step (S101) is a step for identifying a generation style, which is the style of an image to be generated, based on an input language or a sample image. This step may be performed by the style identification unit 3 of the system 1. The style identification unit 3 is an element for identifying a generation style based on an input language or a sample image. The generation style is the style of an image generated by this system. An example of a style is the style of a certain author.

[0019] FIG. 3 is an example of an input screen. As shown in FIG. 3, the system 1 displays a field for inputting information for creating an image. The user enters necessary information in this field, drags and drops an image file, or inputs information for identifying the image (e.g., a URL). Digital data related to the input language, digital data related to the input image, and the like are then input to the system 1. An example of the input unit is a chat input unit. An example of the system 1 is for generating an image such as a manga based on chat (and image file input). The art style may be selectable from a pull tab or a selection unit, and linguistic information for specifying the art style may be input to the system 1 by the user's selection. The input language may include instructions related to the generated image in addition to the above-mentioned art style. Examples of instructions related to the generated image may include one or more of the following: facial expression, direction, background, position of the manga frame, type of effect, position of the effect, size of the effect, type of speech bubble, position of the speech bubble, size of the speech bubble, dialogue, font of the dialogue, color of the content, size of the content, position of the content, and context of the content.

[0020] Identifying an Art Style Based on Linguistic Information Examples of art styles include purpose, drawing style, type of illustration, manga name, character name, author name, and user original. When a user inputs linguistic information related to the artwork in an input field, the art style is identified, for example, as follows. The language analysis unit 21 of the system 1 analyzes the linguistic information, which is, for example, digital information. Then, the language analysis unit 21 references the art style dictionary 23 and extracts the art style information from the linguistic information. For example, the art style dictionary stores keywords indicating art styles and related words related to each keyword. The language analysis unit 21 then compares the input linguistic information with whether the keywords or related words indicating the art style are included. If the keywords or related words are included in the linguistic information, the language analysis unit 21 identifies the art style based on the keywords or related words. The identified generated art style may be stored in a storage unit as appropriate. The keywords and related words may be a set of multiple words. The linguistic information may be registered in the system 1 in advance and input to the system 1 by selecting it.

[0021] Examples of keywords indicating an art style are "patent drawing," "design image," "ink painting," "impressionism," "Demon Slayer: Kimetsu no Yaiba (registered trademark)," "Kingdom (registered trademark)," "Hatsune Miku (registered trademark)," "author A," "manga title B," "character C," "Chagall," and "Hello Kitty (registered trademark)." Other examples of keywords indicating an art style are "Dragon Ball (registered trademark)" and "Son Goku." An example of author A is "Walt Disney (registered trademark)" (Walt Disney (registered trademark) style). Examples of keywords indicating an art style are "caricature," "realistic," "deformed," "different colors," and "coloring book style" (uncolored). Once these art styles are input, the generated image should reflect that art style. To achieve this, machine learning can be performed to build a (second) trained model for each art style.

[0022] Examples of related words related to a keyword are "drawings used in a patent," "for a patent" and "drawings," and "patent application" and "drawings." Other examples of related words related to a keyword are "Hello Kitty (registered trademark)," "Kitty-chan," "Kitty-chan," and "Kitee-chan." Related words may include words that are reminiscent of the keyword or incorrect entries of the keyword.

[0023] For example, suppose a user inputs, "Please create patent drawings based on the handwritten drawings I will now input." Digital information related to "Please create patent drawings based on the handwritten drawings I will now input" is then input to the system 1 as linguistic information (input method). The linguistic information is appropriately stored in the memory. The style identification unit 3 (the language analysis unit 21) of the system 1 references the style dictionary 23, reads keywords and related words from the style dictionary 23, and compares them with the linguistic information. As a result, the style identification unit 3 (the language analysis unit 21) of the system 1 determines that the linguistic information contains "patent" and "drawings," which are related words to "patent drawings." As a result, the style identification unit 3 (the language analysis unit 21) of the system 1 selects "patent drawings" as the creation style related to the linguistic information. Information related to the selected creation style, "patent drawings," is appropriately stored in the memory.

[0024] Identifying Art Style Based on Input Images, etc. A case will be described in which a user inputs several images into system 1 to identify an art style. For example, systems that identify an art style based on an input image and build a learning model, such as the image generation unit and character variable generation unit in Japanese Patent No. 7169027, are already known. Therefore, this invention can also appropriately employ known machine learning and learning model building methods. For example, a user uses a terminal (client) to drag and drop an image file or input information for identifying an image (e.g., a URL). Digital information related to the image file or information for identifying the image is then input to system 1. System 1 may obtain digital information related to the image file based on the information for identifying the image.

[0025] The system 1 may store several reference images and output them for display on the user's device, and when the user selects a reference image, the reference image or information identifying the reference image may be input to the system 1.

[0026] The art style identification unit 3 reads whether or not the image file contains author-related information. If the image file contains author-related information, the art style identification unit 3 stores the author-related information in the storage unit. The art style identification unit 3 references the art style dictionary 23 and compares the author-related information with keywords and related words. If the author-related information contains any of the keywords and related words, information regarding the generated art style may be obtained based on the author-related information. The author-related information may be embedded in the image or may be information added as additional information separately from the image. The art style dictionary 23 may store, in association with keywords, one or more of the following information: information regarding the life or death of the author (copyright holder) of the art style, the author's date of death, the remaining copyright term, and information regarding whether the art style is in the public domain (freely available).

[0027] If the image file does not contain author-related information, the art style identification unit 3 performs an operation of identifying the art style based on the image file. The art style identification unit 3 may use the input image to crawl a website to identify web pages containing similar images and identify the art style. Alternatively, the input image and art style may be input as training data to build a learning model for identifying the art style, and the image may be input into the learned learning model to identify the art style. Image searches on websites are already known, for example, through Google's (registered trademark) image input site. Therefore, this invention can also utilize known similar image search systems as appropriate. The created art style obtained in this manner may be stored in a storage unit as appropriate. Furthermore, learning may be performed from existing illustrations, and tags representing the images may be generated and added to the generated image. For example, a known algorithm such as DeepDanboru may be used to create identification information such as tags for the generated image, and a prompt may be created using the tags. Furthermore, identification information such as tags for the generated image may be created and added to the generated image using a model such as VLM that returns the similarity of unknown (class-unspecified) tags in natural language.

[0028] The user also inputs multiple images of a manga character X that he or she has created. The multiple images of the character X are then input to the system 1. The art style identification unit 3 uses the images of the character X as teacher data (training data) to generate a learning model (second learning model). The system 1 can obtain a trained model with the character X as its art style. From now on, the system 1 will be able to output images related to the character X simply by inputting a rough image (for example, an image that identifies posture information) into the additional learning model. The system 1 will be able to select this learning model, and when this learning model is selected, information for selecting the learning model, for example, as linguistic information, will be input to the system 1.

[0029] Reward Determination Step (S102) The reward determination step (S102) is a step for determining whether a reward will be generated when the generated style is used. The reward determination unit 5 of the system 1 may execute this step. The reward determination unit 5 is an element for determining whether a reward will be generated when the generated style is used. This step is for illustrative purposes and may be performed simultaneously with the style identification step (S101). The system 1 may include a style utilization database 25. The style utilization DB 25 may store style utilization information, such as whether or not a reward will be generated, the details of the reward, and the conditions for using the style, in association with the style. The style dictionary 23 and the style utilization DB 25 may be separate storage units, or a single database may exist, with the style dictionary 23 and the style utilization DB 25 being considered as part of that database. Furthermore, if author-related information is included in the image file, the reward determination unit 5 may read the included author-related information and determine whether a reward will be generated when the generated style is used.

[0030] Remuneration refers to the conditions (obligations) for using an art style. Examples of remuneration include one or more of the following: copyright royalties, royalties for the use of derivative works, non-exercise of moral rights, trademark royalties, design royalties, know-how royalties, site usage fees, original art usage fees, tribute fees, donations, cash, cryptocurrency, points, and trading stamps. Other examples of remuneration include requests to clearly state the author's name, to add the author's name to the generated image, and to agree to count the use of the art style when generating an image. Other examples of remuneration include soft currency, hard currency, points for subscriptions, rewards for completing surveys, and incentives for actions such as "liking" others. These may be used as remuneration to enable the generation of generated images. Copyrights for art styles do not necessarily arise. Even if copyrights arise, it may be difficult to collect copyright royalties due to the complexity of litigation. However, by using this system 1, it becomes possible to give some kind of "reward" to those who provide their own style of painting. Providing such a digital system will provide a strong motivation for creating and disseminating creative images.

[0031] Trademark Infringement Checker The reward determination unit 5 may check whether using a generated image may constitute a trademark infringement. In this example, for example, the system checks whether identical or similar marks exist in a prior trademark database based on the image style. The image style may be linguistic information or an image. The presence of similar marks based on language is implemented, for example, through a similar language search on the Japan Patent Office's former digital library, JPLATPAT. Therefore, the reward determination unit 5 may use such a website, or may independently build a database and refer to a similar language dictionary to select prior registered trademarks with marks similar to the input language. The system 1 then stores in advance the identification information (e.g., similar group codes) of designated goods and services likely to be related to the image. If a prior trademark contains this identification information, the system 1 may store it in the storage unit as a potentially related prior trademark. For example, 26A01 (related to manga) and 42P01 (design ideas) are stored in the memory, and if there are prior registered trademarks with similar marks, the system checks whether their similar group codes are included. If the prior registered trademarks are stored in association with those similar group codes, it can be determined that creating a design idea, for example, could potentially infringe on the prior registered trademark, and the design idea can be stored in the memory. Since trademark image search sites are already known, the system 1 can use those sites to obtain information on prior registered trademarks that may be similar when using an input image or style.

[0032] Copyright Information Checker The remuneration determination unit 5 may have a function for checking copyright information. For example, in association with a keyword, one or more of the following may be stored: the remaining copyright term (particularly the remaining copyright term in the user's country of residence), whether the work is copyright-free, whether it is open source, whether it is public domain, and whether the author is alive or dead, and the need for remuneration may be determined based on this information. Furthermore, the user may be prompted to input the degree of reliance (e.g., a numerical value). If the reliance is high (exceeding a threshold), the name of the author on whom the work was relied may be displayed, and copyright information may be read from the storage unit and output. For example, if the style (keyword) is that of an author who died a considerable time ago (more than a threshold), information stored in association with the author, such as the author's death information and whether the work is in the public domain, may be read from the storage unit and output. Furthermore, if the author of a certain style (keyword) is deceased but within the threshold (the copyright expiration date), the compensation determination unit 5 may read the author's date of death from the storage unit, have the calculation unit perform a calculation to determine the copyright expiration date, and output either the author's date of death or the remaining copyright period (countdown), or both. Furthermore, a user may add a statement to a generated image waiving their own copyright. In this case, another user may freely create a new generated image based on that generated image. However, if that user generates an image based on a style requiring compensation, the usage conditions for that style are attached to the generated image, and other users will also be subject to those attached usage conditions.

[0033] Cases where no fee is paid: For example, the fee determination unit 5 reads out "patent drawing" from the storage unit as style information. Alternatively, the fee determination unit 5 receives "patent drawing" from the style identification unit 3 as style information. In this case, information indicating that no fee is required is stored in the style utilization DB 25 (or the style dictionary 23) in association with "patent drawing." The fee determination unit 5 reads out the information indicating that no fee is required, stores it in the storage unit as appropriate, and outputs it to the image generation unit 9. Upon receiving the information indicating that no fee is required, the image generation unit 9 executes the image generation process. However, even when no fee is required, the system 1 may output information to the user's terminal for displaying rights management information, such as information about the author of the style, on the display unit of the user's terminal.

[0034] When a reward is generated For example, the reward determination unit 5 reads out "character C" as art style information from the storage unit. The art style usage DB 25 stores usage information for character C in association with "character C." The usage information for character C includes information regarding the generation of a reward when an image is generated using the system 1 using the art style of character C. This may be, for example, if a flag regarding the generation of a reward is stored in association with "character C." The reward determination unit 5 then reads out the flag regarding the generation of a reward, and may store the information regarding the generation of a reward in the storage unit as appropriate, or may output it to the reward explanation unit 7.

[0035] Remuneration Explanation Step (S103) The remuneration explanation step (S103) is a step for outputting a style fee offer, which is an offer regarding the generation of a reward when the reward determination step (S102) determines that a reward will be generated. The remuneration explanation unit 7 may execute this step. The remuneration explanation unit 7 is an element for outputting a style fee offer when the reward determination unit 5 determines that a reward will be generated. The style fee offer is an offer regarding the generation of a reward when the generated style is used. For example, the remuneration explanation unit 7 outputs the style fee offer to the user's terminal. Then, the display unit (e.g., monitor) of the user's terminal displays the style fee offer. In this way, the user can understand that the use of the style will generate a reward. The style fee offer simply indicates that a reward will be generated, and the conditions may be viewable via a link or the like. In addition to information about the payment of fees, the request for paid art styles may also include one or more of the following information: whether the author of the generated art style is alive or dead, the remaining copyright term, and whether the generated art style is in the public domain.

[0036] When the style fee offer is displayed on the display unit of the user's terminal and the user inputs an input to decline the offer, information to the effect that the offer has been declined is output from the user's terminal to the system 1. When information to the effect that the offer has been declined is input to the system 1, the system 1 may perform processing to start over from the style identification step (S101).

[0037] Image Generation Process (S104) The image generation process (S104) is a process for generating an image based on an original style based on the input language or sample image, when acceptance is input in response to the request for paid art style, to obtain the generated image. Of course, even if the system 1 determines in the remuneration determination process (S102) that no remuneration is required, the system 1 generates an image based on an original style based on the input language or sample image, to obtain the generated image. This process may be performed by the image generation unit 9. The image generation unit 9 is an element for generating an image based on an original style based on the input language or sample image, when acceptance is input in response to the request for paid art style, to obtain the generated image. The image generation unit 9 may generate an image from multiple input variables using an image generation model trained by machine learning using training image data. The image generation model generated by machine learning may be generated using a paid service via an API, such as "DALL-E," or may be a commercially available and openly released model, such as "Stable Diffusion XL." The image generation model generated by machine learning may be an application of StyleGAN, or may appropriately utilize publicly available AI such as chatGPT (registered trademark). For details of the image generation process, please refer to, for example, the book "Admiring Your LoRA."

[0038] The image generation unit 9 may include a first trained model 11 and a second trained model 13. That is, the image generation unit 9 may have an additional training unit. The first trained model 11 is a model for generating an image (first-stage virtual image) based on an input language or a sample image. However, if a second trained model exists, it is not necessary to actually create an image using only the first trained model 11. The first trained model can be said to be a model for generating an image by machine learning based on either or both of the input language and the sample image.

[0039] The second trained model 13 is a model for performing either or both of additional training and fine tuning regarding the generated art style. The system 1 may have a plurality of second trained models 13 corresponding to the art style. In this case, by using the second trained models 13 corresponding to the art style, it becomes possible to quickly create an image. An example of the second trained model 13 is an additional trained model for character X.

[0040] The image generation unit 9 preferably includes a digital watermark or file-embedded rights information related to the identified style in the generated image. That is, the image generation unit 9 preferably adds rights management information to the generated image. For example, the rights management information (fingerprint information) may be visible when the generated image is enlarged or viewed through a polarizing filter. The rights management information may also be added to the generated image in a form that cannot be deleted. The rights management information may be a creator identification tag such as "Generated by..." or a tag for identifying the author of the original image such as "Inspired by...". In this way, the rights management information may be information for paying a fee to the source of the style reference. The rights management information may also be information for identifying a fee. The information for identifying a fee may be a URL or information for identifying a storage area in the storage unit, or may be stored in association with the style in either or both of the style dictionary 23 and the style usage DB 25. The system 1 stores the image generated by the image generation unit 9 (generated image) in a storage unit as one of the reference images. When another user references the generated image and uses its style, it is preferable that a fee be paid to the rights holder of the style used when the system 1 generated the generated image. Another example of rights management information may include the names of licensor A and licensee B. Such rights management information may clearly indicate "permission" in the form of "Original by A Inspired by B" or "This generated image B is authorized by A original copyright right holder, detail is written in http: / / ...". This allows B to use the service in a socially and legally safe manner, and also enables B to clearly state the rights of the generated images (for example, whether they are recursively learnable or have a volatile license), making it flexible to withstand future changes in law and changes in user demand.

[0041] An example of the generated image is a cartoon image. The cartoon image preferably includes a frame frame layer related to the frame frames of the cartoon image, a speech bubble layer related to the speech bubbles within the frame frames, a character layer related to the characters within the frame frames, and a background layer related to the background within the frame frames. The image generation unit 9 preferably controls the cartoon image so that when the frame frames change depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frames.

[0042] Program The above system can be implemented by installing a program in the storage unit of a computer or server. In other words, this specification also provides a program for implementing a computer as the above system, a program for causing a computer to execute the above procedures, and a recording medium storing such a program. A computer or server with this program installed can execute the above processes and store user-specific information. An example of a recording medium is a non-transitory recording medium that can be read by a computer. Examples of non-transitory recording media include a CD-ROM, a DVD, a USB memory, and a hard disk.

[0043] A chat tool that generates new UGC images by allowing users to select a reference image and style, and then provide detailed instructions such as facial expressions and acting as prompts, while retaining the characteristics of the image compressed using the latent space.

[0044] This system allows a chat input section and a sample image input section to be displayed on the user's terminal. In addition, in a reference image display mode for displaying a reference image, the system allows the reference image to be displayed on the user's terminal. The user can then input information about the style by referring to the reference image. In addition, in a style selection mode, the system presents examples of styles to the user, allowing the user to specify the style of the generated image by selecting one.

[0045] A user wants to create a manga and launches a chat tool app installed on the device, which then allows the device to create manga, images, and videos via chat.

[0046] A user may input "I want to draw a manga" into the chat section. The system 1 then analyzes the input language and enters the manga creation mode. When the user selects the manga mode, the selection information is input into the system, and the system 1 may enter the manga creation mode.

[0047] A user inputs a concept for a manga into the chat section. For example, the user verbally inputs an outline of the manga. System 1 then analyzes the verbal information and retrieves example frame borders from the storage unit. System 1 then outputs the example frame borders to the user's device. The display unit of the user's device displays the example frame borders. The user can adjust the borders that make up the frame borders as needed by inputting verbal input into the chat section. In this way, a draft of the frame border layers is created and stored in the storage unit. By specifying each frame that makes up the frame border, the user can generate an image to be created in the specified frame. Furthermore, it is preferable that the mask can blend the influence of the calculation results of other layers (layers) using a value between 0 and 1 (a decimal). The decimal indicates the blending ratio. Masks may be used like frame borders, free-form shapes like speech bubbles, text on speech bubbles, overlapping characters, foreground and background, effect lines, compositing processes like manga symbols, and shading expressions like screen tones, or any combination of two or more of these. Furthermore, since the mask itself can be treated as an image with a shade range of 0 to 1, the creation of this mask itself can be an ingenious example of UGC (user-generated content), and by displaying it in a gallery, it can become an asset that can be used by other users.

[0048] 4A and 4B are conceptual diagrams showing examples of layers of a generated cartoon. As shown in Fig. 4A, the frame border layer specifies a frame border.

[0049] A user may input language containing "character C" into the chat section, select an image of "character C," or select a second learning model related to "character C." Alternatively, a user may input multiple images into the system 1, and the system 1 may determine that the images are related to "character C." The system 1 then determines that the style of the image to be generated is related to "character C."

[0050] The system 1 reads from the storage unit the rights management information stored in association with the character C. Because the system 1 stores the rights management information in this way, even if it is uncertain whether a copyright will be created, it is possible to generate a fee by using the character C in the system 1. The system 1 may read the rights management information stored in association with the referenced character C. The system 1 then determines whether or not using the character C (or the trained model related to the character C) requires a fee.

[0051] When the system 1 determines that using the character C (or the trained model for the character C) requires a reward, the system 1 outputs information about the reward and the usage conditions to the user's terminal. The display unit of the user's terminal displays the reward and the usage conditions.

[0052] If the user agrees to the rewards and terms of use, the user inputs the consent into the user's terminal. The user's terminal then outputs the consent information to the system 1. The system 1 receives the consent information and inputs it into the system 1.

[0053] Once consent is entered into the system 1, an image based on the art style can be generated. For example, a user may enter "Please create an image of character C holding a weapon and preparing for battle" into the chat input unit. The terms of use for character C are consented to as described above. The language analysis unit analyzes the terms "holding a weapon" and "prepared for battle," creates a model using the first learning model, and creates an image of character C using the second learning model, resulting in the creation of an image of character C holding a weapon and preparing for battle. Figure 4(b) shows an example of a character image in a specified frame. In this example, the character is accompanied by an indication of the author and the art style used, and rights management information can be referenced in relation to this. In this example, text is displayed. Alternatively, the image may contain code information, such as a QR code (registered trademark), in a form that is not visible to the user. The rights management information can be accessed by reading the code information using a reading device such as a camera.

[0054] Furthermore, for example, a background image may be specified in the chat input section. Alternatively, a user may input, "I'm going to input a background image file, so please make it a cartoon-style background," and then drag and drop the background image file into the chat input section. The background of the specified frame frame will then be based on the input image. The created background image may be stored in the background layer of the specified frame frame. In this case, the background image may be read in association with the frame frame specification information. FIG. 4(d) is a diagram showing an example of a background image for a specified frame.

[0055] Furthermore, the user may input lines of character C into the chat input unit. When lines of character C are input into the system 1, the system 1 performs linguistic analysis of the lines and creates example speech bubbles based on the learning model. The created speech bubbles and lines are output to the user's terminal. The user's terminal receives the information and displays the speech bubbles and lines on the display unit. In response, the user can adjust the speech bubbles and lines using the chat input unit. Examples of such input are "Make the speech bubble a little smaller" and "Emphasis on... in the lines." The system performs linguistic analysis and adjusts the speech bubbles and lines. Speech bubbles may be stored in a speech bubble layer and lines may be stored in a dialogue layer, associated with frame frames. FIG. 4( c) is a diagram showing an example of a speech bubble layer. Note that the speech bubble layer and the dialogue layer may be separate.

[0056] By repeating this process for other frame frames, a manga can be created. In this case, the image generation unit 9 can generate manga pages by overlapping each layer. In this case, the lower layer may delete the portion below the content of the upper layer. Rights management information may be stored for each frame frame in the created manga image. Rights management information may also be stored for the entire manga.

[0057] This system 1 may have a mode conversion function that allows a created manga to be adapted to suit the intended use, such as adjusting it for a foreign country or converting it to vertical reading for tablets.

[0058] The user selects the English mode. English mode instruction information is then input to the system 1. Based on program instructions, the system 1 refers to a translation learning model (translation engine) and translates the dialogue into English. The system 1 also changes the frame frames in the frame frame layer from right to left and left to right, creating updated frame frames.

[0059] The system 1 reads the content of each layer, such as the speech bubble layer related to the speech bubbles within the frame frame, the character layer related to the characters within the frame frame, and the background layer related to the background within the frame frame, stored in association with each frame frame, and regenerates images to match the updated frame frame. At this time, the dialogue is converted to the English-translated dialogue described above. In this manner, the created manga (image) can be easily converted into a manga for a foreign market. The generated manga for a foreign market may be stored in an appropriate storage unit. The generated manga for a foreign market may also be output from the system 1 to a user's terminal, and the manga may be displayed on the display unit of the user's terminal. The user may then adjust the manga for a foreign market by inputting commands into the chat input unit. In other words, the image generation unit 9 preferably controls the manga image so that when the frame frame changes depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frame. Figure 5 shows the manga of Figure 4 converted into a manga for a foreign market. In this way, the frame layout is automatically converted, and the character positions and language are also automatically converted accordingly. Furthermore, customization for the conversion destination can be performed by referring to a conversion dictionary. In this example, the background color of the speech bubble changes based on the information in the conversion dictionary.

[0060] The present invention can be used in the information industry because it can easily generate images, cartoons, etc.

[0061] REFERENCE SIGNS LIST 1 Image generation system 3 Art style identification unit 5 Reward determination unit 7 Reward explanation unit 9 Image generation unit 11 First trained model 13 Second trained model 21 Language analysis unit 23 Art style dictionary 25 Art style usage database

Claims

1. A system for generating an image based on an input language or a sample image, comprising: a painting style specifying unit that specifies a generation painting style, which is a painting style of the image to be generated, based on the input language or the sample image; a reward determination unit that determines whether a reward will be generated when using the generation painting style; a reward explanation unit that outputs a painting style paid application, which is an application regarding that a reward will be generated when using the generation painting style, when the reward determination unit determines that a reward will be generated; and an image generation unit that generates an image based on the generation painting style based on the input language or the sample image and obtains a generated image when a commitment is input for the painting style paid application.

2. The system according to claim 1, wherein the painting style paid application includes any one or more types of information regarding the life and death of the author of the generation painting style, the remaining period of the copyright, and whether the generation painting style is in the public domain.

3. The system according to claim 1, wherein the image generation unit includes: a first pre-trained model for generating an image based on the input language or the sample image; and a second pre-trained model for performing additional learning and fine-tuning regarding the generation painting style.

4. The system according to claim 1, wherein the painting style is the painting style of a certain author.

5. The system according to claim 1, wherein the generated image has an electronic watermark regarding the specified painting style or includes file-embedded type right information.

6. The system according to claim 1, wherein the generated image is added with both information regarding the author of the generation painting style and information regarding the author of the generated image.

7. The system according to claim 1, wherein the generated image is a comic image.

8. The system according to claim 7, wherein the comic image includes: a frame layer regarding the frame of the comic image; a speech bubble layer regarding the speech bubbles within the frame; a character layer regarding the characters within the frame; and a background layer regarding the background within the frame.

9. The system according to claim 8, wherein the image generation unit controls the comic image so that when the frame changes according to the input language, at least one of the background, the character, and the speech bubble changes along with the change of the frame.

10. A program for causing a computer to execute: a painting style specifying step of specifying a generation painting style, which is a painting style of an image to be generated, based on an input language or a sample image; a reward determination step of determining whether a reward is generated when using the generation painting style; a reward explanation step of outputting a paid painting style application, which is an application regarding the generation of a reward when using the generation painting style, when it is determined in the reward determination step that a reward is generated; and an image generation step of generating an image based on the generation painting style based on the input language or the sample image and obtaining a generated image when a commitment is input for the paid painting style application.

11. A non-transitory information recording medium readable by a computer storing the program according to claim 10.

Images