System for generating images based on input language or sample images - Patent Application 20070122997

Abstract

[Problem] To provide a system in which digital processing for paying a reward to a right holder, including an original creator, is performed even when it is difficult to determine whether or not unique characteristics of an image used as training data are included in an image to be generated. [Solution] A system for generating an image on the basis of an input language or a sample image, the system comprising: a style identification unit that identifies, on the basis of the input language or the sample image, a generation style, which is the style of an image to be generated; a reward determination unit that determines whether or not the use of the generation style will require a reward; a reward explanation unit that, if the reward determination unit determines that the use of the generation style will require a reward, outputs a chargeable style offer, which is an offer indicating that the use of the generation style will require a reward; and an image generation unit that, if acceptance of the chargeable style offer is input, generates an image on the basis of the input language or the sample image and on the basis of the generation style, and obtains the generated image.

Description

[Technical Field]

[0001] The present invention relates to a system for generating images based on input language or sample images. [Background technology]

[0002] Japanese Patent Publication No. 7169027 describes a character image generation device and a learning model generation device. This device has an image generation unit, a character variable generation unit, and a posture variable generation unit. The image generation unit is an element for generating an image from multiple input variables using an image generation model trained by machine learning using training image data. The character variable generation unit is an element for generating multiple character variables corresponding to the characteristics of each character type. The posture variable generation unit is an element for generating multiple posture variables corresponding to the postures of the characters. Then, this character image generation device inputs the plurality of character variables generated by the character variable generation unit and the plurality of posture variables generated by the posture variable generation unit into the image generation model as the plurality of variables, thereby generating a character image of a character of a type specified by the plurality of character variables and having a posture specified by the plurality of posture variables. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7169027 Summary of the Invention [Problem to be solved by the invention]

[0004] Systems that generate character images based on input information like this are already known. However, if the generated image contains the characteristics of the image used as training data, copyright law issues arise. Even if it is difficult to determine whether the generated image contains the characteristics of the image used as training data, it is desirable to perform digital processing that ensures that rights holders, including the original author, are paid compensation. [Means for solving the problem]

[0005] This invention is based on the finding that a digital system can be provided that appropriately pays remuneration to authors and original authors by determining whether or not remuneration will be generated based on input information.

[0006] The first invention relates to a system 1 for generating an image. The system 1 generates an image based on an input language or a sample image. This system 1 includes an art style identification unit 3, a reward determination unit 5, a reward explanation unit 7, and an image generation unit 9. The style identification unit 3 is an element for identifying a generated style based on the input language or a sample image. The generated style is the style of the image generated by this system. An example of a style is the style of a certain author. The remuneration determination unit 5 is an element for determining whether or not a remuneration is generated when the generated image style is used. The remuneration explanation unit 7 is an element for outputting a style fee offer when the remuneration determination unit 5 determines that a remuneration will be generated. The style fee offer is an offer regarding the generation of a remuneration when the generated style is used. The image generation unit 9 is an element for generating an image based on a generated image style based on the input language or a sample image, when consent is input to the image style fee request, and obtaining the generated image.

[0007] The style fee offer may include one or more of the following information: whether the author of the generated style is alive or dead; the remaining copyright term; and whether the generated style is in the public domain.

[0008] The image generation unit 9 may include a first trained model 11 and a second trained model 13. The first trained model 11 is a model for generating an image (first-stage virtual image) based on an input language or a sample image. There is no need to actually create an image. The second trained model 13 is a model for performing additional training and fine tuning on the generated image style.

[0009] It is preferable that the image generation unit 9 includes a digital watermark related to the specified style or file-embedded copyright information in the generated image. The generated image may include both information about the author of the generated style and information about the author of the generated image.

[0010] An example of a generated image is a cartoon image. A frame border layer relating to the frame borders of manga images, A speech bubble layer for speech bubbles within a frame frame, Character layers for characters within a frame, and Background layer for the background within the frame border, Preferably, it contains: Preferably, the image generating unit 9 controls the comic image so that when the frame frame changes depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frame.

[0011] The second invention relates to a program for causing a computer to function as the above-mentioned system, a program for causing a computer to execute an image generation method, a non-transitory information recording medium that can be read by a computer and that stores such a program, and the like. The image generation process using a computer includes a style identification process (S101), a remuneration determination process (S102), a remuneration explanation process (S103), and an image generation process (S104). The style specification step (S101) is a step for specifying a creation style, which is the style of an image to be generated, based on an input language or a sample image. The remuneration determination step (S102) is a step for determining whether or not a remuneration is generated when the generated image style is used. The fee explanation step (S103) is a step for outputting a style fee offer, which is an offer regarding the payment of a fee when the generated style is used, if it is determined in the fee determination step (S102) that a fee will be paid. The image generating step (S104) is a step for generating an image based on the generated style based on the input language or the sample image, when consent is input to the style fee request, to obtain the generated image. [Effects of the Invention]

[0012] According to this invention, it is possible to provide a digital system that determines whether or not a fee will be paid based on input information, thereby ensuring that the author or original creator is appropriately paid a fee. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram illustrating an image generation system. [Figure 2] FIG. 2 is a flowchart illustrating the image generation method. [Figure 3] Figure 3 is an example of the input screen. [Figure 4] FIG. 4 is a conceptual diagram showing an example of each layer of the generated cartoon. [Figure 5] FIG. 5 shows the cartoon of FIG. 4 converted into a cartoon for a foreign market. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following describes embodiments of the present invention with reference to the drawings. The present invention is not limited to the embodiments described below, and also includes appropriate modifications of the embodiments below within the scope obvious to those skilled in the art.

[0015] FIG. 1 is a block diagram illustrating a system for generating an image (image generation system). As shown in FIG. 1, image generation system 1 includes a style identification unit 3, a reward determination unit 5, a reward explanation unit 7, and an image generation unit 9. These elements are virtual, and each element may be implemented by a single processor. System 1 may be implemented by a computer, and the processor may store programs and instructions in a memory unit and implement each process and function based on the instructions. System 1 may be a server that provides information to a user's client, or applications and programs may be installed on the user's client.

[0016] 2 is a flowchart illustrating the image generation method. As shown in FIG. 2, this method includes a style identification step (S101), a reward determination step (S102), a reward explanation step (S103), and an image generation step (S104). These steps are virtual, and some steps may be performed simultaneously.

[0017] A computer has an input unit, an output unit, a control unit, a calculation unit, and a memory unit, and each element is connected by a bus or the like to enable the exchange of information. Various information input from the input unit may be converted into digital information such as binary data, and the digital information may be processed on the computer. The memory unit may store a program or various information. When predetermined information is input from the input unit, the control unit reads the program stored in the memory unit. The control unit then reads the information stored in the memory unit as appropriate and transmits it to the calculation unit. The control unit also transmits the input information as appropriate to the calculation unit. The calculation unit performs calculation processing using the received various information based on instructions from the program and stores the results in the memory unit. The control unit reads the calculation results stored in the memory unit and outputs them from the output unit. In this way, various processes and steps are performed. Each unit or means executes these various processes. The computer may have a processor, and the processor may realize various functions and steps. The computer may be standalone. Some of the functions of the computer may be distributed between a server and a terminal. In this case, it is preferable that the server and the terminal are able to exchange information via a network such as the Internet or an intranet. The computer may include a processor and a memory coupled to the processor. The memory may store instructions that, when executed by the processor, cause the computer to perform various processes or function as various elements. The computer may be provided with various training data to construct a learning model and perform various calculations through machine learning. In this case, the computer may perform various analyses using the learning model created through machine learning and deep learning of AI (artificial intelligence).

[0018] Style identification process (S101) The style identification step (S101) is a step for identifying a creation style, which is the style of an image to be generated, based on an input language or a sample image. This step may be performed by the style identification unit 3 of the system 1. The style identification unit 3 is an element for identifying a creation style based on an input language or a sample image. The creation style is the style of an image generated by this system. An example of a style is the style of a certain author.

[0019] FIG. 3 is an example of an input screen. As shown in FIG. 3, system 1 displays a field for inputting information for creating an image. A user enters necessary information in this field, drags and drops an image file, or inputs information for identifying an image (e.g., a URL). Digital data related to an input language, digital data related to the input image, and the like are then input to system 1. An example of the input unit is a chat input unit. An example of system 1 is for generating an image such as a cartoon based on chat (and image file input). The art style may be selectable from a pull tab or a selection unit, and linguistic information for specifying the art style may be input to system 1 by the user's selection. The input language may include instructions related to the generated image in addition to the art style. Examples of instructions related to the generated image may include one or more of the following: facial expression, direction, background, position of the cartoon frame, type of effect, position of the effect, size of the effect, type of speech bubble, position of the speech bubble, size of the speech bubble, dialogue, font of the dialogue, color of the content, size of the content, position of the content, and context of the content.

[0020] Identifying art styles based on linguistic information Examples of art styles include purpose, art style, type of art, manga name, character name, author name, and user original. When a user inputs linguistic information related to a drawing into an input field, the art style is identified, for example, as follows. The language analysis unit 21 of the system 1 analyzes the linguistic information, which is, for example, digital information. Then, the language analysis unit 21 refers to an art style dictionary 23 and extracts the art style information from the linguistic information. For example, the art style dictionary stores keywords that indicate art styles and related words associated with each keyword. The language analysis unit 21 then compares the input linguistic information with whether it contains keywords or related words that indicate the art style. If the linguistic information contains keywords or related words, the language analysis unit 21 identifies the art style based on the keywords or related words. The identified generated art style may be stored in a storage unit as appropriate. The keywords and related words may be a set of multiple words. The linguistic information may be registered in the system 1 in advance and input to the system 1 by selecting it.

[0021] Examples of keywords indicating artistic styles are "patent drawings," "design images," "ink painting," "impressionism," "Demon Slayer: Kimetsu no Yaiba (registered trademark)," "Kingdom (registered trademark)," "Hatsune Miku (registered trademark)," "author A," "manga title B," "character C," "Chagall," and "Hello Kitty (registered trademark)." Other examples of keywords indicating artistic styles are "Dragon Ball (registered trademark)" and "Son Goku." An example of author A is "Walt Disney (registered trademark)" (Walt Disney (registered trademark) style). Examples of keywords indicating artistic styles are "caricature," "realistic," "deformed," "different colors," and "coloring book style" (uncolored). If these artistic styles are input, the generated images should reflect that style. To achieve this, machine learning can be performed to build a (second) trained model for each artistic style.

[0022] Examples of related words related to a keyword are "drawings used in a patent," "for a patent" and "drawings," and "patent application" and "drawings." Other examples of related words related to a keyword are "Kitty-chan (registered trademark)," "Kitty-chan," "Kitty-chan," and "Kitee-chan." Related words may include words that are reminiscent of the keyword or incorrect entries of the keyword.

[0023] For example, suppose a user inputs, "Please create patent drawings based on the handwritten drawings I will now input." Then, digital information related to "Please create patent drawings based on the handwritten drawings I will now input" is input to the system 1 as linguistic information (input method). The linguistic information is appropriately stored in the memory. The style identification unit 3 (the language analysis unit 21) of the system 1 refers to the style dictionary 23, reads keywords and related words from the style dictionary 23, and compares them with the linguistic information. As a result, the style identification unit 3 (the language analysis unit 21) of the system 1 determines that the linguistic information contains "patent" and "drawings," which are related words to "patent drawings." As a result, the style identification unit 3 (the language analysis unit 21) of the system 1 selects "patent drawings" as the creation style related to the linguistic information. Information related to the selected creation style, "patent drawings," is appropriately stored in the memory.

[0024] Identifying the style of painting based on input images, etc. A case will be described in which a user inputs several images into system 1 to identify an art style. For example, devices that identify an art style based on input images and build a learning model, such as the image generation unit and character variable generation unit in Japanese Patent No. 7169027, are already known. Therefore, this invention can also appropriately adopt known machine learning and learning model building methods. For example, a user uses a terminal (client) to drag and drop an image file or input information for identifying an image (e.g., a URL). Then, digital information related to the image file or information for identifying the image is input to system 1. System 1 may obtain digital information related to the image file based on the information for identifying the image.

[0025] The system 1 may store several reference images and output them for display on the user's terminal, and when the user selects a reference image, the reference image or information identifying the reference image may be input to the system 1.

[0026] The art style identification unit 3 reads whether or not the image file contains author-related information. If the image file contains author-related information, the art style identification unit 3 stores the author-related information in the storage unit. The art style identification unit 3 references the art style dictionary 23 and compares the author-related information with keywords and related words. If the author-related information contains any of the keywords and related words, the art style information may be obtained based on the author-related information. The author-related information may be embedded in the image, or may be added as additional information separately from the image. The art style dictionary 23 may store, in association with keywords, one or more of the following information: information on the life or death of the author (copyright holder) of the art style, the author's date of death, the remaining copyright term, and information on whether the art style is in the public domain (freely available).

[0027] If the image file does not contain author-related information, the art style identification unit 3 performs an operation of identifying the art style based on the image file. The art style identification unit 3 may use the input image to crawl a website, identify web pages containing similar images, and identify the art style. Alternatively, the input image and art style may be input as training data to build a learning model for identifying the art style, and the image may be input into the learned learning model to identify the art style. Image searches on websites are already known, for example, through Google's (registered trademark) image input site. Therefore, this invention can also utilize known similar image search systems as appropriate. The created art style obtained in this manner may be stored in a storage unit as appropriate. Alternatively, the system may learn from existing illustrations, generate tags to represent the images, and add them to the generated images. For example, a known algorithm such as DeepDanboru may be used to create identification information such as tags for the generated images, and prompts may be created using the tags. Furthermore, a model such as VLM that returns the similarity in natural language for unknown (class-unspecified) tags may be used to create identification information such as tags for the generated image and add it to the generated image.

[0028] The user also inputs multiple images of the manga character X that he or she has created. Then, multiple images of the character X are input to the system 1. The art style identification unit 3 uses the images of the character X as teacher data (training data) to generate a learning model (second learning model). The system 1 can obtain a trained model with the character X as its art style. Then, in the future, it will be possible to output an image related to the character X simply by inputting a rough image (for example, an image that identifies posture information) into the additional learning model. The system 1 will be able to select this learning model, and when this learning model is selected, information for selecting the learning model, for example, as linguistic information, will be input to the system 1.

[0029] Remuneration judgment process (S102) The reward determination step (S102) is a step for determining whether a reward will be generated when the generated style is used. This step may be executed by the reward determination unit 5 of the system 1. The reward determination unit 5 is a component for determining whether a reward will be generated when the generated style is used. This step is for illustrative purposes only and may be executed simultaneously with the style identification step (S101). The system 1 may include a style utilization database 25. The style utilization database 25 may store style utilization information, such as whether or not a reward is generated, the details of the reward, and the conditions for using the style, in association with the style. The style dictionary 23 and the style utilization database 25 may be separate storage units, or a single database may exist, and the style dictionary 23 and the style utilization database 25 may be considered to be part of the database. Furthermore, if author-related information is included in the image file, the reward determination unit 5 may read the included author-related information and determine whether or not a reward will be generated when the generated style is used.

[0030] Remuneration refers to a condition (obligation) for using an art style. Examples of remuneration include one or more of the following: copyright royalties, royalties for the use of derivative works, non-exercise of moral rights, trademark royalties, design royalties, know-how royalties, site usage fees, original art usage fees, tribute fees, donations, cash, cryptocurrency, points, and trading stamps. Other examples of remuneration include an offer to clearly state the author's name, to add the author's name to the generated image, or to agree to count the use of the art style when generating an image. Other examples of remuneration include soft currency, hard currency, points for subscriptions, rewards for completing surveys, and incentives for actions such as "liking" others. These may be used as remuneration to enable the generation of generated images. Copyright does not necessarily exist for an art style. Even if copyright does exist, it may be difficult to collect royalties due to the complexity of litigation. However, by using this system 1, it becomes possible to provide some kind of "reward" to those who provide art styles. Providing such a digital system will provide a strong motivation for creating and disseminating creative images.

[0031] Trademark Infringement Checker The remuneration determination unit 5 may check whether or not the use of the generated image may constitute an infringement of a trademark right. In this example, for example, the system checks whether there are identical or similar marks in a prior trademark database based on the style of the artwork. The style of the artwork may be linguistic information or an image. The search for similar marks based on language is implemented using similar language searches, such as those on the Japan Patent Office's former digital library, jplatpat. Therefore, the remuneration determination unit 5 may use such websites, or it may build its own database and refer to a similar language dictionary to select prior registered trademarks with marks similar to the input language. Then, the system 1 pre-stores identifying information (e.g., similar group codes) of designated goods and services likely to be related to the image. If a prior trademark contains this identifying information, the system 1 may store it in its memory as a potentially related prior trademark. For example, the system may store trademarks such as 26A01 (related to manga) and 42P01 (design ideas) in its memory, and if a prior registered trademark with a similar mark is found, it checks whether those similar group codes are included. If the prior registered trademark is stored in association with the similar group code, for example, if a design idea is created, it may be determined that it may infringe the prior registered trademark, and the determination may be stored in the memory. Image search sites for trademarks are already known, so System 1 may use those sites to obtain information on prior registered trademarks that may be similar when using the input image or style.

[0032] Copyright Information Checker The compensation determination unit 5 may have a function for checking copyright information. For example, in association with a keyword, one or more of the following may be stored: the remaining copyright term (particularly the remaining copyright term in the user's country of residence), whether the work is copyright-free, whether it is open source, whether it is public domain, and whether the author is alive or dead, and the need for compensation may be determined based on this information. Furthermore, the user may be prompted to input the degree of reliance (e.g., a numerical value), and if the reliance is high (exceeding a threshold), the name of the author on whom the work was relied may be displayed and the copyright information may be read from the storage unit and output. For example, if the style (keyword) is that of an author who died a considerable time ago (more than a threshold), information stored in association with the author, such as the author's death information and whether the work is in the public domain, may be read from the storage unit and output. Furthermore, if the author of a certain style (keyword) is deceased but within the threshold (the copyright expiration date), the compensation determination unit 5 may read the author's date of death from the storage unit, have the calculation unit perform a calculation to determine the copyright expiration date, and output either the author's date of death or the remaining copyright period (countdown), or both. A user may also add a statement to a generated image waiving their own copyright. In this case, another user may freely create a new generated image based on that generated image. However, if that user generates an image based on a style requiring compensation, the usage conditions for that style are attached to the generated image, and other users will also be subject to those attached usage conditions.

[0033] If no reward is paid For example, the remuneration determination unit 5 reads out "patent drawing" as style information from the storage unit. Alternatively, the remuneration determination unit 5 receives "patent drawing" as style information from the style identification unit 3. In this case, information indicating that no remuneration is required is stored in the style utilization DB 25 (or style dictionary 23) in association with the "patent drawing." The remuneration determination unit 5 reads out the information indicating that no remuneration is required, stores it in the storage unit as appropriate, and outputs it to the image generation unit 9. Upon receiving the information indicating that no remuneration is required, the image generation unit 9 executes the image generation process. However, even if no remuneration is required, the system 1 may output information to the user's terminal for displaying rights management information, such as information about the author related to the style, on the display unit of the user's terminal.

[0034] When rewards are generated For example, the reward determination unit 5 reads out "character C" as art style information from the storage unit. The art style usage DB 25 stores usage information for character C in association with "character C." The usage information for character C includes information regarding the occurrence of a reward when an image is generated using the system 1 using the art style of character C. This may be, for example, if a flag regarding the occurrence of a reward is stored in association with "character C." The reward determination unit 5 then reads out the flag regarding the occurrence of a reward, and may store the information regarding the occurrence of a reward in the storage unit as appropriate, or may output it to the reward explanation unit 7.

[0035] Remuneration explanation process (S103) The fee explanation step (S103) is a step for outputting a style fee offer, which is an offer regarding the payment of a fee for using the generated style, when the fee determination step (S102) determines that a fee will be generated. This step may be executed by the fee explanation unit 7. The fee explanation unit 7 is an element for outputting a style fee offer when the fee determination unit 5 determines that a fee will be generated. The style fee offer is an offer regarding the payment of a fee for using the generated style. For example, the fee explanation unit 7 outputs the style fee offer to the user's terminal. Then, the display unit (e.g., a monitor) of the user's terminal displays the style fee offer. In this way, the user can understand that a fee will be generated when using the style. The style fee offer simply indicates that a fee will be generated, and the conditions may be viewable via a link or the like. Furthermore, the style fee offer may include, in addition to information regarding the payment of a fee, one or more of the following information: whether the author of the generated style is alive or dead, the remaining copyright term, and whether the generated style is in the public domain.

[0036] When the style fee offer is displayed on the display unit of the user's terminal and the user inputs an input to decline the offer, information to the effect that the offer has been declined is output from the user's terminal to the system 1. When information to the effect that the offer has been declined is input to the system 1, the system 1 may perform the process again from the style identification step (S101).

[0037] Image generation process (S104) The image generation step (S104) is a step for generating an image based on the generated style based on the input language or sample image, when consent is input to the request for paid image style, to obtain the generated image. Of course, even if the system 1 determines in the remuneration determination step (S102) that no remuneration is required, the system 1 generates an image based on the generated style based on the input language or sample image, to obtain the generated image. This step may be executed by the image generation unit 9. The image generation unit 9 is an element for generating an image based on the generated style based on the input language or sample image, when consent is input to the request for paid image style, to obtain the generated image. The image generation unit 9 generates an image based on the generated style based on the input language or sample image, to obtain the generated image, using an image generation model machine-learned using training image data. The image generation model generated by machine learning may be generated using a paid service via API, such as "DALL-E," or may be a commercially available, openly released model, such as "Stable Diffusion XL." The image generation model generated by machine learning may be an application of StyleGAN, or may appropriately utilize a publicly available AI such as chatGPT (registered trademark). For details of the image generation process, please refer to, for example, the book "Enjoy Your LoRA."

[0038] The image generation unit 9 may include a first trained model 11 and a second trained model 13. That is, the image generation unit 9 may have an additional learning unit. The first trained model 11 is a model for generating an image (first-stage virtual image) based on an input language or a sample image. However, if a second trained model exists, it is not necessary to actually create an image using only the first trained model 11. The first trained model can be said to be a model for generating an image by machine learning based on either or both of the input language and the sample image.

[0039] The second trained model 13 is a model for performing either or both of additional training and fine tuning on the generative art style. The system 1 may have a plurality of second trained models 13 corresponding to art styles. In this case, by using the second trained models 13 corresponding to the art styles, it becomes possible to quickly create images. An example of the second trained model 13 is an additional trained model for character X.

[0040] The image generation unit 9 preferably includes a digital watermark or file-embedded rights information related to the identified style in the generated image. In other words, the image generation unit 9 preferably adds rights management information to the generated image. For example, the rights management information (fingerprint information) may be visible when the generated image is enlarged or viewed through a polarizing filter. The rights management information may also be permanently added to the generated image. The rights management information may be a creator identification tag such as "Generated by..." or a tag identifying the author of the original image such as "Inspired by...." In this way, the rights management information may be information for paying a fee to the source of the style reference. The rights management information may also be information for identifying a fee. The information for identifying a fee may be a URL or information identifying a storage area in the storage unit, or may be stored in association with the style in either or both of the style dictionary 23 and the style utilization DB 25. The system 1 stores the image generated by the image generation unit 9 (the generated image) in the storage unit as one of the reference images. Furthermore, if another user references the generated image and uses its style, it is preferable that compensation also be paid to the rights holder of the style used by system 1 when generating the generated image. Another example of rights management information may include the names of licensor A and licensee B. Such rights management information may clearly indicate the "permission" in the form of "Original by A Inspired by B" or "This generated image B is authorized by A original copyright holder, detail is written in http: / / ...". This allows B to use the service in a socially and legally safe manner, and also allows for clear statements about the rights in the generated image (e.g., whether it is recursively learnable or has a volatile license), making it flexible to withstand future changes in law and user demands.

[0041] An example of a generated image is a cartoon image. A frame border layer relating to the frame borders of manga images, A speech bubble layer for speech bubbles within a frame frame, Character layers for characters within a frame, and Background layer for the background within the frame border, Preferably, it contains: Preferably, the image generating unit 9 controls the comic image so that when the frame frame changes depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frame.

[0042] program The above system can be implemented by installing a program in the storage unit of a computer or server. In other words, this specification also provides a program for implementing a computer as the above system, a program for causing a computer to execute the above procedures, and a recording medium storing such a program. A computer or server with this program installed can execute the above processes and store user-specific information. An example of a recording medium is a non-transitory recording medium that can be read by a computer. Examples of non-transitory recording media include a CD-ROM, a DVD, a USB memory, and a hard disk. [Example]

[0043] A chat tool that generates new UGC images by allowing users to select a reference image and style, and then providing detailed prompts such as facial expressions and acting, while retaining the characteristics of the image compressed using latent space.

[0044] This system allows a chat input section and a sample image input section to be displayed on the user's terminal. In addition, in a reference image display mode for displaying a reference image, the system allows the reference image to be displayed on the user's terminal. The user can then input information about the style by referring to the reference image. In addition, in a style selection mode, the system presents examples of styles to the user, allowing the user to specify the style of the generated image by selecting one.

[0045] A user wants to create a manga and launches a chat tool app installed on their device, which then allows them to create manga, images, and videos via chat.

[0046] A user may input "I want to draw a manga" into the chat section. System 1 then analyzes the input language and enters manga creation mode. When the user selects manga mode, the selection information may be input into the system, and system 1 may enter manga creation mode.

[0047] A user inputs an idea for a manga into the chat section. For example, the user inputs an outline of the manga in language. System 1 then analyzes the language information and reads out example frame borders from the memory unit. System 1 then outputs the example frame borders to the user's device. The display unit of the user's device displays the example frame borders. By inputting language into the chat section, the user can adjust the borders that make up the frame borders as needed. In this way, a draft of the frame border layers is created and stored in the memory unit. By specifying each frame that makes up the frame border, the user can generate an image to be created in the specified frame. Furthermore, it is preferable that the mask can blend the influence of the calculation results of other layers using a decimal value between 0 and 1. The decimal value indicates the blending ratio. Masks can be used as frame lines, free shapes like speech bubbles, text on speech bubbles, overlapping characters, foreground and background, effect lines, compositing processes like cartoon symbols, and shading expressions like screen tones, or any combination of two or more. Furthermore, because the mask itself can be treated as an image with a shade range between 0 and 1, the creation of this mask itself can be an ingenious way to create UGC (user-generated content), and by displaying it in a gallery, it can become an asset that other users can use.

[0048] 4A and 4B are conceptual diagrams showing examples of layers of a generated cartoon. As shown in Fig. 4A, the frame border layer specifies a frame border.

[0049] A user may input language containing "character C" into the chat section, select an image of "character C," or select a second learning model related to "character C." Alternatively, a user may input multiple images into system 1, and system 1 may determine that the images are related to "character C." System 1 then determines that the style of the image to be generated is related to "character C."

[0050] System 1 reads from the storage unit the rights management information stored in association with character C. Because system 1 stores the rights management information in this way, even if it is uncertain whether a copyright will be created, it is possible to generate a fee by using character C in this system 1. System 1 may read the rights management information stored in association with the referenced character C. System 1 then determines whether or not using character C (or a trained model related to character C) requires a fee.

[0051] When the system 1 determines that using the character C (or the trained model for the character C) requires a reward, it outputs information about the reward and the usage conditions to the user's terminal. The display unit of the user's terminal displays the reward and the usage conditions.

[0052] If the user agrees to the reward and terms of use, the user inputs the consent into the user's terminal. The user's terminal then outputs the consent information to the system 1. The system 1 receives the consent information and inputs it into the system 1.

[0053] Once consent is entered into the system 1, it becomes possible to generate an image based on the art style. For example, a user may enter "Please create an image of character C holding a weapon and preparing for battle" into the chat input unit. The terms for use of character C are consented to as described above. The language analysis unit analyzes the terms "holding a weapon" and "prepared for battle" and creates a model using the first learning model. Then, using the second learning model, it creates an image of character C, generating an image of character C holding a weapon and preparing for battle. Figure 4(b) shows an example of a character image in a specified frame. In this example, the character is accompanied by a copyright notice and an indication of the art style used, and rights management information can be referenced in relation to this. In this example, text is displayed. Alternatively, the image may contain code information, such as a QR code (registered trademark), in a form that is not visible to the user. Reading the code information using a reading device such as a camera allows access to the rights management information.

[0054] Furthermore, for example, a background image may be specified in the chat input section. Alternatively, a user may input, "I'm going to input a background image file, so please make it a cartoon-style background," and then drag and drop the background image file into the chat input section. The background of the specified frame frame will then be based on the input image. The created background image may be stored in the background layer of the specified frame frame. In this case, the background image may be read in association with the frame frame specification information. FIG. 4(d) shows an example of a background image for a specified frame.

[0055] Furthermore, the user may input the lines of character C into the chat input unit. When the lines of character C are input into system 1, system 1 performs linguistic analysis of the lines and creates example speech bubbles based on the learning model. The created speech bubbles and lines are output to the user's terminal. The user's terminal receives the information and displays the speech bubbles and lines on the display unit. In response, the user can adjust the speech bubbles and lines using the chat input unit. Examples of such input are "Make the speech bubble a little smaller" and "Emphasis on the line...". The system performs linguistic analysis and adjusts the speech bubbles and lines. The speech bubbles may be stored in a speech bubble layer and the lines in a line layer, associated with the frame frames. Figure 4(c) is a diagram showing an example of a speech bubble layer. Note that the speech bubble layer and the line layer may be separate.

[0056] By repeating this process for other frame frames, a manga can be created. In this case, the image generation unit 9 can generate manga pages by stacking each layer. At this time, the lower layer may delete the part below the content of the upper layer. Rights management information may be stored for each frame frame in the created manga image. Rights management information may also be stored for the entire manga.

[0057] This system 1 may have a mode conversion function that allows a created manga to be adapted to suit the intended use, such as adjusting it for a foreign country or converting it to vertical reading for tablets.

[0058] The user selects the English mode. Then, English mode instruction information is input to the system 1. Based on the program's instructions, the system 1 refers to a translation learning model (translation engine) and translates the dialogue into English. The system 1 also changes the frame frames in the frame frame layer from right to left and left to right, and creates updated frame frames.

[0059] The system 1 reads the content of each layer, such as the speech bubble layer related to the speech bubbles within the frame frame, the character layer related to the characters within the frame frame, and the background layer related to the background within the frame frame, stored in association with each frame frame, and regenerates images to match the updated frame frame. At this time, the dialogue is converted to the English-translated dialogue described above. In this manner, the created manga (image) can be easily converted into a manga for a foreign market. The generated manga for a foreign market may be stored in an appropriate storage unit. The generated manga for a foreign market may also be output from the system 1 to a user's terminal, and the manga may be displayed on a display unit of the user's terminal. The user may then adjust the manga for a foreign market by inputting commands into the chat input unit. In other words, the image generation unit 9 preferably controls the manga image so that, when the frame frame changes depending on the input language, one or more of the background, characters, and speech bubbles change in accordance with the change in the frame frame. Figure 5 shows the manga of Figure 4 converted into a manga for a foreign market. In this way, the frame layout is automatically converted, and the character positions and language are also automatically converted accordingly. Furthermore, customization for the conversion destination can be performed by referring to a conversion dictionary. In this example, the background color of the speech bubble changes based on the information in the conversion dictionary. [Industrial Applicability]

[0060] The present invention can be used in the information industry because it can easily generate images, cartoons, etc. [Explanation of symbols]

[0061] 1. Image generation system 3. Style Identification Section 5 Remuneration Judgment Department 7. Remuneration Explanation Section 9 Image generation unit 11 First trained model 13 Second trained model 21 Language Analysis Department 23 Art Style Dictionary 25 Art Style Usage Database

Claims

1. 1. A system for generating images based on input language or sample images, comprising: a style specification unit that specifies a style of an image to be generated based on the input language or a sample image; a reward determination unit that determines whether a reward will be generated when the generated image style is used; a remuneration explanation unit that outputs a style fee offer, which is an offer regarding the generation of a remuneration when the remuneration determination unit determines that a remuneration will be generated, when the generated style is used; The system further comprises an image generating unit that, when consent is input to the request for paid style payment, generates an image based on the input language or the sample image and a generated image based on the generated style.

2. 10. The system of claim 1, The style fee offer includes one or more of the following information: whether the author of the generated style is alive or dead; the remaining copyright term; and whether the generated style is in the public domain.

3. 10. The system of claim 1, The image generation unit a first trained model for generating images based on an input language or a sample image; a second trained model for additional training and fine-tuning of the generated style; Including, the system.

4. 10. The system of claim 1, The system, wherein the art style is the art style of a certain author.

5. 10. The system of claim 1, The generated image may include a digital watermark relating to the identified style or may include file-embedded rights information.

6. 10. The system of claim 1, The generated image is added with both information about the author of the generated style and information about the author of the generated image.

7. The system of claim 1 , wherein the generated image is a cartoon image.

8. 8. The system of claim 7, The cartoon image is a frame border layer relating to frame borders of the comic image; a speech bubble layer relating to speech bubbles within the frame frame; a character layer for characters within the frame; and a background layer relating to the background within the frame frame; Including, the system.

9. 9. The system of claim 8, The image generation unit controls the comic image so that when the frame frame changes according to the input language, one or more of the background, the character, and the speech bubble change in accordance with the change in the frame frame.

10. On the computer, a style specification step of specifying a generation style, which is a style of an image to be generated, based on an input language or a sample image; a reward determination step of determining whether or not a reward will be generated when the generated image style is used; a remuneration explanation step of outputting a style fee offer, which is an offer regarding the generation of a remuneration when the remuneration determination step determines that a remuneration will be generated, for using the generated style; and an image generation step of generating an image based on the input language or the sample image and the generated image in accordance with the generated image style when consent is input to the request for paid style payment.

11. A computer-readable non-transitory information recording medium storing the program according to claim 10.

Images