Image processing system and method for generating decorative character image

Abstract

The present invention generates an image of a character using a machine learning model. This image processing system comprises: an information acquisition unit that acquires character information for designating a character and a living organism image being an image of a living organism including a face; and an image generation unit that, by inputting input information to an image generation model using the character information and the living organism image, causes the image generation model to generate a decorative character image. The decorative character image is an image showing a decorative character being a character to which a decoration showing the face of the living organism shown in the living organism image is applied. The image generation model is trained so as to generate, on the basis of the inputted information, an image to which the decoration showing the face of the living organism shown in the living organism image is applied.

Description

Image processing system and method for generating decorative character images

[0001] The present specification relates to a technique for generating an image showing characters.

[0002] Patent Literature 1 discloses a technique for training a neural network using a group of prototype characters as input patterns and a group of transformed characters as training patterns. In this technique, when an unlearned character is input to the neural network, a transformed character obtained by transforming the unlearned character according to a transformation rule is output.

[0003] Japanese Patent Application Publication No. 5-266226

[0004] Thus, there is a need for a technique for generating images of characters.

[0005] This specification discloses a technique for generating images of characters using machine learning models.

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] An image processing system comprising: an information acquisition unit that acquires character information for specifying characters and a biological image that is an image of a biological image including a face, wherein at least a portion of the character information and the biological image are acquired based on user input; an image generation unit that uses the character information and the biological image to cause an image generation model to generate a decorated character image by inputting input information into the image generation model, wherein the decorated character image is an image showing decorated characters that are characters that are decorated to show the face of the biological image shown in the biological image, and the image generation model has been trained to generate a decorated image showing the face of the biological image based on the input information; a display control unit that displays the decorated character image on a display unit; and an image forming unit that prints, embroiders, or cuts the decorated character image displayed on the display unit onto an object.

[0008] According to the above configuration, a decorative character image is generated by inputting input information into an image generation model using character information and a biological image, including information acquired based on a user's input. As a result, a decorative character image corresponding to the user's input can be generated using a machine learning model. Furthermore, the decorative character image is displayed and printed, embroidered, or cut, so that a decorative character image corresponding to the user's input can be provided to the user.

[0009] [Application Example 2] A method for generating a decorative character image, comprising: a first step of using a biological image, which is an image of a biological organism including a face, to generate feature information based on a plurality of feature points of the face of the biological organism shown in the biological image; and a second step of causing an image generation model to generate a decorative character image by inputting input information, including an input image showing characters and the feature information, into the image generation model, wherein the decorative character image is an image showing decorative characters decorated to show the face of the biological organism shown in the biological image, and the image generation model is trained to generate an image decorated to show the face of the biological organism shown in the biological image based on the input information.

[0010] According to the above configuration, a decorated character image is generated by inputting input information including an input image showing characters and facial feature information of a creature generated using a creature image into an image generation model. As a result, a decorated character image that reflects the facial features of the creature shown in the creature image can be generated using a machine learning model.

[0011] [Application Example 3] A method for generating a decorated character image, comprising: a first step of using a creature image, which is an image of a creature including a face, to generate color information based on multiple colors possessed by the creature shown in the creature image; and a second step of causing an image generation model to generate a decorated character image by inputting input information, including an input image showing characters and the color information, into the image generation model, wherein the decorated character image is an image showing decorated characters decorated to show the face of the creature shown in the creature image, and the image generation model is trained to generate an image decorated to show the face of the creature shown in the creature image based on the input information.

[0012] According to the above configuration, a decorated character image is generated by inputting input information, including an input image showing a character and color information of a creature generated using a creature image, into an image generation model. As a result, a decorated character image that reflects color characteristics of the creature shown in the creature image can be generated using a machine learning model.

[0013] The technology disclosed in this specification can be realized in various forms, such as an image processing system, a method for generating decorative character images, an image processing device, a computer program for realizing the functions of those systems, methods, or devices, a recording medium on which that computer program is recorded (e.g., a non-temporary recording medium), etc.

[0014] 1. A block diagram showing the configuration of a system 1000 according to a first embodiment. A block diagram showing an example of an image generation model 900. A flowchart of decorated character generation processing according to the first embodiment. A diagram showing examples of images used in decorated character generation processing. A diagram showing an example of a selection screen WI1. A diagram showing an example of an output screen WI2. A flowchart of decorated character generation processing according to a second embodiment. An explanatory diagram for the second embodiment. A block diagram showing an example of an image generation model 900B according to a modified example. A diagram showing an example of a character outline image TEI.

[0015] A. First Embodiment: A1. Configuration of System 1000 FIG. 1 is a block diagram showing the configuration of system 1000 of the first embodiment. System 1000 includes a terminal device 100, an image processing server 200, and a printer 300. Terminal device 100 and printer 300 are connected to a local area network LN and are communicatively connected to each other. Local area network LN is connected to the Internet IT, and image processing server 200 is connected to the Internet IT. Therefore, terminal device 100 and image processing server 200 are communicatively connected via local area network LN and Internet IT. In a modified example, as described below, a sewing machine 400 or a cutting machine 500 may be provided instead of printer 300.

[0016] The image processing server 200 is, for example, a computer, such as a cloud server, operated by a business that manufactures and sells the printer 300 .

[0017] The image processing server 200 includes a processor 210, a storage device 220, a graphics processing unit 260 (hereinafter referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 220 includes a volatile storage device 222 and a non-volatile storage device 224.

[0018] The processor 210 is a device configured to process data, such as a central processing unit (CPU). The volatile storage device 222 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 224 is, for example, a flash memory. The non-volatile storage device 224 stores data for the computer programs PGs and the image generation model 900. The image generation model 900 is a program module that forms a trained machine learning model.

[0019] The computer program PGs and the image generation model 900 are provided, for example, in a form of being uploaded by a business operator that operates the image processing server 200. The processor 210 executes the computer program PGs to cooperate with the terminal device 100 and execute a decorated character generation process, which will be described later.

[0020] The GPU 260 is a computing device configured to execute various numerical calculations such as image processing and machine learning. The GPU 260 executes various calculations in accordance with instructions from the processor 210. In this embodiment, the GPU 260 is used to execute calculations of the image generation model 900. The communication interface 270 is an interface for connecting to the Internet (IT), and is, for example, a wired interface conforming to Ethernet (registered trademark).

[0021] The terminal device 100 is a computer, for example, a mobile terminal such as a smartphone, a tablet computer, etc. In a modified example, the terminal device 100 may be a personal computer.

[0022] The terminal device 100 includes a processor 110, a storage device 120, a display unit 140, an operation unit 150, and a communication interface 170. These elements are connected to each other via a bus. The storage device 120 includes a volatile storage device 122 and a non-volatile storage device 124.

[0023] The processor 110 is a device configured to process data, such as a CPU. The volatile storage device 122 is a DRAM, and the non-volatile storage device 124 is a flash memory. The non-volatile storage device 124 stores data of the computer program PGt and the biological image CI.

[0024] The computer program PGt is an application program that is provided, for example, in the form of being downloaded from a server connected via the Internet IT. By executing the computer program PGt, the processor 110 functions as an application that executes the decorative character generation process described above in cooperation with the image processing server 200.

[0025] The organism image CI is an image showing an organism, and is prepared by, for example, a user of the terminal device 100 and stored in the non-volatile storage device 224. For example, the organism image CI is a photographic image generated by a user taking a picture of an organism using the camera function of the terminal device 100. Alternatively, the organism image CI may be an image downloaded from a free resource site by a user operating a browser. The organism image CI is used in a decorative character generation process, as described below.

[0026] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive user operations, such as buttons or a touch panel overlaid on the display unit 140. In this embodiment, the display unit 140 and the operation unit 150 form a so-called touch screen. The user can input various instructions to the image processing server 200 by operating the operation unit 150. The display unit 140 may display operation elements (e.g., buttons, sliders, etc.), and the displayed elements may be operated through operation of the operation unit 150.

[0027] The printer 300 is a device that prints an image on a printing medium such as paper using color materials such as ink or toner.

[0028] A2. Image Generation Model 900 Figure 2 is a block diagram showing an example of an image generation model 900. The image generation model 900 may be any of various models that use input image data to generate output image data based on the input image. In this embodiment, the image generation model 900 includes a diffusion model 960, control models 991 and 992, and image encoders 971 and 972.

[0029] In this embodiment, the diffusion model 960 is a machine learning model called Stable Diffusion. Stable Diffusion is a model that synthesizes a high-resolution image using a latent diffusion model. The technology for synthesizing a high-resolution image using a latent diffusion model is disclosed in, for example, the following paper: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjoern Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models", arXiv:2112.10752, April 13, 2022, http: / / arxiv.org / abs / 2112.10752

[0030] Data for pre-trained Stable Diffusion models is made publicly available on the Internet by Stability AI. In this embodiment, the data for the publicly available pre-trained model is used as the data for the image generation model 900. The image generation model 900 includes a text encoder 910, an image encoder 920, a latent variable model 930, and an image decoder 940. The text encoder 910 is configured to convert text PT into a vector tv (such a vector tv is also called a text embedding). The image encoder 920 is configured to convert an input image II into a latent variable lvi. The latent variable model 930 is configured to perform a noise addition process that adds noise to the latent variable lvi and a process that outputs a processed latent variable lvo by performing a de-diffusion process that removes noise from the noisy latent variable. The latent variable model 930 includes a neural network called a U-Net (not shown) for performing the de-diffusion process. The image decoder 940 is configured to generate an output image OI using the latent variable lvo. The latent variable model 930 uses the vector tv obtained from the text encoder 910 for conditioning in the de-diffusion process. By executing this de-diffusion process, the latent variable model 930 can generate a latent variable lvo that corresponds to an image conditioned by the text PT. The text encoder 910 is pre-trained so that the vector tv obtained from the text PT and the image represented by the text PT can be associated. As the text encoder 910, a text encoder pre-trained using a technique called CLIP (Contrastive Language-Image Pre-training) is used. CLIP is a technique released by OpenAI.

[0031] The image generation model 900 can generate a new output image OI represented by the text PT by using a vector tv from the text encoder 910 to perform a de-diffusion process using randomly generated noise. This technique is also called txt2img. The image generation model 900 can also generate an output image OI by using an input image II and the text PT, where the input image II is modified according to the text PT. This technique is also called img2img. In this embodiment, the img2img technique is used to generate a decorated character image.

[0032] 2, the input image II is generated by a character image generation unit 211 and a smoothing processing unit 212. The character image generation unit 211 and the smoothing processing unit 212 are functional units realized by the processor 210 of the image processing server 200 executing the computer program PGs. The character image generation unit 211 generates a character image FI using character information TXI. The smoothing processing unit 212 performs a smoothing process on the character image FI to generate an input image II showing characters.

[0033] The control models 991 and 992 use a conditional image generation technique called ControlNet to provide additional conditions to the diffusion model 960 so that the characteristics of the biological image CI are reflected in the output image OI. The control models 991 and 992 are convolutional neural networks (CNNs) that include a copy of the U-net encoder of the diffusion model 960 and a zero-convolution layer. The zero-convolution layer is a convolution layer trained with the initial values ​​of parameters (weights and biases) set to zero. The U-net encoders of the control models 991 and 992 receive input of data similar to that of the U-net encoder of the diffusion model 960 (not shown), as well as condition data VTf and VTc indicating additional conditions to be provided to the diffusion model 960. In the first control model 991, the additional condition is a condition regarding the facial expression (facial structure) of the biological image CI. In the second control model 992, the additional condition is a condition regarding the color of the biological image CI.

[0034] Data output from each block of the encoder of the control models 991 and 992 is processed by a zero convolution layer corresponding to each block and input to the corresponding block of the decoder of the U-net of the diffusion model 960. This imposes additional conditions on the diffusion model 960, and an output image OI is generated in accordance with the additional conditions. The technology of such control models 991 and 992 (ControlNet technology) is disclosed, for example, in the following paper: Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, "Adding Conditional Control to Text-to-Image Diffusion Models", arXiv:2302.05543v3, November 26, 2023, https: / / arxiv.org / pdf / 2302.05543

[0035] When the condition images FPI, CPI are input, the image encoders 971, 972 generate and output the above-mentioned condition data VTf, VTc by converting the condition images FPI, CPI into a data format that can be input to the control models 991, 992. The condition data VTf, VTc are, for example, vectors with a predetermined number of dimensions. The image encoders 971, 972 use image encoders that have been pre-trained using the above-mentioned technology called CLIP.

[0036] As shown in FIG. 2 , the condition image FPI input to the first image encoder 971 is generated by the feature point extraction unit 213. The feature point extraction unit 213 is a functional unit realized by the processor 210 of the image processing server 200 executing the computer program PGs. The feature point extraction unit 213 uses the organism image CI to generate, as the condition image FPI, a feature point image that indicates multiple points on the face of the organism represented by the organism image CI. Hereinafter, the condition image FPI input to the first image encoder 971 will also be referred to as the feature point image FPI.

[0037] As shown in FIG. 2 , the condition image CPI input to the second image encoder 972 is generated by a background removal processor 214 and a color reduction processor 215. The background removal processor 214 and the color reduction processor 215 are functional units implemented by the processor 210 of the image processing server 200 executing the computer program PGs. The background removal processor 214 performs background removal processing on the biological image CI to generate a processed biological image CIr in which the background has been removed from the biological image CI. The color reduction processor 215 uses the processed biological image CIr to generate a color palette image as the condition image CPI, which indicates the primary colors contained in the biological image shown in the processed biological image CIr. Hereinafter, the condition image CPI input to the second image encoder 972 will also be referred to as a color palette image CPI. Details of the processing performed by these functional units 211-215 will be described later.

[0038] As can be seen from the above explanation, the input information input to the image generation model 900 includes the text PT, the input image II, the feature point image FPI, and the color palette image CPI.

[0039] A3. Decorative Character Generation Process FIG. 3 is a flowchart of the decorative character generation process of the first embodiment. The decorative character generation process uses an image generation model 900 to generate an image of characters decorated with a creature's face (e.g., a dog's face) (hereinafter, also referred to as a decorative character image OI). The decorative character generation process is executed by a terminal device 100 (processor 110) functioning as an application and an image processing server 200 (processor 210) working together. The terminal device 100 (application) mainly executes processes related to a screen functioning as a user interface (referred to as a UI screen) and processes for controlling the printer 300 (or, in a modified example, a sewing machine 400 or a cutting machine 500). The image processing server 200 mainly executes processes for generating a decorative character image based on user instructions acquired via the terminal device 100.

[0040] In S110, the image processing server 200 acquires character information from the user via the terminal device 100. That is, the character information is acquired based on user input. The acquired character information includes information indicating the character to be processed and font information indicating the font type of the character. In this embodiment, the character to be processed is the alphabet. The character to be processed may include some or all of kana characters (hiragana or katakana), kanji characters, numbers, and Arabic characters. The font type is selected, for example, from multiple font candidates prepared in advance. For example, the number of characters to be processed is not limited to one. That is, the number k of characters to be processed may be an integer equal to or greater than two. In this embodiment, the three alphabet characters "ABC" are acquired as the characters to be processed (k=3).

[0041] Specifically, the terminal device 100 displays an input screen (not shown) on the display unit 140, which includes an input field for inputting characters to be processed and a list of selectable fonts. On the input screen, the user inputs one or more characters to be processed in the input field and selects one font from the list. The terminal device 100 transmits character information, including information indicating the characters input by the user and information indicating the font selected by the user, to the image processing server 200. As a result, the image processing server 200 acquires the character information.

[0042] In S115, the image processing server 200 (character image generation unit 211) generates a character image FI based on the character information. The character image FI is an image representing the character to be processed. Furthermore, the character represented by the character image FI has the shape of the font represented by the font information. If the character to be processed is multiple characters, the character images FI are generated one character at a time. For example, if the character to be processed is "ABC," three character images FI representing each character are generated. Note that, in a modified example, one font image representing three characters may be generated.

[0043] 4A and 4B are diagrams showing examples of images used in the decorative character generation process. Fig. 4A shows a character image FI representing the character "A" TX. For example, the color of the characters in the character image FI is black, and the color of the parts other than the characters (i.e., the background BG) is white. The data for each character image FI is generated in the same data format as the input image II to be input to the image generation model 900.

[0044] In this embodiment, the data of the input image II is RGB image data. RGB image data is bitmap data that represents color values ​​using three color components: red (R), green (G), and blue (B). Images II and OI are rectangular images with two sides parallel to a first direction Dx and two sides parallel to a second direction Dy perpendicular to the first direction Dx. The input image II is represented by the color values ​​of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy (the color values ​​indicate the gradation values ​​of red (R), green (G), and blue (B) (e.g., values ​​greater than or equal to zero and less than or equal to 255)). The number of pixels in the first direction Dx and the number of pixels in the second direction Dy of the input image II accepted by the image generation model 900 are each predetermined. The data of the generated character image FI is RGB image data of the size (number of pixels in the first direction Dx and the second direction Dy) that the input image II should have.

[0045] In S120, the image processing server 200 performs a smoothing process on the character image FI to generate a processed character image. The generated processed character image is the input image II to be input to the diffusion model 960. The smoothing process is, for example, a smoothing process using a Gaussian filter. The standard deviation σ of the Gaussian filter is set to, for example, 20 to 50. Alternatively, the smoothing process may be performed using various smoothing filters, such as a mean filter or a median filter.

[0046] Fig. 4B shows an input image II (processed character image) generated using the character image FI of Fig. 4A. The outer edges of the character TXs shown in the input image II are blurred by the smoothing process.

[0047] In S125, the image processing server 200 acquires data of the biological image CI from the terminal device 100. Specifically, the terminal device 100 displays a file selection screen (not shown) on the display unit 140. The user operates the file selection screen to select data of the biological image CI (i.e., an image file) to be used to generate a decorated character image. The terminal device 100 transmits (i.e., uploads) the data of the biological image CI designated by the user to the image processing server 200. As a result, the image processing server 200 acquires the data of the biological image CI. Therefore, the biological image CI is an image designated by the user.

[0048] FIG. 4C shows an example of an acquired biological image CI. The biological image CI in FIG. 4C is an image showing the upper body of a dog, including the dog's face. In this embodiment, the biological image CI is assumed to be an image including an image of a biological creature's face viewed from the front. The biological image is not limited to a dog, but may be, for example, an image of other mammals such as a cat or a bear, a bird such as a parakeet or a parrot, an insect, or a reptile. The biological image may also be an image of a person. The biological image CI in FIG. 4C is, for example, a photographic image generated by a user taking a photo of their pet dog using the camera function of the terminal device 100. Instead of a photographic image, the biological image may be an image showing CG or an illustration.

[0049] In S130, the image processing server 200 (feature point extraction unit 213) uses the biological image CI to generate a feature point image FPI showing multiple feature points Pt on the face of the biological organism shown in the biological image CI. FIG. 4D illustrates a feature point image FPI generated using the biological image CI of FIG. 4C. The image processing server 200 detects multiple feature points Pt constituting the eyes, nose, and mouth included in the biological image CI, for example, using the facial landmark detection function of MediaPipe. The image processing server 200 generates a feature point image FPI showing the detected multiple feature points Pt with circles. MediaPipe is a technology released by Google (registered trademark). The feature point image FPI may also be an image including a mesh formed by straight lines connecting the multiple feature points Pt. The feature point image FPI is generated in a predetermined size (number of pixels in the first direction Dx and the second direction Dy) accepted by the first image encoder 971.

[0050] In S135, the image processing server 200 performs vector conversion processing on the feature point image FPI. Specifically, the image processing server 200 inputs the feature point image FPI to the first image encoder 971 ( FIG. 2 ) and causes the first image encoder 971 to generate a vector indicating the features of the feature point image FPI. The generated vector is also called a facial feature vector because it indicates the facial features of the organism shown in the organism image CI. The generated facial feature vector is the first condition data VTf to be input to the first control model 991. Hereinafter, the first condition data VTf will also be referred to as the facial feature vector VTf.

[0051] In S140, the image processing server 200 inputs the facial feature vector VTf to the first control model 991. The first control model 991 processes the facial feature vector VTf using an encoder or a zero convolution layer, and inputs the data obtained to the U-net decoder of the diffusion model 960. As a result, the facial features of the biological image CI (e.g., the positions of the eyes, nose, and mouth) are reflected in the output image OI (i.e., the decorated character image).

[0052] The first control model 991 has been trained using facial feature vectors generated using a large number of biological images for training (e.g., photographic images of animal faces such as dogs, cats, and bears) so that the facial features of these biological images are reflected in the output image OI. The specific training method is disclosed in the above-mentioned paper on ControlNet technology.

[0053] In S145, the image processing server 200 (background removal processing unit 214) performs background removal processing on the biological image CI to generate a processed biological image CIr based on the biological image CI. FIG. 4E shows a processed biological image CIr generated using the biological image CI of FIG. 4C. As shown in FIG. 4E, the processed biological image CIr is an image in which the background has been removed from the biological image CI, in other words, an image obtained by painting the background of the biological image CI white. A known method is used for the background removal processing. For example, the background removal processing uses a machine learning model called U2-Net to separate the foreground from the background and remove the background.

[0054] U2-Net is disclosed in the following paper: Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaiane and Martin Jagersand, "U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection", arXiv:2005.09007v3, March 8, 2022, https: / / arxiv.org / abs / 2005.0900 7v3

[0055] In S150, the image processing server 200 generates a color palette image CPI using the processed organism image CIr. Specifically, the image processing server 200 clusters the pixels of the processed organism image CIr, excluding background pixels (pixels filled with white), into M groups (M is an integer equal to or greater than 2). In other words, the image processing server 200 divides the colors contained in the processed organism image CIr, excluding the background color, into M groups. Clustering is performed using a known algorithm, such as the x-means or k-means algorithm, so that pixel values ​​(i.e., pixel colors) similar to each other belong to the same group. M is a predetermined value that is significantly smaller than the number of colors possessed by the organism depicted in the processed organism image CIr. M is, for example, a value between 3 and 10, and in this embodiment, it is 4.

[0056] The image processing server 200 calculates the representative color of each of the M groups and the number of pixels belonging to each of the M groups. The representative color of a group is, for example, a color represented by the average or median value of each RGB component of the pixels belonging to that group. The image processing server 200 generates a color palette image CPI based on the representative colors and the number of pixels of the M groups. FIG. 4(F) shows an example of the color palette image CPI. The color palette image CPI is an image showing the color distribution of the processed biological image CIr. For example, the color palette image CPI includes M partial images CA1-CA4 (M=4 in the example of FIG. 4(F)). The color palette image CPI is an image of a predetermined size. The M partial images CA1-CA4 correspond one-to-one to the M groups described above. Each partial image is a monochromatic image having the representative color of the corresponding group. The areas of the M partial images CA1-CA4 are determined so that the area ratio of the partial images CA1-CA4 matches the ratio of the number of pixels of the corresponding groups. The M partial images CA1-CA4 are arranged in the color palette image CPI in descending order of area.

[0057] In S155, the image processing server 200 performs vector conversion processing on the color palette image CPI. Specifically, the image processing server 200 inputs the feature point image FPI to the second image encoder 972 ( FIG. 2 ) and causes the second image encoder 972 to generate a vector representing the characteristics of the color palette image CPI. The generated vector is also referred to as a color feature vector because it represents the color characteristics of the organism shown in the organism image CI. The generated color feature vector is the second condition data VTc to be input to the second control model 992. Hereinafter, the second condition data VTc will also be referred to as the color feature vector VTc.

[0058] In S160, the image processing server 200 inputs the color feature vector VTc to the second control model 992. The second control model 992 processes the color feature vector VTc using an encoder or a zero convolution layer, and inputs the data obtained to the U-net decoder of the diffusion model 960. As a result, the color features of the biological image CI (e.g., the representative colors and the proportions of each color) are reflected in the output image OI (i.e., the decorated character image).

[0059] The second control model 992 has been trained using color feature vectors generated using a large number of learning biological images (e.g., photographic images of animal faces such as dogs, cats, and bears) so that the color features of these biological images are reflected in the output image OI. The specific training method is the same as that used for training the first control model 991, and is disclosed in the above-mentioned paper on ControlNet technology.

[0060] In S165, the image processing server 200 acquires a text PT (prompt) for decorative characters to be input into the diffusion model 960. In this embodiment, an image of a "dog" is used as the biological image CI. Furthermore, in this embodiment, the English alphabet is used as the characters to be processed. For this purpose, "dog, alphabet" is used as the text PT. For example, the image processing server 200 analyzes the biological image CI acquired in S125 to identify the type of creature (e.g., dog, cat) shown in the biological image CI and identify the English words "e.g., dog, cat" that represent the identification result. For example, a known machine learning model called a classifier may be used to analyze the biological image CI. The image processing server 200 identifies the type of character (e.g., alphabet) to be processed based on the character information acquired in S110 and identifies the English word "alphabet" that represents the identification result. The image processing server 200 acquires a text PT obtained by arranging the identified English words. In a modified example, the English word indicating the type of organism shown in the organism image CI may be input by the user into the terminal device 100 when the organism image CI is uploaded to the image processing server 200 in S125, and may be transmitted from the terminal device 100 to the image processing server 200.

[0061] In S170, the image processing server 200 inputs input information including an input image II ( FIG. 4B ), text PT (e.g., “dog, alphabet”), and a seed value into the image generation model 900 (diffusion model 960) to generate an output image OI. The seed value is a random number that adds variation to the generated image. For example, n different random numbers (n is an integer equal to or greater than 2) are used as the seed value. In the process of generating the output image OI, data output from the two control models 991 and 992 is also input to the diffusion model 960, as described above.

[0062] When there are k characters to be processed, the image processing server 200 generates a total of (k × n) output images OI, i.e., n for each character. The image processing server 200 generates (k × n) output images OI by inputting (k × n) pieces of input information one by one into the image generation model 900. When there are three characters to be processed, "ABC," (3 × n) output images OI are generated. For example, the n pieces of input information for generating n decorated character images of "A" each include an input image II ( FIG. 4(B) ) representing the character "A," text PT, and one of n seed values. The image processing server 200 inputs the n pieces of input information into the image generation model 900 to generate output images OI representing n decorated character "A."

[0063] FIG. 4G illustrates an output image OI1 as an example of the output image OI. This output image OI1 is generated when input information including the input image II of FIG. 4A, the feature point image FPI of FIG. 4D, the color palette image CPI of FIG. 4F, and the text PT "dog, alphabet" is input to the image generation model 900. As described above, generation using img2img can generate an output image OI by modifying the input image II according to the text PT. Therefore, the output image OI1 is an image in which the dog and the letter "A" are fused together, since the input image II showing the letter "A" is modified according to the text PT "dog, alphabet." Then, by inputting the images FPI and CPI showing the face and color features of the dog shown in the biological image CI together, the output image OI1 is an image showing a decorated character in which the letter "A" is decorated with a dog's face. Although not shown, for example, assume that an image of a cat is used as the biological image CI, the alphabet "C" is used as the input image II, and "cat, alphabet" is used as the text PT. In this case, the image generation model 900 can generate an output image OI showing decorated characters in which the alphabet "C" is decorated with a cat's face. In this way, the image generation model 900 can be said to be trained to generate decorated images showing the face of a living creature shown in the biological image CI based on the input information. For example, the image generation model 900 generates decorated images showing the face of a living creature shown in the biological image CI based on images and text. Note that "trained" refers to, for example, adjusting the parameters of the image generation model 900 using decorated images showing the face of a living creature shown in the biological image CI as training data. Furthermore, all or part of the decorated images showing the face of a living creature shown in the biological image CI may be stored in the image generation model 900. Hereinafter, the output image OI showing decorated characters will also be referred to as a decorated character image OI.

[0064] In the decorated character image OI1 of FIG. 4A , the image of the dog is not photo-realistic but rather animated or illustrated. This is because a stable diffusion model (also called a checkpoint) trained to generate animated or illustrated images is used as the diffusion model 960 of the image generation model 900. In the decorated character image OI1, the dog's facial features reflect the facial features of the dog shown in the biological image CI. In other words, the decorated character image OI1 is an image showing decorated characters, which are characters decorated to show the face of the creature shown in the biological image CI. This is because the feature point image FPI generated using the biological image CI is input to the image generation model 900. In the decorated character image OI1, the color of the decorated characters containing the dog reflects the color of the dog shown in the biological image CI. This is because the color palette image CPI generated using the biological image CI is input to the image generation model 900. By reflecting the facial features and color of the dog shown in the biological image CI in the decorative character image OI1, the user can feel that the decorative character shown in the decorative character image OI1 resembles the dog shown in the biological image CI.

[0065] 3 , the image processing server 200 displays a selection screen WI1 including n types of decorative character images OI on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the selection screen WI1 and transmits the generated data to the terminal device 100. The terminal device 100 displays the selection screen WI1 on the display unit 140 using the received data.

[0066] 5 is a diagram showing an example of a selection screen WI1. The selection screen WI1 in FIG. 5 includes n types of generated decorative character images OI (e.g., OI1-OI3), a message MS1, a selection cursor CS, a selection button BT1, and a redo button BT2. The message MS1 is a message that prompts the user to select a desired character from the n types of decorative character images OI.

[0067] In S180, the image processing server 200 receives a selection instruction from the user via the terminal device 100 to select a decorative character image OI to be output. Specifically, the user moves the cursor CS on the selection screen WI1 of FIG. 5 to select a desired decorative character image OI and taps the selection button BT1 to input a selection instruction for the decorative character image OI. When a decorative character image OI is selected, the terminal device 100 acquires the selected decorative character image OI. When a decorative character image OI different from the selected decorative character image OI is selected, the terminal device 100 acquires the selected different decorative character image OI. Upon receiving the selection instruction for the decorative character image OI, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected decorative character image OI, to the image processing server 200. Note that, when the undo button BT2 on the selection screen WI1 is tapped, the decorative character generation process of FIG. 3 is executed from the beginning, for example.

[0068] In S185, the image processing server 200 displays an output screen WI2 including a decorative character image to be output on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the output screen WI2 including the decorative character image OI of the type selected by the selection instruction, and transmits the generated data to the terminal device 100. More specifically, when a certain decorative character image OI is acquired by the terminal device 100, the image processing server 200 generates data for the output screen WI2 using the certain decorative character image OI, and when a decorative character image OI different from the certain decorative character image OI is acquired by the terminal device 100, the image processing server 200 generates data for the output screen WI2 using the different decorative character image OI. In other words, when a certain decorative character image OI is selected by the terminal device 100, the image processing server 200 generates data for the output screen WI2 using the certain decorative character image OI, and when a decorative character image OI different from the certain decorative character image OI is selected, the image processing server 200 generates data for the output screen WI2 using the different decorative character image OI. Then, the terminal device 100 displays the output screen WI2 on the display unit 140 using the received data.

[0069] FIG. 6 is a diagram illustrating an example of an output screen WI2. As shown in FIG. 6, the output screen WI2 includes an output image TI, a print button BT3, and a redo button BT4. The output image TI includes three decorated character images OI (OI1, OI4, and OI5) representing the three characters "ABC" to be processed. The three decorated character images OI are images generated using the same seed value as the seed value used to generate the decorated character image OI1 selected by the user. When the user checks the output image TI and prints it, they tap the print button BT3 on the output screen WI2. When the print button BT3 is tapped, processing proceeds to S190. When the redo button BT4 is tapped on the output screen WI2, the decorated character generation process of FIG. 3 is executed from the beginning, for example.

[0070] In S190, the image processing server 200 generates a print job for printing an output image TI ( FIG. 6 ) including the decorative character image OI. In S195, the image processing server 200 transmits the print job to the printer 300, causing the printer 300 to print the output image TI. Specifically, the image processing server 200 may transmit the print job to the printer 300 via the terminal device 100, or may transmit the print job directly to the printer 300 without going through the terminal device 100. Upon receiving the print job, the printer 300 prints the output image TI including the decorative character image OI on paper in accordance with the print job.

[0071] According to the present embodiment described above, the system 1000 acquires character information TXI for specifying characters and a creature image CI, which is an image of a creature including a face (S110 and S125 in FIG. 3 ). The character information TXI and the creature image CI are acquired based on user input. The system 1000 uses the character information TXI and the creature image CI to input input information to the image generation model 900, thereby causing the image generation model 900 to generate a decorated character image OI (S170 in FIG. 3 ). The decorated character image OI is a character decorated to represent the face of the creature shown in the creature image CI (FIG. 4(G)). The system 1000 displays the decorated character image OI on the display unit 140 of the terminal device 100 (S185 in FIG. 3 ). The system 1000 prints an output image TI, including the decorated character image OI displayed on the display unit 140, on paper (S190 and S195 in FIG. 3 ). As a result, a decorated character image OI can be generated in response to a user's input using the image generation model 900. For example, a decorated character image OI showing decorated characters decorated with the face of a creature shown in the creature image CI can be generated. Furthermore, since the decorated character image OI is displayed and printed, the decorated character image OI in response to the user's input can be provided to the user.

[0072] Furthermore, according to this embodiment, the system 1000 generates an input image II showing characters using the character information TXI (steps S115 and S120 in FIG. 3). The system 1000 generates biological information showing the characteristics of the biological image CI (e.g., a feature point image FPI and a color palette image CPI) (steps S135 and S155 in FIG. 3). Input information including the input image II and the biological information is input to the image generation model 900 (steps S170 in FIGS. 2 and 3). As a result, a decorated character image OI can be generated that reflects the characteristics of the biological image CI (e.g., a dog).

[0073] Furthermore, according to this embodiment, the system 1000 uses the biological image CI to generate, as biological information, a feature point image FPI, which is feature information based on a plurality of feature points Pt on the face of the biological organism shown in the biological image CI (S130 in FIG. 3 ). As a result, information based on the plurality of feature points Pt on the face of the biological organism shown in the biological image CI is input to the image generation model 900, so that a decorative character image OI can be generated that reflects the facial features (e.g., facial features and facial expressions) of the biological organism shown in the biological image CI.

[0074] Furthermore, according to this embodiment, the system 1000 uses the organism image CI to generate, as organism information, a color palette image CPI, which is color information based on the colors of the organism (e.g., a dog) depicted in the organism image CI (S145, S150 in FIG. 3 ). As a result, information based on the colors of the organism depicted in the organism image CI is input to the image generation model 900, so that a decorated character image OI can be generated that reflects the color characteristics of the organism depicted in the organism image CI.

[0075] Furthermore, according to this embodiment, the system 1000 performs background removal processing to remove the background from the biological image CI to generate a processed biological image CIr (S145 in FIG. 3 ), and identifies multiple colors using the processed biological image CIr (S150 in FIG. 3 ). As a result, the influence of the background can be eliminated and the multiple colors of the biological organism depicted in the biological image CI can be appropriately identified. Furthermore, according to this embodiment, the system 1000 identifies M colors by clustering multiple pixels included in the processed biological image CIr into M groups. As a result, a color palette image CPI showing an appropriate number of colors can be generated. For example, when generating a decorative character image OI with an anime or illustration style, specifying a relatively small number of colors M (four colors in this embodiment) allows the colors of the biological organism to be appropriately specified. Furthermore, in the color palette image CPI, the areas of the M partial images CA1-CA4 are determined so that the area ratio of the partial images CA1-CA4 matches the ratio of the pixel numbers of the corresponding groups. As a result, the balance of the color scheme of the creature shown in the decorative character image OI can be made closer to the balance of the color scheme of the creature shown in the creature image CI.

[0076] Furthermore, according to this embodiment, the system 1000 performs a smoothing process on the character image FI based on the character information TXI to generate the input image II (S120 in FIG. 3). When the edges of the characters shown in the input image II are blurred, the volume of the decoration applied to the characters in the generated decorated character image OI is likely to be larger than when the edges are sharp. As a result, a more attractive decorated character image OI can be generated.

[0077] Furthermore, according to this embodiment, the biological image CI is an image stored in a memory (e.g., the non-volatile storage device 124) by a user and is an image specified by the user, so that a decorative character image OI can be generated according to the user's preferences.

[0078] Furthermore, according to this embodiment, the character information TXI includes font information indicating the type of font. The shape of the characters displayed in the input image II has the shape of the font indicated by the font information. As a result, the font information can be used to specify the font of the characters displayed in the input image II, thereby increasing the variety of the decorative character images OI that can be generated.

[0079] Furthermore, according to this embodiment, the system 1000 generates, for example, n types of input information using one piece of character information TXI and one piece of biological image CI, and generates n types of decorated character images OI by inputting the n types of input information one by one into the image generation model 900 (S170 in FIG. 3 , OI1-OI3 in FIG. 5 ). The system 1000 displays the n types of decorated character images OI on the display unit 140 (S175 in FIG. 3 , FIG. 5 ). The system 1000 prints an image selected by the user from the n types of decorated character images OI (S190, S195 in FIG. 3 ). As a result, the decorated character image OI selected by the user from the n types of decorated character images OI generated using one piece of character information TXI and one piece of biological image CI is printed, making it possible to print a decorated character image OI that is more in line with the user's preferences.

[0080] As can be seen from the above explanation, the feature point image FPI and the color palette image CPI of this embodiment are examples of biological information.

[0081] B. Second Example In the second example, the content of the decorative character generation process differs from that in the first example. In the second example, as shown in FIG. 1 , in addition to the same information as in the first example, correspondence information 710 and a biological image group CIg are pre-stored in the non-volatile storage device 224 of the image processing server 200. The biological image group CIg includes data on a plurality of biological images CI, including, for example, biological images CI1-CI3, which will be described later. In the decorative character generation process of the second example, a biological image CI selected from the biological image group CIg of the image processing server 200 is used, and therefore, biological image CI data does not need to be stored in the non-volatile storage device 124 of the terminal device 100. The correspondence information 710 will be described later.

[0082] 7 is a flowchart of the decorated character generation process of the second embodiment. In S210, the image processing server 200 displays a sample selection screen WI3 including a plurality of sample decorated character images SI on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the sample selection screen WI3 and transmits the generated data to the terminal device 100. The terminal device 100 displays the sample selection screen WI3 on ​​the display unit 140 using the received data.

[0083] 8A and 8B are explanatory diagrams of the second embodiment. Fig. 8A shows an example of a sample selection screen WI3. This sample selection screen WI3 includes a plurality of sample decorative character images SI (e.g., SI1-SI3), a message MS3, a selection cursor CS, and selection buttons BTs. The message MS3 prompts the user to select a desired character from the plurality of sample decorative character images SI.

[0084] Each of the sample decorated character images SI corresponds one-to-one to one of the biological images CI included in the biological image group CIg. For example, the sample decorated character image SI1 in Fig. 8A corresponds to the biological image CI1 in Fig. 8B. The sample decorated character image SI2 corresponds to the biological image CI2 in Fig. 8B. The sample decorated character image SI3 corresponds to the biological image CI3 in Fig. 8B.

[0085] Furthermore, each of the sample decorated character images SI corresponds to a text PT. For example, the sample decorated character images SI1 and SI2 correspond to the text PT "dog, alphabet." The sample decorated character image SI3 corresponds to the text PT "bear, alphabet."

[0086] In the second embodiment, data for each sample decorated character image SI (SI1-SI3) is generated in advance using the corresponding biological image CI and text PT and stored in the non-volatile storage device 224 of the image processing server 200. For example, a smoothing process is performed on a character image FI of a specific character in a specific font ("A" or "B" in the example of FIG. 8A) to generate an input image II. Furthermore, the processes of S130-S160 and S170 in FIG. 3 are performed using the generated input image II, the corresponding biological image CI, and the corresponding text PT to generate a sample decorated character image SI.

[0087] For this reason, the sample decorated character image SI reflects the facial and color characteristics of the creatures shown in the corresponding creature images CI. For example, the facial and color characteristics of the creatures constituting the decorated characters in sample decorated character image SI1 are similar to the facial and color characteristics of the corresponding creature image CI1. Similarly, the facial and color characteristics of the creatures constituting the decorated characters in sample decorated character images SI2 and SI3 are similar to the facial and color characteristics of the corresponding creature images CI2 and CI3, respectively.

[0088] In S220, the image processing server 200 selects one sample decorated character based on a user's selection instruction. Specifically, the user moves the cursor CS on the sample selection screen WI3 of FIG. 8A to select a desired sample decorated character image SI and taps the selection button BTs to input a selection instruction for the sample decorated character image SI. Upon receiving the selection instruction for the sample decorated character image SI, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected sample decorated character image SI, to the image processing server 200. The image processing server 200 selects the sample decorated character image SI selected by the user in accordance with the received information.

[0089] In S230, the image processing server 200 acquires the biological image CI and text PT corresponding to the selected sample decorated character image SI. Specifically, the correspondence information 710 is information that records each sample decorated character image SI in association with the biological image CI and text PT used to generate the sample decorated character image SI. The image processing server 200 references the correspondence information 710 to acquire the biological image CI and text PT used to generate the selected sample decorated character image SI.

[0090] In S240, the image processing server 200 executes the processes of S110-S120 in Fig. 3. That is, the image processing server 200 acquires character information TXI from the user via the terminal device 100, and generates an input image II (e.g., Fig. 4B) using the character information TXI.

[0091] In S250, the image processing server 200 executes the processes of S130-160 in Fig. 3. That is, the image processing server 200 generates a face feature vector VTf and a color feature vector VTc using the biological image CI acquired in S230, and inputs them to the control models 991 and 992, respectively.

[0092] In S260, the image processing server 200 executes the processes of S170 to S195 in Fig. 3. That is, the image processing server 200 inputs input information including the input image II and the text PT into the image generation model 900 (diffusion model 960), generates a decorated character image OI, and displays or prints the generated decorated character image OI.

[0093] According to the second embodiment described above, the system 1000 selects one sample decorated character image SI from among a plurality of sample decorated character images SI, including a first decorated character image SI1 generated by inputting input information generated using a first biological image CI1 into the image generation model 900 and a second decorated character image SI2 generated by inputting input information generated using a second biological image CI2 into the image generation model 900, based on a user instruction (S220 in FIG. 7 ). When the first decorated character image SI1 is selected, the image processing server 200 acquires the first biological image CI1 as the biological image to be used, and when the second decorated character image SI2 is selected, the image processing server 200 acquires the second biological image CI2 as the biological image to be used (S230 in FIG. 7 ). When the first decorative character image SI1 is selected, the system 1000 generates input information using the first biological image CI1, and when the second decorative character image SI2 is selected, the system 1000 generates input information using the second biological image CI2, and inputs the generated input information into the image generation model 900 (S260 in Figure 7).

[0094] As a result, it is possible to generate and print a decorative character image OI that meets the user's preferences. For example, even when a user looks at the biological images CI (CI1-CI3), the user may be unable to visualize the decorative character image OI that will be generated. In this case, even if the user is prompted to determine the decorative character image OI to be used, the user may be unable to determine an appropriate biological image CI that will enable the user to generate the desired decorative character image. In this embodiment, the user simply selects a sample decorative character image SI of the desired style from among multiple sample decorative character images SI generated using each of the multiple biological images CI. This allows the user to display and print a decorative character image OI of the desired style for the desired characters. Therefore, it is possible to easily display and print a decorative character image OI that meets the user's intentions.

[0095] C. Modifications (1) The image generation model 900 (FIG. 2) in each of the above embodiments is an example and is not limited to this. The image generation model may be modified as appropriate. FIG. 9 is a block diagram showing an example of a modified image generation model 900B. In the modification of FIG. 9, a third control model 993 and a third image encoder 973 are added to the configuration of the image generation model 900 in FIG. 2.

[0096] The third control model 993, like the control models 991 and 992, is a model based on a conditional image generation technology called ControlNet, and has a similar configuration to the control models 991 and 992. The third control model 993 receives third condition data VTe indicating additional conditions to be applied to the diffusion model 960. In the third control model 993, the additional conditions are conditions on the shape (outline) of the character image FI.

[0097] When a condition image TEI indicating an additional condition is input, the third image encoder 973 generates and outputs the above-mentioned third condition data VTe by converting the condition image TEI into a data format that can be input to the third control model 993. The third condition data VTe is a vector with a predetermined number of dimensions, similar to the condition data VTf and VTc. The third image encoder 973, like the image encoders 971 and 972, is an image encoder that has been pre-trained using a technology called CLIP.

[0098] As shown in FIG. 9 , the condition image TEI input to the third image encoder 973 is generated by the edge extraction unit 216. The edge extraction unit 216 is a functional unit implemented by the processor 210 of the image processing server 200 executing the computer program of the second embodiment. The edge extraction unit 216 performs edge extraction processing on the character image FI to generate a character outline image as the condition image TEI. Hereinafter, the condition image TEI input to the third image encoder 973 will also be referred to as the character outline image TEI. A known edge extraction processing, such as the "findContours" function of the OpenCV (Open Source Computer Vision Library), can be used for the edge extraction processing. FIG. 10 is a diagram showing an example of the character outline image TEI. FIG. 10 shows the character outline image TEI indicating the outline OL of the character image FI in FIG. 4A.

[0099] As can be seen from the above explanation, the input information input to the image generation model 900B of the modified example includes the text PT, the input image II, the feature point image FPI, the color palette image CPI, and also the character outline image TEI. The character outline image TEI is information different from the input image II showing the characters, and can be said to be shape information showing the shapes of the characters shown in the input image II.

[0100] That is, in the decorative character generation process of the modified example, the following process is executed in addition to the process of FIG. 3 . The image processing server 200 executes edge extraction processing on the character image FI to generate a character outline image TEI, and inputs the character outline image TEI to the image generation model 900B. That is, the character outline image TEI is input to the third image encoder 973 ( FIG. 9 ) of the image generation model 900B. The third image encoder 973 generates a vector indicating the characteristics of the character outline image TEI. The generated vector is also referred to as a character shape vector because it indicates the characteristics of the character outline image TEI, i.e., the characteristics of the shape of the character (e.g., "A") included in the input image II. The generated character shape vector is third condition data VTe to be input to the third control model 993. Hereinafter, the third condition data VTe is also referred to as a character shape vector VTe. The character shape vector VTe is input to the third control model 993. The third control model 993 inputs data obtained by processing the character shape vector VTe using an encoder or a zero-convolution layer to the U-net decoder of the diffusion model 960. As a result, the shape of the character (for example, the outline of the letter "A") is reflected in the output image OI (i.e., the decorated character image).

[0101] The third control model 993 has been trained so that image features indicating edges such as contours are reflected in the output image O1. For the third control model 993, for example, a model called "Canny" by ControlNet is adopted.

[0102] Thus, in this modification, the input information input to the image generation model 900B is information different from the input image II and includes a character outline image TEI, which is shape information indicating the shape of the character. As a result, shape information indicating the shape of the character is input to the image generation model 900B separately from the input image II, making it possible to control the shape of the character in the generated decorated character image OI. For example, when executing the decorated character generation process of each of the above embodiments to generate a decorated character image OI, depending on the type of character, font type, and biological image CI used, the shape of the character may be distorted in the decorated character image OI, reducing the readability of the character. In such cases, adding a character outline image TEI to the input information, as in the modification, allows the shape of the character to be maintained in the generated decorated character image OI, thereby improving the readability of the character.

[0103] (2) In the above modification, a model called "Canny" from ControlNet is adopted as the third control model 993. Alternatively, a model called "depth" from ControlNet may be adopted as the third control model 993. In this case, the image processing server 200 inputs the character image FI to a depth estimation model to generate an estimated depth image indicating the estimated depth of the character image FI for each pixel. The image processing server 200 inputs the estimated depth image to the third image encoder 973 of the image generation model 900B. Even in this case, the estimated depth image is input to the image generation model 900B as shape information indicating the shape of the character, separate from the input image II. Therefore, in this case as well, the shape of the character in the generated decorated character image OI can be controlled.

[0104] (3) As shown in FIG. 1 , the system 1000 may include a sewing machine 400 and a cutting machine 500. The sewing machine 400 is a device that forms an image on fabric by embroidering the fabric with thread according to embroidery data. The cutting machine 500 is a device that expresses an image in the shape of a sheet-like material by cutting the sheet-like material with a blade according to cut data. The sheet-like material may be, for example, paper, cloth, a resin sheet such as urethane foam, or leather.

[0105] For example, in the above embodiment, the system 1000 causes the printer 300 to print the output image TI including the decorative character image OI. Alternatively, the system 1000 may cause the sewing machine 400 to embroider the output image TI. The system 1000 may also cause the cutting machine 500 to cut the output image TI. Alternatively, the system 1000 may express the output image TI using a combination of these devices. For example, the system 1000 may use the printer 300 to print the output image TI on a sheet material, and then use the cutting machine 500 to cut the sheet material on which the output image TI has been printed along the outlines of the decorative characters included in the output image TI.

[0106] (4) In the first embodiment, the image processing server 200 acquires the biological image CI selected by the user from the terminal device 100 (S125 in FIG. 3 ). Alternatively, a plurality of biological image CI prepared by the administrator of the image processing server 200 may be stored in the non-volatile storage device 224 of the image processing server 200. In this case, the image processing server 200 may acquire the biological image CI to be used from the plurality of biological image CI stored in its own non-volatile storage device 224, in response to a selection instruction from the user, or automatically.

[0107] (5) In the second embodiment, the input information associated with the sample decorated character image SI in the correspondence information 710 is the biological image CI and the text PT. The input information associated with the sample decorated character image SI may include other information. The other information may include, for example, a seed value or a parameter (also called a control weight) indicating the strength with which the data output from the control models 991 and 992 influences the decorated character image OI.

[0108] (6) The diffusion model 960 of the image generation model 900 in each of the above embodiments is a machine learning model called stable diffusion, but is not limited to this. Instead of the diffusion model 960, various models trained to generate an image decorated with decorations representing the face of a creature shown in a biological image when input information including information based on the biological image showing the face of the creature may be used for the image generation model 900. For example, when a style image and a content image are input, the image generation model 900 may employ an image generation model (e.g., a style transfer model called multimodal style transfer (MST)) that generates an image in which the style of the style image is applied to the content image. In this case, for example, by inputting the input image II of this embodiment as the content image and the biological image CI of this embodiment or the sample decorated character image SI as style images into the MST model, a decorated character image in which decorations are applied to the characters shown in the input image II can be generated. Even in this case, data based on the feature point image FPI and the condition image CPI generated using the organism image CI is input to the MST model via the control models 991, 992 and the image encoders 971, 972. This allows the facial features and color features of the organism shown in the organism image CI to be reflected in the decorative character image generated by the MST model. Furthermore, other well-known image generation models, such as the Adain model, the StyTR2 model, and the U-GAT-IT model, may also be employed, in addition to the MST model.

[0109] Furthermore, instead of the diffusion model 960, a diffusion model different from Stable Diffusion, such as a diffusion model called "VQ-Diffusion" or "GLIDE", may be adopted.

[0110] "VQ-Diffusion" is disclosed in the following paper: Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo, "Vector Quantized Diffusion Model for Text-to-Image Synthesis", arXiv:2111.14822v3, March 3, 2022, https: / / arxiv.org / abs / 2111.14822v3

[0111] "GLIDE" is disclosed in the following paper: Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen, "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models", arXiv:2112.10741v3, March 8, 2022, https: / / arxiv.org / abs / 2112.10741v3

[0112] (7) The control models 991 and 992 are models that use a conditional image generation technology called "controlnet," but are not limited to this and other models may be used. For example, a model that uses a conditional image generation technology called "IP-Adapter" may be used.

[0113] "IP-Adapter" is disclosed in the following paper: Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, Wei Yang, "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models", arXiv:2308.06721v1, August 13, 2023, https: / / arxiv.org / abs / 2308.06721v1

[0114] (8) The image generation model 900 in each of the above embodiments includes two image encoders 971, 972 and two control models 991, 992. Alternatively, the image generation model may include the first image encoder 971 and the first control model 991, but not the second image encoder 972 or the second control model 992. In this case, steps S145 to S160 in FIG. 3 may be omitted.

[0115] Furthermore, the image generation model may include the second image encoder 972 and the second control model 992, but may not include the first image encoder 971 and the first control model 991. In this case, S130-S140 in FIG. 3 may be omitted.

[0116] (9) The decorative character generation process in each of the above embodiments is merely an example and may be modified as appropriate. For example, the character information acquired in S110 of FIG. 3 may not include font information indicating the type of font specified by the user. In this case, for example, the character image FI generated in S115 of FIG. 3 is generated using a predetermined font. Furthermore, the smoothing process in S120 may be omitted.

[0117] 3, n types of decorated character images OI with different seed values ​​are generated for each character to be processed. Alternatively, only one type of decorated character image OI may be generated for each character to be processed. In this case, steps S175 and S180 in FIG. 3 may be omitted.

[0118] 3, an input image II is generated for each character to be processed, and a decorated character image OI is generated for each character. Alternatively, the image processing server 200 may generate an input image including multiple characters and input input information including the input image to the image generation model 900, thereby generating a single decorated character image including multiple decorated characters.

[0119] (10) In the system 1000 of each of the above embodiments, the terminal device 100 mainly executes processing related to a screen that functions as a user interface (referred to as a UI screen) and processing for controlling the printer 300. Alternatively, the terminal device 100 may execute part of the image processing executed by the image processing server 200 on behalf of the image processing server 200.

[0120] For example, the terminal device 100 may execute the process of generating the input image II (S110-S120), the process of generating the feature point image FPI (S125, S130), the process of generating the color palette image CPI (S125-S150), and the process related to displaying and printing the decorated character image OI (S175-S195) in the decorated character generation process of Fig. 3. The image processing server 200 may execute only the process of generating the decorated character image OI using the image generation model 900 (S135, S140, S155, S160, S170). The same applies to the decorated character generation process of Fig. 7.

[0121] 3 may be executed by the terminal device 100. In this case, the system 1000 may be configured with only the terminal device 100 and the printer 300, and the image processing server 200 may be omitted.

[0122] (11) In each of the above embodiments, the decorative character image OI is generated by the user of the terminal device 100 in order to print the decorative character image OI. However, the generation of the decorative character image OI may also be performed by, for example, the manufacturer of the printer 300, the sewing machine 400, or the cutting machine 500. In this case, for example, a computer (terminal or server) owned by the manufacturer executes the decorative character generation process shown in FIGS. 3 and 7 . In this case, for example, decorative character images OI are generated for all alphabets, and data for the decorative character images OI is stored in the memory of the printer 300, the sewing machine 400, or the cutting machine 500 during manufacture of the printer 300, the sewing machine 400, or the cutting machine 500. The printer 300, the sewing machine 400, or the cutting machine 500 uses the data for the decorative character image OI to print, embroider, or cut the decorative character image OI in response to, for example, a user's operation.

[0123] (12) In the above embodiments and modifications, the processor 210 may cause the GPU 260 to execute various operations. Note that the GPU 260 may be omitted.

[0124] (11) The processing performed by the terminal device 100 in FIG. 1 may be performed by an image processing device such as a digital camera, a printer, or a multifunction peripheral.

[0125] In each of the above embodiments, a part of the hardware configuration may be replaced with software, and conversely, a part or all of the software configuration may be replaced with hardware. For example, the processing by the image generation model 900 (FIG. 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0126] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and DVD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0127] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0128] 1000...system, 100...terminal device, 110...processor, 120...storage device, 140...display unit, 150...operation unit, 170...communication interface, 200...image processing server, 210...processor, 211...character image generation unit, 212...smoothing processing unit, 213...feature point extraction unit, 214...background removal processing unit, 215...color reduction processing unit, 216...edge extraction unit, 220...volatile storage device, 260...graphics processing unit, 270...communication interface, 300...processor Linter, 400...sewing machine, 500...cutting machine, 710...correspondence information, 900, 900B...image generation model, 910...text encoder, 920...image encoder, 930...latent variable model, 940...image decoder, 960...diffusion model, 971, 972, 973...image encoder, 991, 991, 993...control model, CI...biological image, CIr...processed biological image, CPI...color palette image, FI...character image, FPI...feature point image green, II...input image ,IT...Internet, LN...Local Area Network, OI...Decorative character image, PGs, PGt...Computer program, PT...Text, SI...Sample decorative character image, TEI...Character outline image, TI...Output image, TXI...Character information, VTc...Color feature vector, VTe...Character shape vector, VTf...Facial feature vector, WI1...Selection screen, WI2...Output screen, WI3...Sample selection screen

Claims

1. An image processing system comprising: an information acquisition unit that acquires character information for specifying characters and a biological image that is an image of a biological organism including a face, wherein at least a portion of the character information and the biological image are acquired based on user input; an image generation unit that uses the character information and the biological image to cause an image generation model to generate a decorated character image by inputting input information to the image generation model, wherein the decorated character image is an image showing decorated characters that are characters that are decorated to show the face of the biological organism shown in the biological image, and the image generation model has been trained to generate a decorated image showing the face of the biological organism shown in the biological image based on the input information; a display control unit that displays the decorated character image on a display unit; and an image forming unit that prints, embroiders, or cuts the decorated character image displayed on the display unit onto an object.

2. An image processing system according to claim 1, wherein the image generation unit uses the character information to generate an input image showing characters, uses the biological image to generate biological information showing characteristics of the biological image, and inputs input information including the input image and the biological information into the image generation model.

3. An image processing system as described in claim 2, wherein the biological information includes feature information based on a plurality of feature points on the face of the organism shown in the biological image, and the image generation unit uses the biological image to generate feature information based on a plurality of feature points on the face of the organism shown in the biological image.

4. An image processing system as described in claim 2, wherein the biological information includes color information based on multiple colors possessed by the organism shown in the biological image, and the image generation unit uses the biological image to generate color information based on the multiple colors possessed by the organism shown in the biological image.

5. An image processing system as described in claim 4, wherein the image generation unit generates a processed organism image based on the organism image by performing background removal processing to remove the background from the organism image, and separates the colors contained in the processed organism image into the multiple colors.

6. An image processing system according to claim 2, wherein the image generation unit generates a character image based on the character information, and generates the input image based on the character image by performing processing including a smoothing process on the character image.

7. An image processing system according to claim 1, wherein the biological image is an image stored in a memory by a user and specified by the user.

8. An image processing system according to claim 1, wherein the character information includes font information indicating a font type, and the characters shown in the input image have the shape of the font indicated by the font information.

9. An image processing system according to claim 1, further comprising: a sample selection unit that selects one decorated character image based on a user's instruction from a plurality of sample decorated character images including a first decorated character image generated by inputting the input information generated using a first biological image into the image generation model, and a second decorated character image generated by inputting the input information generated using a second biological image into the image generation model; wherein the information acquisition unit acquires the first biological image as the biological image when the first decorated character image is selected, and acquires the second biological image as the biological image when the second decorated character image is selected; and wherein the image generation unit generates the input information using the first biological image as the biological image when the first decorated character image is selected, and generates the input information using the second biological image as the biological image when the second decorated character image is selected.

10. An image processing system as described in claim 1, wherein the image generation unit generates n types (n is an integer of 2 or more) of input information using one piece of character information and one piece of biological image, and generates n types of decorative character images by inputting the n types of input information one by one into the image generation model, the display control unit displays the n types of decorative character images on the display unit, and the image forming unit prints, embroiders, or cuts an image selected by a user from the n types of decorative character images onto the object.

11. An image processing system according to claim 1, wherein the input information is information different from the input image showing characters and includes shape information showing the shape of the characters.

12. A method for generating a decorated character image, comprising: a first step of generating feature information indicating a plurality of feature points of the face of the organism shown in a biological image, which is an image of the organism including a face; and a second step of causing an image generation model to generate a decorated character image by inputting input information including an input image indicating characters and the feature information into the image generation model, wherein the decorated character image is an image showing decorated characters that are decorated to indicate the face of the organism shown in the biological image, and the image generation model has been trained to generate a decorated image that is decorated to indicate the face of the organism shown in the biological image based on the input information.

13. A method for generating a decorated character image, comprising: a first step of using a creature image, which is an image of a creature including a face, to generate color information based on multiple colors of the creature shown in the creature image; and a second step of causing an image generation model to generate a decorated character image by inputting input information, including an input image showing characters and the color information, into the image generation model, wherein the decorated character image is an image showing decorated characters decorated to show the face of the creature shown in the creature image, and the image generation model has been trained to generate a decorated image showing the face of the creature shown in the creature image based on the input information.

14. A generation method according to claim 12 or 13, further comprising the step of generating a processed character image based on a character image by performing processing including a smoothing process on the character image, wherein the input image is the processed character image.

15. A generation method according to claim 12 or 13, further comprising the step of acquiring character information including font information indicating the type of font, wherein the characters shown in the input image have the shape of the font indicated by the font information.

16. A generation method as described in claim 13, wherein the first step includes a step of generating a processed biological image based on the biological image by performing background removal processing to remove the background from the biological image, and a step of separating the colors contained in the processed biological image into the plurality of colors by performing color reduction processing on the processed biological image to reduce the number of colors.

Images