Image processing system and method for generating decorative character images

Abstract

We generate images of characters using a machine learning model. [Solution] The image processing system comprises an information acquisition unit that acquires character information for specifying characters and a biological image which is an image of a living organism including a face, and an image generation unit that uses the character information and the biological image to input the input information to an image generation model and causes the image generation model to generate a decorated character image. The decorated character image is an image showing a decorated character which is a character that has been decorated to represent the face of the living organism shown in the biological image. The image generation model is trained to generate an image that has been decorated to represent the face of the living organism shown in the biological image based on the input information.

Description

Technical Field

[0001] This specification relates to a technique for generating an image representing characters.

Background Art

[0002] Patent Document 1 discloses a technique for training a neural network by using a group of prototype characters as input patterns and a group of deformed characters as teacher patterns. In this technique, when an unlearned character is input to the neural network, a deformed character obtained by deforming the unlearned character according to deformation rules is output.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Thus, a technique for generating an image of a character is required.

[0005] This specification discloses a technique for generating an image of a character by using a machine learning model.

Means for Solving the Problems

[0006] The technique disclosed in this specification can be realized as the following application examples.

[0007] [Application Example 1] An image processing system comprising: an information acquisition unit that acquires character information for specifying characters and a biological image which is an image of a living organism including a face, wherein at least a portion of the character information and the biological image is acquired based on user input; an image generation unit that uses the character information and the biological image to input input information to an image generation model and causes the image generation model to generate a decorative character image, wherein the decorative character image is an image showing a decorative character which is a character which is decorated to show the face of a living organism shown in the biological image, and the image generation model is trained to generate an image which is decorated to show the face of a living organism shown in the biological image based on the input information; a display control unit that displays the decorative character image on a display unit; and an image forming unit that prints, embroiders, or cuts the decorative character image displayed on the display unit onto an object.

[0008] According to the above configuration, a decorative character image is generated by inputting the input information, including character information and biological images obtained based on user input, into an image generation model. As a result, a decorative character image corresponding to user input can be generated using a machine learning model. Furthermore, the decorative character image is displayed and then printed, embroidered, or cut, so that the user can be provided with a decorative character image that corresponds to their input.

[0009] [Application Example 2] A method for generating a decorative character image, comprising: a first step of generating feature information based on a plurality of feature points of the face of a living organism shown in a biological image, which is an image of a living organism including a face; and a second step of causing an image generation model to generate a decorative character image by inputting input information including an input image showing characters and the feature information into the image generation model, wherein the decorative character image is an image showing decorative characters with decorations showing the face of a living organism shown in the biological image, and the image generation model is trained to generate an image with decorations showing the face of a living organism shown in the biological image based on the input information.

[0010] According to the above configuration, a decorative character image is generated by inputting input information, including an input image showing characters and characteristic information of the organism's face generated using the organism's image, into an image generation model. As a result, a machine learning model can be used to generate a decorative character image that reflects the facial features of the organism shown in the organism's image.

[0011] [Application Example 3] A method for generating a decorative character image, comprising: a first step of generating color information based on multiple colors possessed by an organism shown in a biological image, which is an image of an organism including a face; and a second step of causing an image generation model to generate a decorative character image by inputting input information including an input image showing characters and the color information into the image generation model, wherein the decorative character image is an image showing decorative characters with decorations showing the face of the organism shown in the biological image, and the image generation model is trained to generate an image with decorations showing the face of the organism shown in the biological image based on the input information.

[0012] According to the above configuration, a decorative character image is generated by inputting input information, including an input image showing characters and color information of the organism generated using the organism image, into an image generation model. As a result, a machine learning model can be used to generate a decorative character image that reflects the color characteristics of the organism shown in the organism image.

[0013] Furthermore, the technologies disclosed herein can be implemented in various forms, for example, in the form of an image processing system, a method for generating decorative character images, and an image processing apparatus, a computer program for realizing the functions of such systems, methods, or apparatuses, and a recording medium (for example, a non-temporary recording medium) on which the computer program is recorded. [Brief explanation of the drawing]

[0014] [Figure 1] A block diagram showing the configuration of system 1000 in the first embodiment. [Figure 2]Block diagram showing an example of an image generation model 900. [Figure 3] Flowchart of the decorative character generation process of the first embodiment. [Figure 4] Diagram showing an example of an image used in the decorative character generation process. [Figure 5] Diagram showing an example of a selection screen WI1. [Figure 6] Diagram showing an example of an output screen WI2. [Figure 7] Flowchart of the decorative character generation process of the second embodiment. [Figure 8] Explanatory diagram of the second embodiment. [Figure 9] Block diagram showing an example of a modified image generation model 900B. [Figure 10] Diagram showing an example of a character outline image TEI.

Mode for Carrying Out the Invention

[0015] A. First Embodiment: A1. Configuration of the System 1000 FIG. 1 is a block diagram showing the configuration of the system 1000 of the first embodiment. The system 1000 includes a terminal device 100, an image processing server 200, and a printer 300. The terminal device 100 and the printer 300 are connected to a local area network LN and are communicably connected to each other. The local area network LN is connected to the Internet IT, and the image processing server 200 is connected to the Internet IT. Therefore, the terminal device 100 and the image processing server 200 are communicably connected via the local area network LN and the Internet IT. In a modified example, as will be described later, instead of the printer 300, a sewing machine 400 or a cutting machine 500 may be provided.

[0016] The image processing server 200 is, for example, a computer operated by a business operator who manufactures and sells the printer 300, for example, a cloud server. [[ID=​The image processing server 200 includes a processor 210, a storage device 220, a graphics processing unit 260 (referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 220 includes a volatile storage device 222 and a non-volatile storage device 224.

[0018] The processor 210 is a device configured to perform data processing, for example, a Central Processing Unit (CPU). The volatile storage device 222 is, for example, a Dynamic Random Access Memory (DRAM), and the non-volatile storage device 224 is, for example, a flash memory. The non-volatile storage device 224 stores the data of each of the computer program PGs and the image generation model 900. The image generation model 900 is a program module that forms a trained machine learning model.

[0019] The computer program PGs and the image generation model 900 are provided in a form that is uploaded, for example, by an operator who operates the image processing server 200. The processor 210 executes the computer program PGs to cooperate with the terminal device 100 to execute the decorative character generation process described later.

[0020] The GPU 260 is an arithmetic device configured to execute various numerical operations such as image processing and machine learning. The GPU 260 executes various operations according to the instructions of the processor 210. In this embodiment, the GPU 260 is used to execute the operations of the image generation model 900. The communication interface 270 is an interface for connecting to the Internet IT, and is, for example, a wired interface compliant with Ethernet (registered trademark).

[0021] The terminal device 100 is a computer, for example, a mobile terminal such as a smartphone or a tablet computer. In a modified example, the terminal device 100 may be a personal computer.

[0022] The terminal device 100 comprises a processor 110, a storage device 120, a display unit 140, an operation unit 150, and a communication interface 170. These elements are connected to each other via a bus. The storage device 120 includes a volatile storage device 122 and a non-volatile storage device 124.

[0023] The processor 110 is a device configured to perform data processing, such as a CPU. The volatile memory device 122 is, for example, DRAM, and the non-volatile memory device 124 is, for example, flash memory. The non-volatile memory device 124 stores data of the computer program PGt and the biological image CI.

[0024] The computer program PGt is an application program provided, for example, by being downloaded from a server connected via the Internet (IT). The processor 110 functions as an application that executes the computer program PGt and performs the decorative character generation process described below in cooperation with the image processing server 200.

[0025] A biological image CI is an image representing a living organism, prepared, for example, by the user of the terminal device 100 and stored in the non-volatile storage device 224. For example, a biological image CI is a photographic image generated when the user takes a picture of a living organism using the camera function of the terminal device 100. Alternatively, a biological image CI may be an image downloaded from a free stock image site by the user operating a browser. The biological image CI is used in the decorative character generation process, as will be described later.

[0026] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive user input, such as buttons or a touch panel superimposed on the display unit 140. In this embodiment, the display unit 140 and the operation unit 150 form a so-called touchscreen. The user can input various instructions to the image processing server 200 by operating the operation unit 150. The display unit 140 may display elements for operation (e.g., buttons, sliders, etc.), and the displayed elements may be operated through the operation of the operation unit 150.

[0027] The printer 300 is a device that prints images onto a substrate, such as paper, using colorants such as ink or toner.

[0028] A2. Image generation model 900 Figure 2 is a block diagram representing an example of an image generation model 900. The image generation model 900 may be various models that use data from an input image to generate data for an output image based on the input image. In this embodiment, the image generation model 900 includes a diffusion model 960, control models 991 and 992, and image encoders 971 and 972.

[0029] The diffusion model 960 in this embodiment is a machine learning model called Stable Diffusion. Stable Diffusion is a model that synthesizes high-resolution images using a latent diffusion model. The technique for synthesizing high-resolution images using a latent diffusion model is disclosed, for example, in the following paper. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjoern Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models", arXiv:2112.10752, April 13, 2022, http: / / arxiv.org / abs / 2112.10752

[0030] Data for a pre-trained Stable Diffusion model is publicly available on the internet by Stability AI. In this embodiment, the publicly available pre-trained model data is used as the data for the image generation model 900. The image generation model 900 includes a text encoder 910, an image encoder 920, a latent variable model 930, and an image decoder 940. The text encoder 910 is configured to convert text PT into vector tv (such vector tv is also called text embedding). The image encoder 920 is configured to convert the input image II into a latent variable lvi. The latent variable model 930 is configured to perform a noise addition process to add noise to the latent variable lvi, and a despreading process to remove noise from the noise-containing latent variable, thereby outputting a processed latent variable lvo. The latent variable model 930 includes a neural network called U-Net for performing the despreading process (not shown). The image decoder 940 is configured to generate an output image OI using the latent variable lvo. The latent variable model 930 uses the vector tv obtained from the text encoder 910 as conditioning during the despreading process. By performing this despreading process, the latent variable model 930 can generate a latent variable lvo that corresponds to an image conditioned by the text PT. The text encoder 910 is pre-trained so that the vector tv obtained from the text PT corresponds to the image represented by the text PT. As the text encoder 910, a pre-trained text encoder is used, which is a technique called CLIP (Contrastive Language-Image Pre-training). CLIP is a technique released by OpenAI.

[0031] The image generation model 900 can generate a new output image OI represented by text PT by performing a despreading process using randomly generated noise with vector tv from the text encoder 910. Such a technique is also called txt2img. The image generation model 900 can also generate an output image OI by using the input image II and text PT to modify the input image II according to text PT. Such a technique is also called img2img. In this embodiment, the img2img technique is used to generate a decorated character image.

[0032] As shown in Figure 2, the input image II is generated by the character image generation unit 211 and the smoothing processing unit 212. The character image generation unit 211 and the smoothing processing unit 212 are functional units realized by the processor 210 of the image processing server 200 executing computer programs PGs. The character image generation unit 211 generates a character image FI using character information TXI. The smoothing processing unit 212 performs a smoothing process on the character image FI to generate an input image II that shows the characters.

[0033] Control models 991 and 992 use a conditional image generation technique called ControlNet to provide additional conditions to the diffusion model 960 so that the features of the biological image CI are reflected in the output image OI. Control models 991 and 992 are CNNs (Convolutional Neural Networks) that include a copy of the U-net encoder of the diffusion model 960 and a zero convolutional layer. The zero convolutional layer is a convolutional layer that is trained with the initial values ​​of its parameters (weights and biases) set to zero. The U-net encoders of control models 991 and 992 are input with the same data as the U-net encoder of the diffusion model 960 (not shown), as well as condition data VTf and VTc indicating additional conditions to be given to the diffusion model 960. In the first control model 991, the additional condition is the facial expression (facial structure) of the organism shown in the biological image CI. In the second control model 992, the additional condition is the color of the organism shown in the biological image CI.

[0034] The data output from each block of the encoders in control models 991 and 992 is processed by the zero convolutional layer corresponding to each block and input to the corresponding block of the U-net decoder in diffusion model 960. This adds an additional condition to diffusion model 960, and an output image OI is generated according to this additional condition. Such control model 991 and 992 techniques (ControlNet techniques) are disclosed, for example, in the following paper. Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, "Adding Conditional Control to Text-to-Image Diffusion Models", arXiv:2302.05543v3, November 26, 2023, https: / / arxiv.org / pdf / 2302.05543

[0035] When conditional image FPI and CPI are input, the image encoders 971 and 972 generate and output the conditional data VTf and VTc described above by converting the conditional image FPI and CPI into a data format that can be input to the control models 991 and 992. The conditional data VTf and VTc are, for example, vectors of a predetermined number of dimensions. The image encoders 971 and 972 use image encoders that have been pre-trained by the technique called CLIP described above.

[0036] As shown in Figure 2, the conditional image FPI input to the first image encoder 971 is generated by the feature point extraction unit 213. The feature point extraction unit 213 is a functional unit realized by the processor 210 of the image processing server 200 executing computer programs PGs. The feature point extraction unit 213 uses a biological image CI to generate a feature point image as the conditional image FPI, which shows multiple points on the face of the organism indicated by the biological image CI. Hereinafter, the conditional image FPI input to the first image encoder 971 will also be referred to as the feature point image FPI.

[0037] As shown in Figure 2, the conditional image CPI input to the second image encoder 972 is generated by the background removal processing unit 214 and the color reduction processing unit 215. The background removal processing unit 214 and the color reduction processing unit 215 are functional units realized by the processor 210 of the image processing server 200 executing computer programs PGs. The background removal processing unit 214 generates a processed biological image CIr from which the background has been removed by performing background removal processing on the biological image CI. The color reduction processing unit 215 uses the processed biological image CIr to generate a color palette image as the conditional image CPI, which shows the main colors contained in the organism shown in the processed biological image CIr. Hereinafter, the conditional image CPI input to the second image encoder 972 will also be called the color palette image CPI. Details of the processing performed by these functional units 211-215 will be described later.

[0038] As can be seen from the above explanation, the input information input to the image generation model 900 includes text PT, input image II, feature point image FPI, and color palette image CPI.

[0039] A3. Decorative character generation process Figure 3 is a flowchart of the decorative character generation process in the first embodiment. The decorative character generation process uses an image generation model 900 to generate an image of a character (hereinafter also called a decorative character image OI) that has been decorated to represent the face of a living creature (for example, the face of a dog). The decorative character generation process is executed collaboratively by a terminal device 100 (processor 110) that functions as an application and an image processing server 200 (processor 210). The terminal device 100 (application) mainly performs processing related to a screen that functions as a user interface (referred to as a UI screen) and processing to control a printer 300 (in modified cases, a sewing machine 400 or a cutting machine 500). The image processing server 200 mainly performs processing to generate decorative character images based on user instructions obtained via the terminal device 100.

[0040] In S110, the image processing server 200 obtains character information from the user via the terminal device 100. The obtained character information includes information indicating the character to be processed and font information indicating the type of font (typeface) of the character. In this embodiment, the character to be processed is the alphabet. The character to be processed may also include kana characters (hiragana or katakana), kanji, numbers, and some or all of Arabic characters. The type of font is selected, for example, from a pre-prepared list of font candidates. For example, the number of characters to be processed is not limited to one. That is, the number of characters to be processed, k, may be an integer of 2 or more. In this embodiment, the three-letter alphabet "ABC" is obtained as the character to be processed (k=3).

[0041] Specifically, the terminal device 100 displays an input screen (not shown) on the display unit 140, which includes an input field for entering the characters to be processed and a list of selectable fonts. On this input screen, the user enters one or more characters to be processed into the input field and selects one font from the list. The terminal device 100 transmits character information, including information indicating the characters entered by the user and information indicating the font selected by the user, to the image processing server 200. The image processing server 200 then acquires the character information.

[0042] In S115, the image processing server 200 (character image generation unit 211) generates a character image FI. The character image FI is an image representing the character to be processed. The character represented by the character image FI has the shape of the font indicated by the font information. If the character to be processed consists of multiple characters, a character image FI is generated for each character. For example, if the character to be processed is "ABC", three character image FIs representing each character are generated. In a modified example, a single font image representing all three characters may be generated.

[0043] Figure 4 shows an example of an image used in the decorative character generation process. Figure 4(A) shows a character image FI representing the character "A" TX. For example, the color of the character in character image FI is black, and the color of the non-character parts (i.e., the background BG) is white. The data for each character image FI is generated in the same data format as the input image II that should be input to the image generation model 900.

[0044] In this embodiment, the data for input image II is RGB image data. RGB image data is bitmap data in which color values ​​are represented by three color components: red (R), green (G), and blue (B). Image II, OI is a rectangular image having two sides parallel to the first direction Dx and two sides parallel to the second direction Dy, which is perpendicular to the first direction Dx. Input image II is represented by the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy (the color values ​​represent the gradation values ​​of red (R), green (G), and blue (B), respectively (for example, values ​​between zero and 255)). The number of pixels in the first direction Dx and the number of pixels in the second direction Dy of input image II accepted by the image generation model 900 are predetermined. The data for the generated character image FI is RGB image data of the size that input image II should have (number of pixels in the first direction Dx and the second direction Dy).

[0045] In S120, the image processing server 200 performs a smoothing process on the character image FI to generate a processed character image. The generated processed character image is the input image II that is input to the diffusion model 960. The smoothing process is, for example, a smoothing process using a Gaussian filter. The standard deviation σ of the Gaussian filter is set to, for example, 20 to 50. Alternatively, a smoothing process using various smoothing filters such as a mean filter or a median filter may be performed.

[0046] Figure 4(B) shows the input image II (processed character image) generated using the character image FI in Figure 4(A). The outer edges of the characters TXs shown in input image II are blurred by the smoothing process.

[0047] In S125, the image processing server 200 acquires biological image CI data from the terminal device 100. Specifically, the terminal device 100 displays a file selection screen (not shown) on the display unit 140. The user operates the file selection screen to select the biological image CI data (i.e., image file) to be used to generate the decorative character image. The terminal device 100 sends (i.e., uploads) the biological image CI data specified by the user to the image processing server 200. As a result, the image processing server 200 acquires the biological image CI data.

[0048] Figure 4(C) shows an example of an acquired biological image CI. The biological image CI in Figure 4(C) is an image showing the upper body of a dog, including its face. In this embodiment, the biological image CI is assumed to be an image that includes a frontal view of the animal's face. The biological image is not limited to dogs; for example, it may be an image of other mammals such as cats or bears, birds such as parakeets or parrots, insects, or reptiles. The biological image may also be an image of a person. The biological image CI in Figure 4(C) is, for example, a photographic image generated when a user takes a picture of their pet dog using the camera function of terminal device 100. The biological image may be a CG or illustration instead of a photographic image.

[0049] In S130, the image processing server 200 (feature point extraction unit 213) uses the biological image CI to generate a feature point image FPI showing multiple feature points Pt of the face of the organism shown in the biological image CI. Figure 4(D) shows the feature point image FPI generated using the biological image CI in Figure 4(C). The image processing server 200 detects multiple feature points Pt that constitute the eyes, nose, and mouth included in the biological image CI, for example, using the face landmark detection function of MediaPipe. The image processing server 200 generates a feature point image FPI showing the detected multiple feature points Pt as circles. MediaPipe is a technology released by Google®. The feature point image FPI may also be an image that includes a mesh composed of straight lines connecting the multiple feature points Pt. The feature point image FPI is generated with a predetermined size (number of pixels in the first direction Dx and the second direction Dy) received by the first image encoder 971.

[0050] In S135, the image processing server 200 performs a vector transformation process on the feature point image FPI. Specifically, the image processing server 200 inputs the feature point image FPI to the first image encoder 971 (Figure 2) and causes the first image encoder 971 to generate a vector representing the features of the feature point image FPI. The generated vector represents the facial features of the organism shown in the biological image CI, and is therefore also called a face feature vector. The generated face feature vector is the first condition data VTf that should be input to the first control model 991. Hereafter, the first condition data VTf will also be referred to as the face feature vector VTf.

[0051] In S140, the image processing server 200 inputs the face feature vector VTf to the first control model 991. The first control model 991 processes the face feature vector VTf using an encoder and a zero convolutional layer, and inputs the resulting data to the U-net decoder of the diffusion model 960. This ensures that the facial features of the biological image CI (e.g., the positions of the eyes, nose, and mouth) are reflected in the output image OI (i.e., the decorative character image).

[0052] The first control model 991 is trained to reflect the facial features of the output image OI in the output image OI, using facial feature vectors generated from a large amount of training biological images (e.g., photographic images of animal faces such as dogs, cats, and bears). The specific training method is disclosed in the ControlNet technology paper mentioned above.

[0053] In S145, the image processing server 200 (background removal processing unit 214) performs background removal processing on the biological image CI to generate a processed biological image CIr. Figure 4(E) shows the processed biological image CIr generated using the biological image CI in Figure 4(C). As shown in Figure 4(E), the processed biological image CIr is an image from which the background has been removed from the biological image CI; in other words, it is an image obtained by filling the background of the biological image CI with white. Known methods are used for background removal processing. For example, background removal processing uses a machine learning model called U2-Net to separate the foreground and background and remove the background.

[0054] U2-Net is disclosed in the following paper. Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaiane and Martin Jagersand, "U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection", arXiv:2005.09007v3, March 8, 2022, https: / / arxiv.org / abs / 2005.09007v3

[0055] In S150, the image processing server 200 generates a color palette image CPI using the processed biological image CIr. Specifically, the image processing server 200 clusters the pixels of the processed biological image CIr, excluding background pixels (pixels filled with white), into groups of M (where M is an integer greater than or equal to 2). Clustering is performed according to a known algorithm, such as the x-means method or the k-means method, so that pixels with similar values ​​(i.e., pixel colors) belong to the same group. M is a predetermined value and is a value that is sufficiently less than the number of colors present in the organism shown in the processed biological image CIr. For example, M is a value of about 3 to 10, and in this embodiment it is 4.

[0056] The image processing server 200 calculates the representative color of each of the M groups and the number of pixels belonging to each of the M groups. The representative color of a group is, for example, a color represented by the average or median value of each RGB component of the pixels belonging to that group. The image processing server 200 generates a color palette image CPI based on the representative colors and pixel counts of the M groups. Figure 4(F) shows an example of a color palette image CPI. The color palette image CPI is an image that shows the color distribution of the processed biological image CIr. For example, the color palette image CPI contains M (M=4 in the example in Figure 4(F)) sub-images CA1-CA4. The color palette image CPI is an image of a predetermined size. The M sub-images CA1-CA4 correspond one-to-one with the M groups mentioned above. Each sub-image is a monochrome image having the representative color of the corresponding group. The areas of the M sub-images CA1-CA4 are determined such that the area ratio of the sub-images CA1-CA4 matches the ratio of the number of pixels in the corresponding groups. The M sub-images CA1-CA4 are arranged from top to bottom in descending order of area in the color palette image CPI.

[0057] In S155, the image processing server 200 performs a vector transformation process on the color palette image CPI. Specifically, the image processing server 200 inputs the feature point image FPI to the second image encoder 972 (Figure 2) and causes the second image encoder 972 to generate a vector representing the features of the color palette image CPI. The generated vector is a vector representing the color features of the organism shown in the biological image CI, and is therefore also called a color feature vector. The generated color feature vector is the second condition data VTc that should be input to the second control model 992. Hereafter, the second condition data VTc will also be referred to as the color feature vector VTc.

[0058] In S160, the image processing server 200 inputs the color feature vector VTc to the second control model 992. The second control model 992 processes the color feature vector VTc using an encoder and a zero convolutional layer, and inputs the resulting data to the U-net decoder of the diffusion model 960. This ensures that the color features of the biological image CI (e.g., representative color and the proportion of each color) are reflected in the output image OI (i.e., decorative character image).

[0059] The second control model 992 is trained to reflect the color features of the output image OI in the output image OI, using a color feature vector generated from a large amount of training biological images (e.g., photographic images of animal faces such as dogs, cats, and bears). The specific training method is the same as that used for the first control model 991 and is disclosed in the ControlNet technology paper mentioned above.

[0060] In S165, the image processing server 200 obtains a text PT (prompt) for decorative characters to be input to the diffusion model 960. In this embodiment, an image of a "dog" is used as the biological image CI. In this embodiment, the English alphabet is used as the characters to be processed. For this reason, "dog, alphabet" is used as the text PT. The image processing server 200 analyzes the biological image CI obtained in S125 to identify the type of organism shown in the biological image CI (e.g., dog, cat) and identifies the English words "e.g., dog, cat" that represent the identification result. For the analysis of the biological image CI, for example, a machine learning model known as a classifier may be used. Based on the character information obtained in S110, the image processing server 200 identifies the type of characters to be processed (e.g., the alphabet) and identifies the English word "alphabet" that represents the identification result. The image processing server 200 obtains a text PT obtained by arranging the identified English words. In a modified version, the English word indicating the type of organism shown in the biological image CI may, for example, be entered by the user into the terminal device 100 when the biological image CI is uploaded to the image processing server 200 in S125, and then transmitted from the terminal device 100 to the image processing server 200.

[0061] In S170, the image processing server 200 inputs input information, including the input image II (Figure 4(B)), the text PT (e.g., "dog,alphabet"), and the seed value, into the image generation model 900 (diffusion model 960) to generate the output image OI. The seed value is a random number used to introduce variation into the generated image. For example, n different random numbers (where n is an integer greater than or equal to 2) are used as the seed value. In the process of generating the output image OI, as described above, data output from the two control models 991 and 992 is also input into the diffusion model 960.

[0062] The image processing server 200 generates a total of (k × n) output image OIs, n for each character, when there are k characters to be processed. The image processing server 200 generates (k × n) output image OIs by inputting (k × n) input information one by one into the image generation model 900. When there are three characters to be processed, "ABC", (3 × n) output image OIs are generated. For example, the n input information pieces for generating n decorated "A" character images each include an input image II (Figure 4(B)) representing the character "A", text PT, and one of n types of seed values. The image processing server 200 inputs each of these n input pieces into the image generation model 900 to generate n output image OIs representing the decorated "A" characters.

[0063] Figure 4(G) shows output image OI1 as an example of an output image OI. This output image OI1 is generated when input information including input image II (Figure 4(A)), feature point image FPI (Figure 4(D)), color palette image CPI (Figure 4(F)), and text PT "dog,alphabet" is input to the image generation model 900. As described above, generation by img2img can generate an output image OI by modifying input image II according to the text PT. For this reason, output image OI1 is an image in which the input image II, which represents the letter "A", is modified according to the text PT "dog,alphabet", resulting in an image in which a dog and the letter "A" are fused together. Furthermore, when the image FPI and CPI showing the features of the dog's face and color as shown in the biological image CI are also input, output image OI1 becomes an image that shows a decorated letter with the letter "A" adorned with the appearance of a dog's face. Although the diagram is omitted, let's assume, for example, that an image of a cat is used as the biological image CI, the letter "C" is used as the input image II, and "cat,alphabet" is used as the text PT. In this case, the image generation model 900 can be made to generate an output image OI that shows a decorated character, where the letter "C" is decorated to represent a cat's face. Thus, it can be said that the image generation model 900 is trained to generate an image that is decorated to represent the face of the organism shown in the biological image CI, based on this input information (for example, images and text). Below, the output image OI that shows the decorated character will also be called the decorated character image OI.

[0064] In the decorative character image OI1 in Figure 4(A), the dog image is not realistic but rather anime or illustration-like. This is because the image generation model 900 employs a Stable Diffusion model (also called a checkpoint) as its diffusion model 960, which is trained to generate anime or illustration-like images. In the decorative character image OI1, the dog's facial features reflect those of the dog shown in the biological image CI. This is because the feature point image FPI generated using the biological image CI is input to the image generation model 900. In the decorative character image OI1, the colors of the decorative characters, including the dog, reflect those of the dog shown in the biological image CI. This is because the color palette image CPI generated using the biological image CI is input to the image generation model 900. By reflecting the facial features and colors of the dog shown in the biological image CI in the decorative character image OI1, the user can perceive that the decorative characters in the decorative character image OI1 resemble the dog shown in the biological image CI.

[0065] In step S175 of Figure 3, the image processing server 200 displays a selection screen WI1 containing n types of decorative character images OI on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the selection screen WI1 and transmits the generated data to the terminal device 100. The terminal device 100 uses the received data to display the selection screen WI1 on the display unit 140.

[0066] Figure 5 shows an example of the selection screen WI1. The selection screen WI1 in Figure 5 includes n generated decorative character images OI (e.g., OI1-OI3), a message MS1, a selection cursor CS, a selection button BT1, and a redo button BT2. Message MS1 is a message prompting the user to select their preferred character from the n decorative character images OI.

[0067] In S180, the image processing server 200 receives a selection instruction from the user via the terminal device 100 to select a decorative character image OI to be output. Specifically, the user inputs a selection instruction for a decorative character image OI by moving the cursor CS on the selection screen WI1 in Figure 5 to select one of their preferred decorative character images OI and tapping the selection button BT1. When the terminal device 100 receives the selection instruction for a decorative character image OI, it sends the content of the selection instruction, i.e., information indicating the selected decorative character image OI, to the image processing server 200. If the redo button BT2 is tapped on the selection screen WI1, for example, the decorative character generation process in Figure 3 will be executed from the beginning.

[0068] In S185, the image processing server 200 displays the output screen WI2, which includes the decorative character image to be output, on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the output screen WI2, which includes the type of decorative character image OI selected by the selection instruction, and transmits the generated data to the terminal device 100. The terminal device 100 uses the received data to display the output screen WI2 on the display unit 140.

[0069] Figure 6 shows an example of the output screen WI2. As shown in Figure 6, the output screen WI2 includes an output image TI, a print button BT3, and a redo button BT4. The output image TI includes three decorative character images OI (OI1, OI4, OI5) representing the three characters "ABC" to be processed. The three decorative character images OI are generated using the same seed value as the one used when generating the decorative character image OI1 selected by the user. The user taps the print button BT3 on the output screen WI2 to confirm the output image TI and to print. When the print button BT3 is tapped, the process proceeds to S190. When the redo button BT4 is tapped on the output screen WI2, for example, the decorative character generation process shown in Figure 3 is executed from the beginning.

[0070] In S190, the image processing server 200 generates a print job to print an output image TI (Figure 6) that includes a decorative character image OI. In S195, the image processing server 200 sends the print job to the printer 300, causing the printer 300 to print the output image TI. Specifically, the image processing server 200 may send the print job to the printer 300 via the terminal device 100, or it may send the print job directly to the printer 300 without going through the terminal device 100. When the printer 300 receives the print job, it prints the output image TI, which includes the decorative character image OI, onto the paper according to the print job.

[0071] According to the embodiment described above, the system 1000 acquires character information TXI for specifying characters and a biological image CI, which is an image of a living organism including a face (S110, S125 in Figure 3). The character information TXI and the biological image CI are acquired based on user input. The system 1000 uses the character information TXI and the biological image CI to input the input information to the image generation model 900, causing the image generation model 900 to generate a decorative character image OI (S170 in Figure 3). The decorative character image OI is a character with decorations that represent the face of the organism shown in the biological image CI (Figure 4(G)). The system 1000 displays the decorative character image OI on the display unit 140 of the terminal device 100 (S185 in Figure 3). The system 1000 prints the output image TI, which includes the decorative character image OI displayed on the display unit 140, onto paper (S190, S195 in Figure 3). As a result, the image generation model 900 can be used to generate decorative character images (OI) that correspond to user input. For example, it can generate a decorative character image (OI) that shows decorative characters with decorations applied to the face of an organism shown in a biological image (CI). Furthermore, the decorative character image (OI) is displayed and printed, so that the user can be provided with a decorative character image (OI) that corresponds to their input.

[0072] Furthermore, according to this embodiment, the system 1000 generates an input image II showing characters using character information TXI (Figure 3, 115, S120). The system 1000 generates biological information showing the characteristics of an organism (e.g., a feature point image FPI and a color palette image CPI) using a biological image CI (Figure 3, S135, S155). The input information, including the input image II and the biological information, is input to the image generation model 900 (Figures 2 and 3, S170). As a result, a decorative character image OI can be generated that reflects the characteristics of the organism (e.g., a dog) shown in the biological image CI.

[0073] Furthermore, according to this embodiment, the system 1000 uses a biological image CI to generate a feature point image FPI as biological information, which is feature information based on multiple feature points Pt of the organism's face shown in the biological image CI (S130 in Figure 3). As a result, information based on multiple feature points Pt of the organism's face shown in the biological image CI is input to the image generation model 900, so that a decorative character image OI that reflects the features of the organism's face shown in the biological image CI (e.g., facial features and expressions) can be generated.

[0074] Furthermore, according to this embodiment, the system 1000 uses a biological image CI to generate a color palette image CPI as biological information, which is color information based on multiple colors possessed by the organism (e.g., a dog) shown in the biological image CI (S145, S150 in Figure 3). As a result, information based on the colors of the organism shown in the biological image CI is input to the image generation model 900, so that a decorative character image OI that reflects the color characteristics of the organism shown in the biological image CI can be generated.

[0075] Furthermore, according to this embodiment, the system 1000 performs a background removal process on the biological image CI to remove the background and generate a processed biological image CIr (S145 in Figure 3), and uses the processed biological image CIr to identify multiple colors (S150 in Figure 3). As a result, the influence of the background is eliminated, and the multiple colors possessed by the organism shown in the biological image CI can be appropriately identified. Furthermore, according to this embodiment, the system 1000 identifies M colors by clustering the multiple pixels contained in the processed biological image CIr into M groups. As a result, a color palette image CPI showing an appropriate number of colors can be generated. For example, when generating an anime or illustration-style decorative character image OI, by identifying a relatively small number of colors M (4 colors in this embodiment), the colors of the anime or illustration-style organism can be appropriately specified. Also, in the color palette image CPI, the areas of the M partial images CA1-CA4 are determined so that the area ratio of the partial images CA1-CA4 matches the ratio of the number of pixels in the corresponding groups. As a result, the color balance of the organism shown in the decorative character image OI can be brought closer to the color balance of the organism shown in the organism image CI.

[0076] Furthermore, according to this embodiment, the system 1000 performs a smoothing process on the character image FI based on the character information TXI to generate the input image II (S120 in Figure 3). If the edges of the characters shown in the input image II are blurred, the volume of decoration applied to the characters in the generated decorative character image OI tends to be larger compared to when the edges are sharp. As a result, a more visually appealing decorative character image OI can be generated.

[0077] Furthermore, according to this embodiment, the biological image CI is an image stored in memory (e.g., a non-volatile storage device 124) by the user and is an image specified by the user. As a result, a decorative character image OI can be generated according to the user's preferences.

[0078] Furthermore, according to this embodiment, the character information TXI includes font information indicating the type of font. The shape of the character shown in the input image II has the shape of the font indicated by the font information. As a result, the font of the character shown in the input image II can be specified by the font information, thus increasing the variations of the generated decorative character image OI.

[0079] Furthermore, according to this embodiment, the system 1000 generates, for example, n types of input information using one character information TXI and a biological image CI, and generates n types of decorative character images OI by inputting each of the n types of input information into the image generation model 900 (S170 in Figure 3, OI1-OI3 in Figure 5). The system 1000 displays the n types of decorative character images OI on the display unit 140 (S175 in Figure 3, Figure 5). The system 1000 prints the image selected by the user from among the n types of decorative character images OI (S190, S195 in Figure 3). As a result, since the decorative character image OI selected by the user is printed from among the n types of decorative character images OI generated using one character information TXI and a biological image CI, it is possible to print a decorative character image OI that is more in line with the user's preferences.

[0080] As can be seen from the above explanation, the feature point image FPI and color palette image CPI in this embodiment are examples of biological information.

[0081] B. Second Example In the second embodiment, the content of the decorative character generation process differs from that of the first embodiment. In the second embodiment, as shown in Figure 1, the non-volatile storage device 224 of the image processing server 200 has, in addition to the same information as in the first embodiment, correspondence information 710 and a group of biological images CIg pre-stored. The group of biological images CIg includes, for example, data for multiple biological image CIs, including biological images CI1-CI3, which will be described later. In the decorative character generation process of the second embodiment, a biological image CI selected from the group of biological images CIg of the image processing server 200 is used, so the non-volatile storage device 124 of the terminal device 100 does not need to store data for biological image CIs. The correspondence information 710 will be described later.

[0082] Figure 7 is a flowchart of the decorative character generation process in the second embodiment. In S210, the image processing server 200 displays a sample selection screen WI3 containing multiple sample decorative character images SI on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the sample selection screen WI3 and transmits the generated data to the terminal device 100. The terminal device 100 uses the received data to display the sample selection screen WI3 on ​​the display unit 140.

[0083] Figure 8 is an explanatory diagram of the second embodiment. Figure 8(A) shows an example of the sample selection screen WI3. This sample selection screen WI3 includes multiple sample decorative character images SI (e.g., SI1-SI3), a message MS3, a selection cursor CS, and selection buttons BTs. Message MS3 is a message prompting the user to select their preferred character from among the multiple sample decorative character images SI.

[0084] Each of the multiple sample decorative character images SI corresponds one-to-one with one of the multiple biological images CI included in the biological image group CIg. For example, sample decorative character image SI1 in Figure 8(A) corresponds to biological image CI1 in Figure 8(B). Sample decorative character image SI2 corresponds to biological image CI2 in Figure 8(B). Sample decorative character image SI3 corresponds to biological image CI3 in Figure 8(B).

[0085] Furthermore, each of the sample decorative character images SI corresponds to a text PT. For example, sample decorative character images SI1 and SI2 correspond to the text PT "dog,alphabet". Sample decorative character image SI3 corresponds to the text PT "bear,alphabet".

[0086] In the second embodiment, the data for each sample decorative character image SI (SI1-SI3) is pre-generated using the corresponding biological image CI and text PT, and stored in the non-volatile storage device 224 of the image processing server 200. For example, a smoothing process is performed on the character image FI of a specific character of a specific font (in the example of Figure 8(A), "A" and "B") to generate the input image II. Furthermore, the processes S130-S160 and S170 in Figure 3 are executed using the generated input image II, the corresponding biological image CI, and the corresponding text PT to generate the sample decorative character image SI.

[0087] For this reason, the sample decorative character images SI reflect the facial and color features of the organisms shown in the corresponding biological images CI. For example, the facial and color features of the organisms that make up the decorative characters in sample decorative character image SI1 are similar to those of the organisms in the corresponding biological image CI1. Similarly, the facial and color features of the organisms that make up the decorative characters in sample decorative character images SI2 and SI3 are similar to those of the organisms in the corresponding biological images CI2 and CI3, respectively.

[0088] In S220, the image processing server 200 selects a sample decorative character based on the user's selection instruction. Specifically, the user selects a preferred sample decorative character image SI by moving the cursor CS on the sample selection screen WI3 in Figure 8(A), and inputs the selection instruction for the sample decorative character image SI by tapping the selection buttons BTs. When the terminal device 100 receives the selection instruction for the sample decorative character image SI, it transmits the content of the selection instruction, i.e., information indicating the selected sample decorative character image SI, to the image processing server 200. The image processing server 200 selects the sample decorative character image SI instructed by the user according to the received information.

[0089] In S230, the image processing server 200 acquires the biological image CI and text PT corresponding to the selected sample decorative character image SI. Specifically, the correspondence information 710 described above is information that records the correspondence between each sample decorative character image SI and the biological image CI and text PT used to generate the sample decorative character image SI. The image processing server 200 acquires the biological image CI and text PT used to generate the selected sample decorative character image SI by referring to the correspondence information 710.

[0090] In S240, the image processing server 200 executes the processes S110-S120 in Figure 3. That is, the image processing server 200 obtains character information TXI from the user via the terminal device 100 and uses the character information TXI to generate the input image II (for example, Figure 4(B)).

[0091] In S250, the image processing server 200 executes the processing shown in S130-160 of Figure 3. Specifically, the image processing server 200 generates a face feature vector VTf and a color feature vector VTc using the biological image CI acquired in S230, and inputs them to the control models 991 and 992, respectively.

[0092] In S260, the image processing server 200 executes the processes shown in S170-S195 in Figure 3. Specifically, the image processing server 200 inputs the input information, including the input image II and text PT, into the image generation model 900 (diffusion model 960) to generate a decorative character image OI, and then displays or prints the generated decorative character image OI.

[0093] According to the second embodiment described above, the system 1000 selects one sample decorative character image SI from among a plurality of sample decorative character images SI, including a first decorative character image SI1 generated when input information generated using a first biological image CI1 is input to the image generation model 900, and a second decorative character image SI2 generated when input information generated using a second biological image CI2 is input to the image generation model 900, based on the user's instructions (S220 in Figure 7). If the first decorative character image SI1 is selected, the image processing server 200 acquires the first biological image CI1 as the biological image to be used, and if the second decorative character image SI2 is selected, it acquires the second biological image CI2 as the biological image to be used (S230 in Figure 7). If the first decorative character image SI1 is selected, the system 1000 generates input information using the first biological image CI1, and if the second decorative character image SI2 is selected, it generates input information using the second biological image CI2, and inputs the generated input information to the image generation model 900 (S260 in Figure 7).

[0094] As a result, it is possible to generate and print decorative character images (OI) that suit the user's preferences. For example, even if a user looks at biological image CIs (CI1-CI3), they may not be able to imagine the decorative character image OI that will be generated. In this case, even if the user is asked to decide which decorative character image OI to use, they may not be able to determine an appropriate biological image CI that can generate the decorative character image they desire. In this embodiment, the user can simply select a sample decorative character image SI of the desired form from among several sample decorative character images SI generated using each of several biological image CIs. This allows the user to display and print a decorative character image OI of the desired form for the desired character. Therefore, it is possible to easily display and print decorative character image OI that suit the user's intentions.

[0095] C. Variations (1) The image generation model 900 (Figure 2) in each of the above embodiments is just an example and is not limited thereto. The image generation model may be modified as appropriate. Figure 9 is a block diagram showing an example of a modified image generation model 900B. In the modified form of Figure 9, a third control model 993 and a third image encoder 973 are added to the configuration of the image generation model 900 in Figure 2.

[0096] The third control model 993, like control models 991 and 992, is a model based on a conditional image generation technique called ControlNet, and has the same configuration as control models 991 and 992. The third control model 993 is input to third condition data VTe, which indicates additional conditions to be given to the diffusion model 960. In the third control model 993, the additional condition is the shape (contour) of the character image FI.

[0097] The third image encoder 973 generates and outputs the third condition data VTe described above by converting the condition image TEI, which indicates additional conditions, into a data format that can be input to the third control model 993 when it is input to the third image encoder 973. The third condition data VTe is a vector of a predetermined number of dimensions, similar to the condition data VTf and VTc. The third image encoder 973, like the image encoders 971 and 972, is an image encoder that has been pre-trained using a technique called CLIP.

[0098] As shown in Figure 9, the conditional image TEI input to the third image encoder 973 is generated by the edge extraction unit 216. The edge extraction unit 216 is a functional unit realized by the processor 210 of the image processing server 200 executing the computer program of the second embodiment. The edge extraction unit 216 generates a character contour image as the conditional image TEI by performing edge extraction processing on the character image FI. Hereinafter, the conditional image TEI input to the third image encoder 973 will also be referred to as the character contour image TEI. For the edge extraction processing, known edge extraction processing methods, such as the "findContours" function of OpenCV (Open Source Computer Vision Library), can be used. Figure 10 shows an example of a character contour image TEI. Figure 10 shows a character contour image TEI showing the contour line OL of the character image FI in Figure 4(A).

[0099] As can be seen from the above explanation, the input information input to the modified image generation model 900B includes text PT, input image II, feature point image FPI, color palette image CPI, and character contour image TEI. The character contour image TEI can be said to be shape information that shows the shape of the character shown in input image II.

[0100] In other words, in the modified character generation process, in addition to the process in Figure 3, the following processes are performed. The image processing server 200 performs edge extraction processing on the character image FI to generate a character contour image TEI, and inputs the character contour image TEI to the image generation model 900B. That is, the character contour image TEI is input to the third image encoder 973 (Figure 9) of the image generation model 900B. The third image encoder 973 generates a vector that shows the features of the character contour image TEI. The generated vector is a vector that shows the features of the character contour image TEI, that is, the shape features of the character (for example, "A") contained in the input image II, and is therefore also called a character shape vector. The generated character shape vector is the third condition data VTe that should be input to the third control model 993. Hereinafter, the third condition data VTe will also be called the character shape vector VTe. The character shape vector VTe is input to the third control model 993. The third control model 993 processes the character shape vector VTe using an encoder and a zero convolutional layer, and inputs the resulting data into the U-net decoder of the diffusion model 960. This ensures that the shape of the character (for example, the outline of the character "A") is reflected in the output image OI (i.e., the decorative character image).

[0101] Furthermore, the third control model 993 is trained so that image features indicating edges, such as contours, are reflected in the output image OI. For example, the third control model 993 employs a model called "Canny" from ControlNet.

[0102] Thus, in this modified example, the input information input to the image generation model 900B is different from the input image II and includes a character contour image TEI, which is shape information indicating the shape of the character. As a result, since shape information indicating the shape of the character is input to the image generation model 900B separately from the input image II, the shape of the character in the generated decorative character image OI can be controlled. For example, when generating a decorative character image OI by executing the decorative character generation process of each of the above embodiments, depending on the type of character used, the type of font, and the biological image CI, the shape of the character in the decorative character image OI may be distorted, reducing the readability of the character. In such cases, by adding the character contour image TEI to the input information as in the modified example, the shape of the character can be maintained in the generated decorative character image OI, thereby improving the readability of the character.

[0103] (2) In the above modified example, the third control model 993 is a model called "Canny" from ControlNet. Alternatively, the third control model 993 may be a model called "depth" from ControlNet. In this case, the image processing server 200 inputs the character image FI to the depth estimation model and generates an estimated depth image that shows the estimated depth of the character image FI for each pixel. The image processing server 200 inputs the estimated depth image to the third image encoder 973 of the image generation model 900B. Even in this case, the estimated depth image is input to the image generation model 900B as shape information that shows the shape of the characters, separately from the input image II. Therefore, in this case as well, the shape of the characters in the generated decorative character image OI can be controlled.

[0104] (3) System 1000 may include a sewing machine 400 and a cutting machine 500, as shown in Figure 1. The sewing machine 400 is a device that forms an image on fabric by embroidering on the fabric with thread according to embroidery data. The cutting machine 500 is a device that expresses an image in the shape of a sheet material by cutting the sheet material with a blade according to cut data. The sheet material may be, for example, paper, cloth, a resin sheet such as urethane foam, or leather.

[0105] For example, in the above embodiment, the system 1000 has the printer 300 print an output image TI including a decorative character image OI. Alternatively, the system 1000 may have the output image TI embroidered by the sewing machine 400. The system 1000 may also have the output image TI cut by the cutting machine 500. Or, the system 1000 may combine these devices to represent the output image TI. For example, the system 1000 may print the output image TI onto a sheet material using the printer 300, and then cut the sheet material with the printed output image TI along the outline of the decorative characters included in the output image TI using the cutting machine 500.

[0106] (4) In the first embodiment described above, the image processing server 200 acquires a biological image CI selected by the user from the terminal device 100 (S125 in Figure 3). Alternatively, the non-volatile storage device 224 of the image processing server 200 may store multiple biological image CIs prepared by the operator of the image processing server 200. In this case, the image processing server 200 may acquire the biological image CI to be used from among the multiple biological image CIs stored in its non-volatile storage device 224, either based on the user's selection instruction or automatically.

[0107] (5) In the second embodiment described above, the input information associated with the sample decorative character image SI in the correspondence information 710 is the biological image CI and the text PT. However, the input information associated with the sample decorative character image SI may include other information. Other information may include, for example, a seed value and a parameter (also called Control Weight) that indicates the strength with which the data output from control model 991 and control model 992 affects the decorative character image OI.

[0108] (6) The diffusion model 960 of the image generation model 900 in each of the above embodiments is a machine learning model called Stable Diffusion, but is not limited to this. Instead of the diffusion model 960, the image generation model 900 may use various models that have been trained to generate images with decorations showing the faces of organisms when input information including information based on biological images showing the faces of organisms is input. For example, the image generation model 900 may employ an image generation model (for example, a style transfer model called MST (Multimodal Style Transfer)) that generates an image in which the style (artistic style) of the style image is applied to the content image when a style image and a content image are input. In this case, for example, by inputting the input image II of this embodiment as the content image and the biological image CI or sample decorated character image SI of this embodiment as the style image into the MST model, a decorated character image can be generated in which decorations are applied to the characters shown in the input image II. Even in this case, data based on feature point images FPI and conditional images CPI generated using biological image CIs are input to the MST model via control models 991 and 992 and image encoders 971 and 972. This allows the facial features and color features of the organism shown in the biological image CIs to be reflected in the decorative character images generated by the MST model. Furthermore, other known image generation models such as the Adain model, StyTR2 model, and U-GAT-IT model may be used instead of the MST model.

[0109] Alternatively, instead of diffusion model 960, a diffusion model different from Stable Diffusion, such as "VQ-Diffusion" or "GLIDE," may be used.

[0110] "VQ-Diffusion" is disclosed in the following paper. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo, "Vector Quantized Diffusion Model for Text-to-Image Synthesis", arXiv:2111.14822v3, March 3, 2022, https: / / arxiv.org / abs / 2111.14822v3

[0111] "GLIDE" is disclosed in the following paper. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen, "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models", arXiv:2112.10741v3, March 8, 2022, https: / / arxiv.org / abs / 2112.10741v3

[0112] (7) The control models 991 and 992 above employ a model that uses a conditional image generation technique called controlnet, but other models may be used. For example, a model that uses a conditional image generation technique called "IP-Adapter" may be used.

[0113] The "IP-Adapter" is disclosed in the following paper. Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, Wei Yang, "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models", arXiv:2308.06721v1, August 13, 2023, https: / / arxiv.org / abs / 2308.06721v1

[0114] (8) The image generation model 900 in each of the above embodiments includes two image encoders 971 and 972 and two control models 991 and 992. Alternatively, the image generation model may include the first image encoder 971 and the first control model 991, but may not include the second image encoder 972 and the second control model 992. In this case, steps S145-S160 in Figure 3 may be omitted.

[0115] Furthermore, the image generation model may include a second image encoder 972 and a second control model 992, but may not include a first image encoder 971 and a first control model 991. In this case, steps S130-S140 in Figure 3 may be omitted.

[0116] (9) The decorative character generation process in each of the above embodiments is an example and may be modified as appropriate. For example, the character information acquired in S110 in Figure 3 does not need to include font information indicating the type of font specified by the user. In this case, for example, the character image FI generated in S115 in Figure 3 is generated using a predetermined font. Also, the smoothing process in S120 may be omitted.

[0117] Furthermore, in S170 of Figure 3, n types of decorative character images OI with different seed values ​​are generated for each character to be processed. Alternatively, only one type of decorative character image OI may be generated for each character to be processed. In this case, S175 and S180 of Figure 3 may be omitted.

[0118] Furthermore, in the decorative character generation process shown in Figure 3, one input image II is generated for each character to be processed, and a decorative character image OI is generated for each character. Alternatively, the image processing server 200 may generate an input image containing multiple characters and input the input information containing this input image to the image generation model 900 to generate a single decorative character image containing multiple decorative characters.

[0119] (10) In each of the above embodiments, the terminal device 100 primarily performs processing related to a screen that functions as a user interface (referred to as a UI screen) and processing to control the printer 300. Alternatively, the terminal device 100 may perform some of the image processing that the image processing server 200 performs on behalf of the image processing server 200.

[0120] For example, terminal device 100 may execute the following processes from the decorative character generation process in Figure 3: the process of generating input image II (S110-S120), the process of generating feature point image FPI (S125, S130), the process of generating color palette image CPI (S125-S150), and the process related to displaying or printing decorative character image OI (S175-S195). The image processing server 200 may then execute only the process of generating decorative character image OI using the image generation model 900 (S135, S140, S155, S160, S170). The same applies to the decorative character generation process in Figure 7.

[0121] Furthermore, the terminal device 100 may perform all of the decorative character generation process shown in Figure 3. In this case, the system 1000 may consist only of the terminal device 100 and the printer 300, and the image processing server 200 may be omitted.

[0122] (11) In each of the above embodiments, the generation of decorative character images OI is performed by the user of the terminal device 100 in order to print the decorative character images OI. However, the generation of decorative character images OI may also be performed by, for example, the manufacturer of the printer 300, sewing machine 400, or cutting machine 500. In this case, for example, a computer (terminal or server) owned by the manufacturer performs the decorative character generation process shown in Figures 3 and 7. In this case, for example, decorative character images OI are generated for all the letters of the alphabet, and the data of the decorative character images OI is stored in the memory of the printer 300, sewing machine 400, or cutting machine 500 when the printer 300, sewing machine 400, or cutting machine 500 is manufactured. The printer 300, sewing machine 400, or cutting machine 500 then uses the data of the decorative character images OI to print, embroider, or cut the decorative character images OI, for example, in response to user operations.

[0123] (12) In each of the above embodiments and the above modifications, the processor 210 may have the GPU 260 perform various calculations. The GPU 260 may be omitted.

[0124] (11) The processing performed by the terminal device 100 in Figure 1 may also be performed by an image processing device such as a digital camera, printer, or multifunction device.

[0125] In each of the above embodiments, some of the configurations implemented by hardware may be replaced with software, and conversely, some or all of the configurations implemented by software may be replaced with hardware. For example, the processing by the image generation model 900 (Figure 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0126] Furthermore, if some or all of the functions of this disclosure are implemented by a computer program, that program may be provided in the form of a computer-readable recording medium (e.g., a non-temporary recording medium). The program may be used while stored on the same or a different recording medium (computer-readable recording medium) as it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and DVD-ROMs, but may also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as hard disk drives.

[0127] The above embodiments and modifications can be combined as appropriate. Furthermore, the above embodiments and modifications are provided to facilitate understanding of this disclosure and do not limit the present invention. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are included. [Explanation of symbols]

[0128] 1000...System, 100...Terminal device, 110...Processor, 120...Storage device, 140...Display unit, 150...Operation unit, 170...Communication interface, 200...Image processing server, 210...Processor, 211...Character image generation unit, 212...Smoothing processing unit, 213...Feature point extraction unit, 214...Background removal processing unit, 215...Color reduction processing unit, 216...Edge extraction unit, 220...Memory device, 260...Graphics processing unit, 270...Communication interface, 300...Printer, 400...Sewing machine, 500...Cutting machine, 710...Correspondence information, 900, 900B...Image generation model, 910...Text encoder, 920...Image encoder, 930...Latent variable model Lu, 940…Image Decoder, 960…Diffusion Model, 971, 972, 973…Image Encoder, 991, 991, 993…Control Model, CI…Biological Image, CIr…Processed Biological Image, CPI…Color Palette Image, FI…Character Image, FPI…Feature Point Image Green, II…Input Image, IT…Internet, LN…Local Area Network, OI…Decorative Character Image, PGs, PGt…Computer Program, PT…Text, SI…Sample Decorative Character Image, TEI…Character Contour Image, TI…Output Image, TXI…Character Information, VTc…Color Feature Vector, VDe…Character Shape Vector, VTf…Face Feature Vector, WI1…Selection Screen, WI2…Output Screen, WI3…Sample Selection Screen

Claims

1. An image processing system, An information acquisition unit that acquires character information for specifying characters and a biological image which is an image of a living organism including a face, wherein at least a portion of the character information and the biological image is acquired based on user input, An image generation unit that uses the aforementioned character information and the aforementioned biological image to input input information into an image generation model, thereby causing the image generation model to generate a decorated character image, wherein the decorated character image is an image of a decorated character which is a character that has been decorated to represent the face of the organism shown in the biological image, and the image generation model is trained to generate an image which has been decorated to represent the face of the organism shown in the biological image based on the input information, A display control unit that displays the decorative character image on the display unit, An image forming unit that prints, embroiders, or cuts the decorative character image displayed on the display unit onto an object, An image processing system equipped with the following features.

2. The image processing system according to claim 1, The image generation unit, Using the aforementioned character information, an input image representing the characters is generated. Using the aforementioned biological image, biological information representing the characteristics of the biological image is generated. An image processing system that inputs input information, including the aforementioned input image and the aforementioned biological information, into the image generation model.

3. The image processing system according to claim 2, The image generation unit is an image processing system that uses the biological image to generate feature information based on multiple feature points of the face of the organism shown in the biological image as the biological information.

4. The image processing system according to claim 2, The image generation unit is an image processing system that uses the biological image to generate color information based on multiple colors possessed by the organism shown in the biological image as the biological information.

5. The image processing system according to claim 4, The image generation unit, A background removal process is performed on the aforementioned biological image to remove the background, thereby generating a processed biological image. An image processing system that identifies the multiple colors using the processed biological image.

6. The image processing system according to claim 2, The image generation unit is an image processing system that generates the input image by performing a process including smoothing on a character image based on the character information.

7. The image processing system according to claim 1, The aforementioned biological image is an image stored in memory by the user and is an image specified by the user, in an image processing system.

8. The image processing system according to claim 1, The aforementioned character information includes font information indicating the type of font, An image processing system in which the shape of the characters shown in the input image has the shape of the font indicated by the font information.

9. The image processing system according to claim 1, The aforementioned image processing system further, The system includes a sample selection unit that selects one decorative character image from a plurality of sample decorative character images, including a first decorative character image generated when the input information generated using a first biological image is input to the image generation model, and a second decorative character image generated when the input information generated using a second biological image is input to the image generation model, based on the user's instructions. The information acquisition unit acquires the first biological image as the biological image when the first decorative character image is selected, and acquires the second biological image as the biological image when the second decorative character image is selected. The image generation unit generates the input information using the first biological image as the biological image when the first decorative character image is selected, and generates the input information using the second biological image as the biological image when the second decorative character image is selected, in this image processing system.

10. The image processing system according to claim 1, The image generation unit, Using one of the aforementioned character information and the aforementioned biological image, n types (where n is an integer of 2 or more) of the aforementioned input information are generated. By inputting each of the n types of input information into the image generation model, n types of decorative character images are generated. The display control unit displays n types of the decorative character images on the display unit. The image forming unit is an image processing system that prints, embroiders, or cuts an image selected by the user from among n types of decorative character images onto the object.

11. The image processing system according to claim 1, An image processing system in which the input information is different from the input image that shows characters, and includes shape information that shows the shape of the characters.

12. A method for generating decorative character images, A first step involves using a biological image, which is an image of an organism including its face, to generate feature information based on multiple feature points of the organism's face shown in the biological image. A second step of causing an image generation model to generate a decorated character image by inputting input information including an input image showing characters and the feature information, wherein the decorated character image is an image showing decorated characters with decorations representing the face of a creature shown in the biological image, and the image generation model is trained to generate an image with decorations representing the face of a creature shown in the biological image based on the input information, A method for generating decorative character images, including those mentioned above.

13. A method for generating decorative character images, A first step involves using a biological image, which is an image of a living organism including its face, to generate color information based on multiple colors possessed by the organism shown in the biological image. A second step of causing an image generation model to generate a decorative character image by inputting input information including an input image showing characters and the color information, wherein the decorative character image is an image showing decorative characters with decorations representing the face of a creature shown in the biological image, and the image generation model is trained to generate an image with decorations representing the face of a creature shown in the biological image based on the input information, A method for generating decorative character images, including those mentioned above.

14. A method for generating according to claim 12 or 13, further, The process includes a step of performing a process that includes smoothing on a character image to generate a processed character image, A generation method wherein the input image is the processed character image.

15. A method for generating according to claim 12 or 13, further, The process includes obtaining character information, including font information that indicates the type of font, A generation method wherein the shape of the characters shown in the input image has the shape of the font indicated by the font information.

16. A method for generating according to claim 13, The first step is, A background removal process is performed on the aforementioned biological image to remove the background, thereby generating a processed biological image. A generation method comprising the step of identifying the multiple colors by performing a color reduction process on the processed biological image to reduce the number of colors.

Images