Image processing system and method for generating decorative character images

Abstract

Generate images of characters using machine learning models. [Solution] The image processing system includes an information acquisition unit that acquires character information for specifying characters and decoration information for specifying positions of decorative elements that constitute decoration relative to the characters, and an image generation unit that causes an image generation model to generate a decorated character image by inputting input information using the character information and decoration information into the image generation model. The decorated character image is an image showing a decorated character, which is a character decorated with decorations including decorative elements. The image generation model is trained to generate an image decorated with decorations including decorative elements based on the input information.

Description

[Technical Field]

[0001] The present specification relates to a technique for generating an image showing characters. [Background technology]

[0002] Patent Document 1 discloses a technology for training a neural network using a group of original characters as input patterns and a group of transformed characters as training patterns. In this technology, when an unlearned character is input to the neural network, a transformed character obtained by transforming the unlearned character according to a transformation rule is output. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-266226 Summary of the Invention [Problem to be solved by the invention]

[0004] Thus, there is a need for a technique for generating images of characters.

[0005] This specification discloses a technique for generating images of characters using machine learning models. [Means for solving the problem]

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] An image processing system comprising: an information acquisition unit that acquires, based on user input, character information for specifying characters and decorative information for specifying positions at which decorative elements that constitute the decoration are to be placed relative to the characters; an image generation unit that causes an image generation model to generate a decorated character image by inputting input information using the character information and the decorative information into the image generation model, wherein the decorated character image is an image showing a decorated character, which is the character that has been decorated with decoration including the decorative elements, and the image generation model has been trained to generate an image decorated with decoration including the decorative elements based on the input information; a display control unit that displays the decorated character image on a display unit; and an image forming unit that, when an instruction from a user is acquired, prints, embroiders, or cuts the decorated character image displayed on the display unit onto an object.

[0008] According to the above configuration, a decorative character image is generated by inputting input information using character information and decoration information acquired based on user input into an image generation model. As a result, a decorative character image corresponding to the character information and decoration information can be generated using a machine learning model. For example, a decorative character image in which decorative elements are arranged at positions specified by the decoration information can be generated. Furthermore, the decorative character image is displayed, and when a user's instruction is received, the decorative character image is printed, embroidered, or cut, so that a decorative character image corresponding to the character information and decoration information can be provided to the user.

[0009] [Application Example 2] A method for generating a decorated character image, which is an image showing decorated characters decorated with decorative elements, comprising: a first step of generating an input image showing a character area showing characters and a related color object arranged to overlap the character area, wherein the related color object is an object having a color related to the decorative element; and a second step of inputting input information including the input image into an image generation model, thereby causing the image generation model to generate the decorated character image, wherein the image generation model has been trained to generate an image decorated with decorations including the decorative elements based on the input information.

[0010] According to the above configuration, an input image showing a character is generated using character information and decoration information, and a decorated character image is generated by inputting input information including the input image into an image generation model. As a result, a decorated character image according to the character information and decoration information can be generated using a machine learning model. For example, a decorated character image in which a decorative element is arranged at a position specified by the decoration information can be generated.

[0011] The technology disclosed in this specification can be realized in various forms, such as an image processing method and an image processing device, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-temporary recording medium) on which that computer program is recorded, and the like. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of a system 1000 according to a first embodiment. [Figure 2] FIG. 9 is a block diagram illustrating an example of an image generation model 900. [Figure 3] 10 is a flowchart of a decorated character generation process according to the first embodiment. [Figure 4] 10A and 10B are diagrams showing examples of images used in decorative character generation processing. [Figure 5] An explanatory diagram of a decorative character image OIf. [Figure 6] 10A and 10B are diagrams showing examples of a selection screen and an output screen. [Figure 7] 10 is a flowchart of a decorated character generation process according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a sample selection screen WI3 according to the second embodiment. [Figure 9] 10 is a flowchart of a decorated character generation process according to a third embodiment. [Figure 10] 10 is a flowchart of an input image generation process. [Figure 11] An illustration of the amount of decoration. [Figure 12] FIG. 10 is a diagram showing an example of an adjustment instruction input screen WI4. [Figure 13] FIG. 10 is a first explanatory diagram of the input image generation process. [Figure 14] FIG. 10 is a second explanatory diagram of the input image generation process. DETAILED DESCRIPTION OF THE INVENTION

[0013] A. First Example: A1. System 1000 Configuration FIG. 1 is a block diagram showing the configuration of a system 1000 according to a first embodiment. The system 1000 includes a terminal device 100, an image processing server 200, and a printer 300. The terminal device 100 and the printer 300 are connected to a local area network LN and are communicatively connected to each other. The local area network LN is connected to the Internet IT, and the image processing server 200 is connected to the Internet IT. Therefore, the terminal device 100 and the image processing server 200 are communicatively connected via the local area network LN and the Internet IT. In a modified example, a sewing machine 400 or a cutting machine 500 may be provided instead of the printer 300, as will be described later.

[0014] The image processing server 200 is, for example, a computer, such as a cloud server, operated by a business that manufactures and sells the printer 300.

[0015] The image processing server 200 includes a processor 210, a storage device 215, a graphics processing unit 260 (referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0016] The processor 210 is a device configured to process data, such as a central processing unit (CPU). The volatile storage device 220 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data for a computer program 231, an image generation model 900, and a motif image group 800. The image generation model 900 is a program module that forms a trained machine learning model.

[0017] The computer program 231 and the image generation model 900 are provided, for example, in a form uploaded by a business operator that operates the image processing server 200. The processor 210 executes the computer program 231 to execute the decorated character generation process described below in cooperation with the terminal device 100. The motif image group 800 includes a plurality of motif images used in the decorated character generation process described below.

[0018] The GPU 260 is a computing device configured to execute various numerical calculations such as image processing and machine learning. The GPU 260 executes various calculations in accordance with instructions from the processor 210. In this embodiment, the GPU 260 is used to execute calculations of the image generation model 900. The communication interface 270 is an interface for connecting to the Internet IT, and is, for example, a wired interface conforming to Ethernet (registered trademark).

[0019] The terminal device 100 is a computer, for example, a mobile terminal such as a smartphone, a tablet computer, etc. In a modified example, the terminal device 100 may be a personal computer.

[0020] The terminal device 100 includes a processor 110, a storage device 115, a display unit 140, an operation unit 150, and a communication interface 170. These elements are connected to each other via a bus. The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130.

[0021] The processor 110 is a device configured to process data, such as a CPU. The volatile storage device 120 is, for example, a DRAM, and the non-volatile storage device 130 is, for example, a flash memory. The non-volatile storage device 130 stores data for a computer program 131.

[0022] The computer program 131 is an application program that is provided, for example, in the form of being downloaded from a server connected via the Internet IT. By executing the computer program 131, the processor 110 functions as an application that executes the decorative character generation process described below in cooperation with the image processing server 200.

[0023] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive user operations, such as buttons or a touch panel overlaid on the display unit 140. In this embodiment, the display unit 140 and the operation unit 150 form a so-called touch screen. The user can input various instructions to the image processing server 200 by operating the operation unit 150. The display unit 140 may display operation elements (e.g., buttons, sliders, etc.), and the displayed elements may be operated through operation of the operation unit 150.

[0024] The printer 300 is a device that prints an image on a printing medium such as paper using color materials such as ink or toner.

[0025] A2. Image generation model 900 2 is a block diagram illustrating an example of an image generation model 900. The image generation model 900 may be any of various models that use data of an input image to generate data of an output image based on the input image. In this embodiment, the image generation model 900 is a machine learning model called Stable Diffusion.

[0026] Stable Diffusion is a model that synthesizes high-resolution images using a latent diffusion model. The technology for synthesizing high-resolution images using a latent diffusion model is disclosed in, for example, the following paper: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjoern Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models", arXiv:2112.10752, April 13, 2022, http: / / arxiv.org / abs / 2112.10752

[0027] Data for the pre-trained Stable Diffusion model is made publicly available on the Internet by Stability AI. In this embodiment, data from the publicly available pre-trained model is used as data for the image generation model 900. The image generation model 900 includes a text encoder 910, an image encoder 920, a latent variable model 930, and an image decoder 940. The text encoder 910 is configured to convert text PT into a vector tv (such a vector tv is also called a text embedding). The image encoder 920 is configured to convert an input image II into latent variables lvi. The latent variable model 930 is configured to perform a noise addition process that adds noise to the latent variables lvi and a process that outputs processed latent variables lvo by performing a de-diffusion process that removes noise from the noisy latent variables. The latent variable model 930 includes a neural network called a U-Net (not shown) for performing the de-diffusion process. The image decoder 940 is configured to generate an output image OI using the latent variables lvo. The latent variable model 930 uses the vector tv obtained from the text encoder 910 for conditioning in the de-diffusion process. By executing such a de-diffusion process, the latent variable model 930 can generate a latent variable lvo that is associated with an image conditioned by the text PT. The text encoder 910 is pre-trained so that the vector tv obtained from the text PT and the image represented by the text PT can be associated. As the text encoder 910, an encoder pre-trained by a technology called CLIP is used. CLIP is a technology released by OpenAI.

[0028] The image generation model 900 can generate a new output image OI represented by the text PT by performing a de-diffusion process using randomly generated noise using the vector tv from the text encoder 910. This technique is also called txt2img. Furthermore, the image generation model 900 can generate an output image OI by using the input image II and the text PT to modify the input image II according to the text PT. This technique is also called img2img. In this embodiment, a decorated character image is generated using the img2img technique. For example, as described below, when the image generation model 900 receives the text PT "letter decorated with flowers," it generates an image decorated with "flowers" for the input image II. For this reason, the image generation model 900 can be said to be a model trained to generate an image decorated with decorative elements (e.g., flowers) based on input information (e.g., an image or text).

[0029] In this embodiment, the data of the input image II and the output image OI are RGB image data. RGB image data is bitmap data that represents color values ​​using three color components: red (R), green (G), and blue (B). Images II and OI are rectangular images with two sides parallel to a first direction Dx and two sides parallel to a second direction Dy perpendicular to the first direction Dx. Images II and OI are represented by the color values ​​of a plurality of pixels arranged in a matrix along the first direction Dx and the second direction Dy (the color values ​​indicate the gradation values ​​of red (R), green (G), and blue (B) (e.g., values ​​greater than or equal to zero and less than or equal to 255)). The number of pixels in the first direction Dx and the number of pixels in the second direction Dy accepted by the image generation model 900 are each predetermined and common to the input image II and the output image OI. Hereinafter, the above data format of the image data accepted by the image generation model 900 will be referred to as the processed data format.

[0030] A3. Decorative character generation process FIG. 3 is a flowchart of the decorated character generation process of the first embodiment. The decorated character generation process is executed by the terminal device 100 (processor 110) functioning as an application and the image processing server 200 (processor 210) working together. The terminal device 100 (application) mainly executes processes related to a screen functioning as a user interface (referred to as a UI screen) and processes for controlling the printer 300 (or, in a modified example, a sewing machine 400 or a cutting machine 500). The image processing server 200 mainly executes processes for generating decorated character images based on user instructions acquired via the terminal device 100. The decorated character images of this embodiment are images representing decorated characters, which are characters decorated with decorative elements (also called motifs). As will be described later, in this embodiment, the decorative elements are "flowers." The decorated character generation process is started, for example, when a start instruction is input to the terminal device 100 (application) and the image processing server 200 acquires the start instruction via the terminal device 100.

[0031] In S100, the image processing server 200 acquires character information from the user via the terminal device 100. The acquired character information includes information indicating the characters to be processed and font information indicating the font type of the characters. In this embodiment, the characters to be processed are alphabetic characters. The characters to be processed may include some or all of kana characters (hiragana or katakana), kanji characters, numbers, and Arabic characters. The font type is selected, for example, from a plurality of font candidates prepared in advance. For example, the number of characters to be processed is not limited to one. In other words, the number k of characters to be processed may be an integer equal to or greater than 2. In this embodiment, it is assumed that the five-character alphabet "HELLO" is acquired as the characters to be processed (k=5).

[0032] Specifically, the terminal device 100 displays an input screen (not shown) on the display unit 140, which includes an input field for inputting characters to be processed and a list of selectable fonts. On the input screen, the user inputs one or more characters to be processed in the input field and selects one font from the list. The terminal device 100 transmits character information, including information indicating the characters input by the user and information indicating the font selected by the user, to the image processing server 200. As a result, the image processing server 200 acquires the character information.

[0033] In S105, the image processing server 200 acquires one motif image MI from the motif image group 800 based on a user instruction acquired via the terminal device 100. FIG. 4 shows an example of an image used in the decorative character generation process. FIG. 4(A) shows multiple motif images MI (MI1-MI3) included in the motif image group 800 (FIG. 1). The data of the motif image MI is the RGB image data described above. The motif image MI is an image including objects (also referred to as related color objects) having colors related to the decorative elements. In this embodiment, the decorative elements are flowers, and therefore the related color objects have colors commonly recognized as flower colors, such as red, yellow, white, blue, and purple. The related color objects can have various shapes and sizes.

[0034] For example, the motif image MI1 in Figure 4(A) includes a star-shaped mark SM (hereinafter also referred to as a star mark) as a related color object. The motif image MI2 includes a circular mark RM (hereinafter also referred to as a circle mark) as a related color object. The motif image MI3 includes a cross-shaped mark CM (hereinafter also referred to as a cross mark) as a related color object. The color, size, and shape of these related color objects affect the color, size, and shape of the decorative elements included in the generated decorative character image.

[0035] The motif images MI included in the motif image group 800 are not limited to the example shown in Fig. 4(A) and may include various images. For example, the motif image MI may be a photographic image including multiple flowers, or an image showing a painting, pattern, or illustration, but preferably includes an associated color object having a color associated with the flower.

[0036] For example, the image processing server 200 transmits thumbnail images of multiple motif images MI included in the motif image group 800 to the terminal device 100. The terminal device 100 displays a selection screen (not shown) including thumbnail images of multiple motif images MI on the display unit 140. The user inputs a selection instruction to select one motif image MI, for example, by tapping a preferred thumbnail image on the selection screen. The terminal device 100 transmits information indicating the motif image MI selected by the user to the image processing server 200. The image processing server 200 acquires the motif image MI selected by the user in accordance with the information transmitted from the terminal device 100. For example, it is assumed that the motif image MI2 in FIG. 4(A) is acquired as the motif image MI to be used.

[0037] In S110, the image processing server 200 acquires a text PTf (prompt) for a decorative character to be input to the image generation model 900. In this embodiment, the text PTf is predetermined text. For this purpose, for example, the image processing server 200 acquires the predetermined text PTf from the non-volatile storage device 230. In this embodiment, the text PTf is, for example, "letter decorated with flowers."

[0038] In S115, the image processing server 200 generates a font image FI. The font image FI is an image representing the character to be processed. The character represented by the font image FI has the shape of the font represented by the font information. If the character to be processed is multiple characters, the font images FI are generated for each character. For example, if the character to be processed is "HELLO," five font images FI representing each character are generated. The data of each font image FI is generated in a processing data format accepted by the image generation model 900. That is, the font image FI is RGB image data and has the size (number of vertical and horizontal pixels) of the input image II described above. FIG. 4(B) shows a font image FI representing the character "E" TX. For example, the color of the character in the font image FI is black, and the color of the part other than the character (i.e., the background BG) is white.

[0039] In S120, the image processing server 200 generates a character outline image OLI. The character outline image OLI is an image showing the outlines OL of characters shown in the font image FI. The image processing server 200 generates the character outline image OLI by, for example, performing edge extraction processing on the data of the font image FI. For the edge extraction processing, a known edge extraction processing, for example, the function "findContours" of OpenCV (Open Source Computer Vision Library), can be used. FIG. 4(C) shows the character outline image OLI showing the outlines OL of the font image FI of FIG. 4(B).

[0040] In S125, the image processing server 200 generates a character mask image MKI using the font image FI. The character mask image MKI is a binary image obtained by setting the values ​​of pixels constituting the characters of the font image FI to "0" and replacing the values ​​of pixels constituting the background of the font image FI with "1." Fig. 4(D) shows the character mask image MKI generated using the font image FI of Fig. 4(B). The character mask image MKI is an image showing the character area TA and background area TB of the font image FI.

[0041] In S130, the image processing server 200 generates a text motif image TMI using the text mask image MKI and the selected motif image MI. The text motif image TMI is an image in which the motif image MI is synthesized into the text area TA indicated by the text mask image MKI, but the motif image MI is not synthesized into the background area BA. Figure 4(E) shows a text motif image TMI generated using the text mask image MKI of Figure 4(D) and the motif image MI of Figure 4(A).

[0042] Specifically, the image processing server 200 determines the relative position of the character mask image MKI relative to the motif image MI to be used, for example, the motif image MI2 in FIG. 4A. The character mask image MKI is positioned at the determined position within the motif image MI2 in FIG. 4A, as indicated by a dashed line. In this embodiment, the position is automatically determined to a predetermined position. Alternatively, the position may be automatically determined to a random position within the motif image MI2. The image processing server 200 positions the character mask image MKI at the determined position relative to the motif image MI2. In this state, the image processing server 200 combines the portion of the motif image MI2 that overlaps with the character region TA of the character mask image MKI with the character region TA of the character mask image MKI. The image processing server 200 then paints the background region BA of the character mask image MKI solid white. This generates a character motif image TMI. In this way, the process of determining the relative position of the character mask image MKI with respect to the motif image MI2 can be said to be a process of determining the portion of the motif image MI2 that should be combined with the character area TA.

[0043] In S135, the image processing server 200 generates an input image IIf for decorative characters by combining the character outline image OLI with the character motif image TMI. Fig. 4(F) shows the input image IIf obtained by combining the character outline image OLI of Fig. 4(C) with the character motif image TMI of Fig. 4(E). In the input image IIf of Fig. 4(F), an outline OL is added to the character area TA.

[0044] The above-described processes of S120-S135 are performed for each of the five font images FI (font images FI for each character of "HELLO") generated in S115. Therefore, at this point, input images IIf for each character of "HELLO" have been generated.

[0045] In S140, the image processing server 200 inputs input information including an input image IIf for decorated characters, text PTf, and a seed value to the image generation model 900 to generate decorated character images OIf. The seed value is a random number for adding variation to the generated image. For example, n different random numbers (n is an integer equal to or greater than 2) are used as the seed value. When there are k characters to be processed, the image processing server 200 generates n decorated character images OIf for each character, for a total of (k×n) decorated character images OIf. The image processing server 200 generates (k×n) decorated character images OIf by inputting the (k×n) pieces of input information one by one to the image generation model 900. When there are five characters to be processed, "HELLO," (5×n) decorated character images OIf are generated.

[0046] For example, n pieces of input information for generating n decorated character images of "E" each include an input image IIf (FIG. 4(F)) representing the character "E," text PTf, and one of n types of seed values. The image processing server 200 inputs the n pieces of input information to the image generation model 900, respectively, to generate decorated character images OIf representing n decorated character images DT of "E." FIG. 5 is an explanatory diagram of the decorated character image OIf. FIG. 5(A) shows images OIfa and OIfb, which are part of the n decorated character images OIf of "E." As described above, the two decorated character images OIfa and OIfb are generated using input information that differs only in the seed value.

[0047] The decorated character image OIfa is an image representing the decorated character DTa. The decorated character DTa is composed of decorative elements Da (Da1-Da7) representing flowers and elements STa and LFab representing stems and leaves. The decorated character image OIfb is an image representing the decorated character DTb. The decorated character DTb is composed of decorative elements Db (Db1-Db7) representing flowers and elements STb and LFb representing stems and leaves. The decorated characters DTa and DTb differ in size and detailed shape of the decorative elements Da and Db and the elements STa, STb, LFab, and LFb representing stems and leaves.

[0048] FIG. 5B shows the input image IIf used to generate the decorated character images OIfa and OIfb of FIG. 5A. The input image IIf of FIG. 5B includes a character area TA indicating a character specified by character information and a plurality of circle marks RM (RM1-RM7) arranged to overlap the character area TA. Decorative elements Da1-Da7 indicating flowers in the decorated character image OIfa of FIG. 5A correspond to the circle marks RM1-RM7 in the input image IIf of FIG. 5B. For example, the positions of the decorative elements Da1-Da7 in the decorated character DTa of the decorated character image OIfa generally coincide with the positions of the circle marks RM1-RM7 in the character area TA of the input image IIf. Similarly, decorative elements Db1-Db7 indicating flowers in the decorated character image OIfb of FIG. 5A correspond to the circle marks RM1-RM7 in the input image IIf of FIG. 5B. For example, the positions of decorative elements Db1-Db7 in decorated character DTb of decorated character image OIfb generally coincide with the positions of circle marks RM1-RM7 in character area TA of input image IIf.

[0049] In this way, the position of the circle mark RM in the character area TA of the input image IIf can be said to specify the positions at which the decorative elements Da and Db will be placed relative to the characters in the generated decorated character image OIf. In other words, the input image IIf includes information specifying the positions at which the decorative elements Da and Db will be placed relative to the characters.

[0050] Furthermore, the colors of decorative elements Da1-Da7 representing flowers in the decorative character image OIfa in Fig. 5(A) are similar to the colors of the corresponding circle marks RM1-RM7 in the input image IIf in Fig. 5(B). For example, the color of decorative element Da1 is similar to the color of circle mark RM1, and the color of decorative element Da3 is similar to the color of circle mark RM3. The same is true for the colors of decorative elements Db1-Db7 in the decorative character image OIfb in Fig. 5(A).

[0051] In this way, it can be said that the color of the circle mark RM in the character area TA of the input image IIf specifies the colors of the decorative elements Da and Db in the decorative character image OIf to be generated. In other words, the input image IIf includes information specifying the colors of the decorative elements Da and Db.

[0052] Furthermore, the contour shape of the decorated character DTa in the decorated character image OIfa in Figure 5(A) is formed, for example, by elements STa and LFab, which mainly represent stems and leaves. The contour shape of the decorated character DTa is similar to the contour shape of the character area TA indicated by the contour line OL in the input image IIf. The same is true for the contour shape of the decorated character DTb in the decorated character image OIfb.

[0053] 3, the image processing server 200 displays a selection screen WI1 including n types of decorative character images OIf on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the selection screen WI1 and transmits the generated data to the terminal device 100. The terminal device 100 displays the selection screen WI1 on the display unit 140 using the received data.

[0054] 6A and 6B are diagrams showing examples of a selection screen and an output screen. The selection screen WI1 in Fig. 6A includes n types of generated decorative character images OIf (e.g., OIfa, OIfb), a message MS1, a selection cursor CS, a selection button BT1, and a redo button BT2. The message MS1 prompts the user to select a desired character from the n types of decorative character images OIf.

[0055] In S150, the image processing server 200 receives a selection instruction from the user via the terminal device 100 to select a decorative character image OIf to be output. Specifically, the user moves the cursor CS on the selection screen WI1 of FIG. 6(A) to select a desired decorative character image OIf, and taps the selection button BT1 to input the selection instruction for the decorative character image OIf. Upon receiving the selection instruction for the decorative character image OIf, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected decorative character image OIf, to the image processing server 200. Note that when the redo button BT2 on the selection screen WI1 is tapped, the decorated character generation process of FIG. 3 is executed from the beginning, for example.

[0056] In S155, the image processing server 200 displays an output screen WI2 including a decorative character image to be output on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the output screen WI2 including the decorative character image OIf of the type selected by the selection instruction, and transmits the generated data to the terminal device 100. The terminal device 100 displays the output screen WI2 on the display unit 140 using the received data.

[0057] As shown in FIG. 6(B), the output screen WI2 includes an output image TI, a print button BT3, and a redo button BT4. The output image TI includes five decorated character images OIf representing the five characters of "HELLO" to be processed. The five decorated character images OIf are images generated using the same seed value as the seed value used to generate the decorated character image OIf selected by the user. The user taps the print button BT3 on the output screen WI2 when checking and printing the output image TI. When the print button BT3 is tapped, the process proceeds to S160. When the redo button BT4 is tapped on the output screen WI2, for example, the decorated character generation process of FIG. 3 is executed from the beginning.

[0058] In S160, the image processing server 200 generates a print job for printing an output image TI (FIG. 6B) including the decorative character image OIf. In S160, the image processing server 200 transmits the print job to the printer 300, causing the printer 300 to print the output image TI. Specifically, the image processing server 200 may transmit the print job to the printer 300 via the terminal device 100, or may transmit the print job directly to the printer 300 without going through the terminal device 100. Upon receiving the print job, the printer 300 prints the output image TI including the decorative character image OIf on paper in accordance with the print job.

[0059] According to the first embodiment described above, the system 1000 acquires character information for specifying characters based on a start command input by a user (S100 in FIG. 3), and acquires a motif image MI as decoration information (S105 in FIG. 3). The motif image MI is used to generate an input image IIf, and includes a circle mark RM that specifies the positions of the decorative elements Da and Db that make up the decoration relative to the characters in the input image IIf (FIG. 4(A)). The system 1000 inputs input information to the image generation model 900 using the character information and the motif image MI, causing the image generation model 900 to generate a decorated character image OIf (S115-S140 in FIG. 3). The system 1000 displays the decorated character image OIf on the display unit 140 of the terminal device 100 (S155 in FIG. 3). The system 1000 prints an output image TI including the decorated character image OIf displayed on the display unit 140 on paper (S160 and S165 in FIG. 3). As a result, the image generation model 900 can be used to generate a decorated character image OIf based on character information and the input image IIf. For example, a decorated character image OIf can be generated in which decorative elements Da and Db are arranged at positions specified by the input image IIf. Furthermore, the display of the decorated character image OIfa and the formation of an output image by printing or embroidery are executed, so that a decorated character based on the character information and decorative information can be provided to the user. For example, if the font image FI is input directly into the image generation model 900, it is not known where and how the decorative elements will be arranged, and it is therefore impossible to predict what kind of decorated character image will be generated. According to this embodiment, a decorated character image OIf with a good appearance in which the decorative elements Da and Db are appropriately arranged can be generated, compared to when the font image FI is input directly into the image generation model 900.

[0060] Furthermore, according to this embodiment, the input image IIf includes a character area TA and a plurality of circle marks RM (RM1-RM7), which are related color objects arranged so as to overlap the character area TA (FIG. 5(B)). The circle marks RM are objects having colors related to the decorative elements Da and Db ("flowers" in this embodiment). As a result, for example, the colors of the decorative elements Da and Db can be specified by the color of the circle marks RM. Therefore, a good-looking decorated character image OIf can be generated in which decorative elements Da and Db of appropriate colors are arranged.

[0061] Furthermore, according to this embodiment, the system 1000 generates a character mask image MKI indicating the character area TA based on character information (S125 in FIG. 3), and then combines the motif image MI with the character area TA specified by the character mask image MKI (S130 in FIG. 3) to generate an input image IIf. As a result, an input image IIf is generated in which the motif image MI including the circle mark RM is combined with the character area TA (FIG. 5(B)). Therefore, the input image IIf indicating the positions where the decorative elements Da and Db should be placed using the circle mark RM can be generated using the motif image MI. Furthermore, the system 1000 automatically determines the combination portion of the motif image MI to be combined with the character area TA (S130 in FIG. 3). As a result, the input image IIf can be easily generated.

[0062] Furthermore, according to this embodiment, the system 1000 selects one motif image MI from among the motif images MI included in the motif image group 800 based on a user instruction (S105 in FIG. 3), and then combines the selected motif image MI with the character area TA to generate an input image IIf (S130 in FIG. 3). As a result, a decorated character image OIf can be generated according to the user's intention.

[0063] Furthermore, according to this embodiment, the input image IIf includes a line (outline OL) indicating the outline of the character area TA (S135 in FIG. 3, FIG. 5B). If the input image IIf did not include the outline OL of the character area TA, the boundary between the character area TA and the background area BA might be unclear. In this case, the outline of the decorated character DT in the generated decorated character image OIf might be unclear, or the outline of the decorated character DT might have a shape far removed from the character, which would likely reduce the readability of the decorated character DT. According to this embodiment, the input image IIf includes the outline OL, which makes it easier for the outline of the decorated character DT to maintain the shape of the character, thereby improving the readability of the decorated character DT. For example, in the example of the decorated character images OIfa and OIfb in FIG. 5A, the outlines of the decorated characters DTa and DTb are formed into the shape of the character by elements STa, STb, LFa, and LFb, which mainly represent stems and leaves, and therefore the readability of the decorated characters DTa and DTb is high.

[0064] Furthermore, according to this embodiment, the acquired character information includes font information indicating the type of font. The shape of the character area TA of the input image IIf has the shape of the font indicated by the font information (FIG. 5(A)). As a result, the type of font can be specified by the font information, so that decorated character images OIf representing decorated characters DT of various shapes can be easily generated.

[0065] Furthermore, according to this embodiment, the system 1000 acquires character information specifying k characters (e.g., "HELLO") (where k is an integer greater than or equal to 2) and generates k pieces of input information corresponding to the k characters. The system 1000 generates k decorated character images OIf by inputting the k pieces of input information one by one into the image generation model 900, and prints an output image TI including the k decorated character images OIf (FIG. 6B). The size (vertical or horizontal) of the input image II and output image OI of the image generation model 900 is a predetermined fixed value. For this reason, if a single decorated character image including multiple decorated characters is generated, the number of pixels constituting one decorated character may decrease, resulting in a decrease in the resolution of the decorated character and a decrease in the image quality of the decorated character. According to this embodiment, such inconveniences can be suppressed and an output image TI showing each decorated character DT with sufficient resolution can be generated and output.

[0066] Furthermore, according to this embodiment, the system 1000 generates n types of input information using one piece of character information and a motif image MI, and generates n types of decorated character images OIf (e.g., OIfa, OIfb) by inputting the n types of input information one by one into the image generation model 900 (FIG. 5(A)). The system 1000 displays the n types of decorated character images OIf on the display unit 140 of the terminal device 100 (S145 in FIG. 3, FIG. 6(A)). The system 1000 prints a decorated character image selected by the user from the n types of decorated character images OIf (S155-S165 in FIG. 3, FIG. 6(B)). As a result, the decorated character image OIf selected by the user from the n types of decorated character images OIf generated using one piece of character information and a motif image MI is printed, so that a decorated character image OIf that is more in line with the user's preferences can be printed.

[0067] As can be seen from the above explanation, the motif image MI in this embodiment is an example of decorative information and a specific image. The circle mark RM in this embodiment is an example of a related color object, and the character mask image MKI is an example of an area image. The outline OL in this embodiment is an example of a first outline.

[0068] B. Second Example In the second embodiment, the contents of the decorated character generation process are different from those in the first embodiment. Fig. 7 is a flowchart of the decorated character generation process in the second embodiment.

[0069] In S210, the image processing server 200 displays a sample selection screen WI3 including a plurality of sample decorative character images OIs on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the sample selection screen WI3 and transmits the generated data to the terminal device 100. The terminal device 100 displays the sample selection screen WI3 on ​​the display unit 140 using the received data.

[0070] Fig. 8 is a diagram showing an example of a sample selection screen in the second embodiment. The sample selection screen WI3 in Fig. 8 includes a plurality of sample decorative character images OIs (e.g., OIs1-OIs3), a message MS3, a selection cursor CS, and selection buttons BTs. The message MS3 is a message that prompts the user to select a desired character from the plurality of sample decorative character images OIs.

[0071] Each of the sample decorated character images OIs corresponds one-to-one to one of the motif images MI included in the motif image group 800. For example, sample decorated character image OIs1 in FIG. 8 corresponds to motif image MI1 in FIG. 4(A). Sample decorated character image OIs2 corresponds to motif image MI2 in FIG. 4(A). Sample decorated character image OIs3 corresponds to motif image MI3 in FIG. 4(A).

[0072] In the second embodiment, data of each sample decorated character image OIs is generated in advance using a corresponding motif image MI1 and stored in the non-volatile storage device 230 of the image processing server 200. Specifically, data of the sample decorated character image OIs is generated by executing S120-S140 of FIG. 3 using a font image FI of a specific character ("A" in the example of FIG. 8) of a specific font, the corresponding motif image MI, and predetermined text PTf. The predetermined text PTf is, for example, "letter decorated with flowers," as in the first embodiment. Therefore, for example, the sample decorated character image OIs1 is generated by inputting input information including an input image IIf generated using the motif image MI1 into the image generation model 900.

[0073] For this reason, the shape, position, size, and color of the decorative elements in the sample decorative character images OIs are influenced by the shape, position, size, and color of the related color objects in the corresponding motif images MI. For example, the shape, size, and color of the decorative element D1 in the sample decorative character image OIs1 are similar to the shape, size, and color of the star mark SM in the corresponding motif image MI1. The shape, size, and color of the decorative element D2 in the sample decorative character image OIs2 are similar to the shape, size, and color of the circle mark RM in the corresponding motif image MI2. The shape, size, and color of the decorative element D3 in the sample decorative character image OIs3 are similar to the shape, size, and color of the cross mark CM in the corresponding motif image MI3. In this way, it can be said that the motif images MI1-MI3 contain information for specifying the shape, size, and color of the decorative elements D1-D3. Furthermore, the color of the entire character in the sample decorative character images OIs (the color of the part excluding the decoration) is similar to the background color of the motif images MI1-MI3.

[0074] In S220, the image processing server 200 selects one sample decorated character based on a selection instruction from the user. Specifically, the user moves the cursor CS on the sample selection screen WI3 of FIG. 8 to select one sample decorated character image OIs of their choice, and taps the selection button BTs to input a selection instruction for the sample decorated character image OIs. Upon receiving the selection instruction for the sample decorated character image OIs, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected sample decorated character image OIs, to the image processing server 200. The image processing server 200 selects the sample decorated character image OIs designated by the user in accordance with the received information.

[0075] In S230, the image processing server 200 acquires the motif image MI and text PTf corresponding to the selected sample decorative character image OIs. Specifically, in the second embodiment, as shown in Fig. 1, the non-volatile storage device 230 of the image processing server 200 pre-stores first correspondence information 710 in addition to the information similar to that in the first embodiment.

[0076] The first correspondence information 710 is information that records each sample decorated character image OIs in association with the motif image MI and text PTf used to generate the sample decorated character image OIs. The image processing server 200 references the first correspondence information 710 to acquire the motif image MI and text PTf (in this embodiment, "letter decorated with flowers") used to generate the selected sample decorated character image OIs.

[0077] In S240, the image processing server 200 acquires text information from the user via the terminal device 100, similar to S100 in FIG.

[0078] In S250, the image processing server 200 executes the processes of S115-165 in Fig. 3. That is, the image processing server 200 generates a decorated character image OIf using the character information acquired in S240 and the motif image MI and text PTf acquired in S230, and displays or prints the generated decorated character image OIf.

[0079] According to the second embodiment described above, the system 1000 selects one sample decorated character image OIs based on a user instruction from among a plurality of sample decorated character images OIs, including a first decorated character image OIs1 generated by inputting input information generated using a first motif image MI1 into the image generation model 900, and a second decorated character image OIs2 generated by inputting input information generated using a second motif image MI2 into the image generation model 900 (S220 in FIG. 7). If the first decorated character image OIs1 is selected, the image processing server 200 acquires the first motif image MI1, and if the second decorated character image OIs2 is selected, the image processing server 200 acquires the second motif image MI2 (S230 in FIG. 7). When the first decorative character image OIs1 is selected, the system 1000 generates input information using the first motif image MI1, and when the sample decorative character image OIs2 is selected, the system 1000 generates input information using the second motif image MI2, and inputs the generated input information into the image generation model 900 (S250 in Figure 7).

[0080] As a result, it is possible to generate and print a decorative character image OIf that meets the user's preferences. For example, even when the user looks at the motif images MI (MI1-MI3), the user may not be able to imagine the decorative character image OIf that will be generated. In this case, even if the user is prompted to select a motif image MI to use, the user may not be able to select an appropriate motif image MI that will enable the user to generate the decorative character image that the user desires. In this embodiment, the user simply selects a sample decorative character image OIs in a desired style from multiple sample decorative character images OIs generated using each of multiple motif images MI. This allows the user to display and print a decorative character image OIf in a desired style for the desired characters. Therefore, it is possible to easily display and print a decorative character image OIf that meets the user's intentions.

[0081] As can be seen from the above explanation, the first motif image MI1 of this embodiment is an example of first decorative information, and the second motif image MI2 is an example of second decorative information.

[0082] C. Third Example In the third embodiment, the contents of the decorated character generation process are different from those in the first embodiment. Fig. 9 is a flowchart of the decorated character generation process in the third embodiment.

[0083] In S310, the image processing server 200 acquires character information from the user via the terminal device 100, similar to S100 in Fig. 3. The acquired character information includes information indicating the character to be processed and font information indicating the type of font (typeface) of the character.

[0084] In S320, similar to S110 in Fig. 3, the image processing server 200 acquires a text PTf (prompt) for a decorative character to be input to the image generation model 900. In this embodiment, the text PTf is predetermined text (for example, "letter decorated with flowers").

[0085] In S330, the image processing server 200 generates a font image FI (FIG. 4(B)) using the character information, similar to S115 in Fig. 3. In S340, the image processing server 200 generates a character outline image OLI (FIG. 4(C)), similar to S120 in Fig. 3.

[0086] In S350, the image processing server 200 executes input image generation processing. Fig. 10 is a flowchart of the input image generation processing. The input image generation processing is processing for generating an input image IIf to be input to the image generation model 900.

[0087] In S405 of Fig. 10, the image processing server 200 receives a selection instruction for selecting the amount of decoration from the user via the terminal device 100. Fig. 11 is an explanatory diagram of the amount of decoration. The upper portions of Figs. 11(A)-(C) show input elements VI for inputting a selection instruction for the amount of decoration. The terminal device 100 displays an input screen (not shown) including the input elements VI on the display unit 140.

[0088] The input element VI includes a slide bar SB and a slider SD that can be moved on the slide bar SB in response to a user operation. The position of the slider SD along the longitudinal direction of the slide bar SB corresponds to multiple levels of decoration amount. For example, in this embodiment, the amount of decoration can be specified in five levels, from level 1 to level 5. In the example of FIG. 11, the position of the slider SD in the input element VI in FIG. 11(A) corresponds to level 1. The position of the slider SD in the input element VI in FIG. 11(B) corresponds to level 3, and the position of the slider SD in the input element VI in FIG. 11(C) corresponds to level 5.

[0089] The user operates the slider SD on the input screen to move the slider SD to a position corresponding to the desired level of decoration amount, and taps the confirm button (not shown). This inputs a selection instruction to select one of five levels of decoration amount to the terminal device 100. Upon receiving the selection instruction for the decoration amount, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected decoration amount, to the image processing server 200.

[0090] In S410, the image processing server 200 acquires the number and placement positions of circle marks RM to be placed in the character area TA according to the selected amount of decoration and the character to be processed. Specifically, in the third embodiment, as shown in Fig. 1, second correspondence information 720 is pre-stored in the non-volatile storage device 230 of the image processing server 200 in addition to information similar to that in the first embodiment. The character to be processed is indicated by the character information acquired in S310 of Fig. 9.

[0091] The second correspondence information 720 is information that records the correspondence between each of the decoration amount levels 1-5 and the number and placement positions of circle marks RM to be placed for each letter of the alphabet. For example, FIG. 11(A) shows the number and placement positions of circle marks RM corresponding to decoration amount level 1 for the letters "H" and "E." At level 1, for example, one circle mark RM is placed near the upper left corner of the alphabet. FIG. 11(B) shows the number and placement positions of circle marks RM corresponding to decoration amount level 3 for the letters "H" and "E." At level 3, for example, three circle marks RM are placed dispersedly in predetermined positions determined for each letter of the alphabet. FIG. 11(C) shows the number and placement positions of circle marks RM corresponding to decoration amount level 5 for the letters "H" and "E." At level 5, for example, five circle marks RM are placed dispersedly in predetermined positions determined for each letter of the alphabet. The position of the circle mark RM is determined to be a position where a good-looking decorated character image can be generated by, for example, repeatedly conducting experiments to generate decorated character images using the image generation model 900.

[0092] The image processing server 200 refers to the second correspondence information 720 and acquires, for each character to be processed, information on the number and arrangement positions of the circle marks RM corresponding to the combination of the selected amount of decoration and the character to be processed.

[0093] In S415, the image processing server 200 generates an input image IIn using the acquired information on the number and arrangement positions of the circle marks RM. For example, if the characters to be processed are the five characters "HELLO," an input image IIn for each of the five characters is generated. In S420, an adjustment instruction input screen WI4 including the generated input image IIn is displayed on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the adjustment instruction input screen WI4 and transmits the generated data to the terminal device 100. The terminal device 100 displays the adjustment instruction input screen WI4 on the display unit 140 using the received data.

[0094] FIG. 12 is a diagram showing an example of the adjustment instruction input screen WI4. The adjustment instruction input screen WI4 includes one or more input images IIn generated in S415. As shown in FIG. 12, the input images IIn (IIn1, IIn2) include character areas TA (TA1, TA2) and circle marks RM arranged so as to overlap the character areas TA. The number and arrangement positions of the circle marks RM are determined in S410 according to information acquired from the second correspondence information 720 in accordance with the amount of decoration. An outline OL is added to the character areas TA, and the color of the character areas TA is set to a default color (e.g., black or white).

[0095] The adjustment instruction input screen WI4 further includes messages MS4 and MS5, change buttons BTa-BTc, and a complete button BTf. The message MS4 indicates that the position, color, size, and character color of the circle mark RM can be changed. The message MS5 indicates that the position, color, and size of the circle mark RM will be reflected in the decoration position, color, and size of the generated decorative character image OIfa.

[0096] For example, the user can input a change instruction to change the position of the circle mark RM relative to the character area TA by touching and moving the circle mark RM with a finger on the adjustment instruction input screen WI4. When the user taps one circle mark RM to select it on the adjustment instruction input screen WI4 and then taps the change button BTa, a UI screen (not shown) for specifying the size of the circle mark RM after the change is displayed as a pop-up. The user can input an instruction to change the size of the circle mark RM via the UI screen. When the user taps one circle mark RM to select it on the adjustment instruction input screen WI4 and then taps the change button BTb, a UI screen (not shown) such as a color palette for specifying the color of the circle mark RM after the change is displayed as a pop-up. The user can input an instruction to change the color of the circle mark RM via the UI screen. When the user taps one character area TA to select it on the adjustment instruction input screen WI4 and then taps the change button BTc, a UI screen (not shown) such as a color palette for specifying the color of the character area TA after the change is displayed as a pop-up. The user can input an instruction to change the color of the character area TA (hereinafter also referred to as character color) via the UI screen. The change instruction input to the terminal device 100 is transmitted to the image processing server 200 .

[0097] 10, the image processing server 200 determines whether or not an instruction to change the color of the text or the color of the circle mark RM has been acquired. If an instruction to change the color of the text or the color of the circle mark RM has not been acquired (S425: NO), the image processing server 200 proceeds to S450. If an instruction to change the color of the text or the color of the circle mark RM has been acquired (S425: YES), the image processing server 200 proceeds to S430.

[0098] In S430, when the image processing server 200 changes the color of the character or the color of the circle mark RM in accordance with the change instruction, the image processing server 200 determines whether the color of the character in the input image IIn is similar to the color of the circle mark RM. For example, if an index value indicating the difference between the color of the character and the color of the circle mark RM is less than a predetermined threshold THc, the image processing server 200 determines that the color of the character and the color of the circle mark RM are similar, and if the index value is equal to or greater than the predetermined threshold THc, the image processing server 200 determines that the color of the character and the color of the circle mark RM are not similar. The index value of the difference is, for example, the Euclidean distance in a predetermined color space (e.g., RGB color space or Lab color space) between the color of the character and the color of the circle mark RM.

[0099] If the character color and the color of the circle mark RM are similar (S430: YES), in S435 the image processing server 200 adds a contour line MOL to the circle mark RM having a color similar to the character color in the input image IIn. If the character color and the color of the circle mark RM are not similar (S430: YES), in S440 the image processing server 200 does not add a contour line MOL to the circle mark RM in the input image IIn.

[0100] In S445, the image processing server 200 changes the color of the text or the color of the circle mark RM in the input image IIn in accordance with the change instruction. When the input image IIn is updated, such as by adding a contour line MOL or changing the text color or the circle mark RM, the updated data is transmitted from the image processing server 200 to the terminal device 100, and the update is reflected in real time on the adjustment instruction input screen WI4 displayed on the display unit 140.

[0101] FIG. 13 is a first explanatory diagram of the input image generation process. In the input image IIna of FIG. 13(A), the character color (the color of the character area TA) and the color of the circle mark RM are not similar. In this case, no outline is added to the circle mark RM in the input image IIna. In the input image IInb of FIG. 13(B), the character color (the color of the character area TA) and the color of the circle mark RM are similar. In this case, an outline MOL is added to the circle mark RM in the input image IInb. A color that is not similar to the color of the character and the color of the circle mark RM is used as the color of the outline MOL. The color of the outline MOL is, for example, predetermined for each expected character color and circle mark RM.

[0102] 10, the image processing server 200 determines whether or not an instruction to change the position or size of the circle mark RM has been acquired. If an instruction to change the position or size of the circle mark RM has not been acquired (S450: NO), the image processing server 200 proceeds to S470. If an instruction to change the position or size of the circle mark RM has been acquired (S450: YES), the image processing server 200 proceeds to S455.

[0103] In S455, the image processing server 200 determines whether the character area TA and the circle mark RM overlap in the input image IIn when the position or size of the circle mark RM is changed in accordance with the change instruction. For example, when a change instruction to move the circle mark RM in a direction away from the character area TA is acquired, the change in the position of the circle mark RM may cause the character area TA and the circle mark RM to no longer overlap. Also, when a change instruction to reduce the size of the circle mark RM that slightly overlaps the character area TA is acquired, the change in size of the circle mark RM may cause the character area TA and the circle mark RM to no longer overlap.

[0104] If the character area TA and the circle mark RM overlap (S455: YES), in S460 the image processing server 200 changes the position or size of the circle mark RM in the input image IIn in accordance with the change instruction. When the input image IIn is updated by changing the position or size of the circle mark RM, the updated data is transmitted from the image processing server 200 to the terminal device 100, and the update is reflected in real time on the adjustment instruction input screen WI4 displayed on the display unit 140.

[0105] If the character area TA and the circle mark RM do not overlap (S455: NO), in S465, the image processing server 200 displays an error message on the adjustment instruction input screen WI4. The error message may include, for example, the following text: "The circle mark cannot be moved to a position where it does not overlap with the characters." For example, the image processing server 200 transmits an instruction to display an error message to the terminal device 100, and the terminal device 100 displays the error message on the adjustment instruction input screen WI4 displayed on the display unit 140 in accordance with the display instruction. Note that when the error message is displayed, the position and size of the circle mark RM are not changed in accordance with the acquired change instruction. As a result, the character area TA and each circle mark RM are maintained in an overlapping state on the adjustment instruction input screen WI4.

[0106] Fig. 14 is a second explanatory diagram of the input image generation process. The input image IInc in Fig. 14 includes a character area TA and circle marks RMo and RMn. For example, if a change instruction to move the circle mark RMo to position P1 indicated by the dashed line is acquired, the circle mark RMo overlaps with the character area TA at the changed position P1. For this reason, in this case, the position of the circle mark RMo is changed to position P1 (YES in S455 of Fig. 10, S460).

[0107] On the other hand, if a change instruction to move the circle mark RMn to position P2 indicated by the broken line is acquired, the circle mark RMn will not overlap the character area TA at the changed position P2. For this reason, in this case, the position of the circle mark RMn is not changed to position P2, and an error message is displayed (NO in S455 of FIG. 10, S465).

[0108] In S470 of Fig. 10, the image processing server 200 determines whether or not the user has completed adjustments to the color, size, position, etc. of the circle mark RM. When the user has completed these adjustments, the user taps the Done button BTf on the adjustment instruction input screen WI4 of Fig. 12. Therefore, the image processing server 200 determines that the user's adjustments have been completed when the Done button BTf is tapped.

[0109] If the user's adjustments are complete (S470: YES), the image processing server 200 acquires data of the final input image IIn after adjustments in S475, and ends the input image generation process. If the user's adjustments are complete (S470: NO), the image processing server 200 returns the process to S425.

[0110] In S380 of Fig. 9 after the input image generation process is completed, the image processing server 200 executes the processes of S140-165 of Fig. 3. That is, the image processing server 200 generates a decorated character image OIf using the final input image IIna acquired in S475 of Fig. 10, the character information, and the text PTf acquired in S330, and then displays or prints the generated decorated character image OIf.

[0111] According to the third embodiment described above, the system 1000 allows the user to input an instruction to specify the position of the circle mark RM relative to the character area TA via the adjustment instruction input screen WI4 (FIGS. 12 and S450 in FIG. 10). The system 1000 generates an input image IIn in which the circle mark RM is arranged at the position specified by the instruction relative to the character area TA (S460 and S470 in FIG. 10). The system 1000 inputs input information including the generated input image IIn to the image generation model 900 (S380 in FIG. 9 and S140 in FIG. 3). As a result, by allowing the user to specify the position of the circle mark RM relative to the character area TA on the adjustment instruction input screen WI4, it is possible to generate and print a decorated character image OIf decorated according to the user's intentions using the image generation model 900.

[0112] Furthermore, according to this embodiment, the system 1000 allows input of an instruction specifying a position of the circle mark RM that overlaps the character area TA, but does not allow input of an instruction specifying a position that does not overlap the character area TA (S450-S465 in FIG. 10). If an input image IIn is generated in which the circle mark RM and the character area TA are arranged so that they do not overlap, an appropriate decorated character image OIf may not be generated. For example, decoration may appear in a position unrelated to the characters in the decorated character image OIf, which may reduce the readability of the characters. According to this embodiment, the inconvenience of generating an input image IIn in which the circle mark RM and the character area TA are arranged so that they do not overlap can be suppressed.

[0113] Furthermore, according to this embodiment, the system 1000 includes a storage area (non-volatile storage device 230) for storing second correspondence information 720 that associates the amount of decoration with the number and positions of circle marks RM to be placed in the character area TA (FIG. 1). The system 1000 acquires an instruction specifying the amount of decoration from the user (S405 in FIG. 10), and acquires the number and positions of circle marks RM associated with the amount of decoration specified by the user's instruction by referring to the second correspondence information 720 (S410 in FIG. 10). The system 1000 generates an input image IIn using the acquired number and positions of circle marks RM (S415 in FIG. 10). For example, a user with little knowledge of design may not be able to appropriately place circle marks RM in the character area TA. In this case, if the placement of circle marks RM were left entirely up to the user, there is a high possibility that an attractive decorated character image OIf would not be generated. According to this embodiment, the user simply specifies the amount of decoration, and an input image IIn is easily generated in which the circle mark RM is placed in an appropriate position. Therefore, an attractive decorated character image OIf can be easily generated. Furthermore, even when the user specifies the circle mark RM, the circle mark RM can be placed in an appropriate initial position relative to the character area TA, so the user can easily adjust the position of the circle mark RM according to their own preferences. Therefore, the burden on the user for generating the decorated character image OIf can be reduced.

[0114] Furthermore, according to this embodiment, the system 1000 receives an instruction from the user to specify at least one of the color of the circle mark RM and the color (character color) of the character area TA (S425 in FIG. 10). If the color of the circle mark RM and the color of the character are similar (YES in S430 in FIG. 10), the system 1000 adds a contour line MOL having a color different from the color of the circle mark RM to the circle mark RM (S435 in FIG. 10). The system 1000 generates an input image IIn including the circle mark RM to which the contour line MOL has been added and the character area TA (S475 in FIG. 10). As a result, the user can easily distinguish and visually recognize the character area TA and the circle mark RM on the adjustment instruction input screen WI4. Therefore, the user can check the position of the circle mark RM relative to the character area TA and set the position and size of the circle mark RM so that an appropriate input image IIn is generated. Furthermore, when generating the decorated character image OIf, an input image IIn that clearly distinguishably represents the character area TA and the circle mark RM is input to the image generation model 900. Therefore, it is possible to increase the likelihood that a decorative element (e.g., a flower) corresponding to the circle mark RM will be appropriately added to the decorated character in the generated decorated character image OIf, and therefore a good-looking decorated character image OIf can be generated.

[0115] As can be seen from the above explanation, in this embodiment, the information on the number and positions of the circle marks RM acquired from the second correspondence information 720, and the instructions on the position, color, and size of the circle marks RM acquired from the user are examples of decorative information. Also, the outline MOL of the circle mark RM in this embodiment is an example of a second outline.

[0116] D. Variations: (1) As shown in FIG. 1, the system 1000 may include a sewing machine 400 and a cutting machine 500. The sewing machine 400 is a device that forms an image on fabric by embroidering the fabric with thread according to embroidery data. The cutting machine 500 is a device that expresses an image in the shape of a sheet-like material by cutting the sheet-like material with a blade according to cut data. The sheet-like material may be, for example, paper, cloth, a resin sheet such as urethane foam, or leather.

[0117] For example, in the above embodiment, the system 1000 causes the printer 300 to print the output image TI including the decorative character image OIf. Alternatively, the system 1000 may cause the sewing machine 400 to embroider the output image TI. The system 1000 may also cause the cutting machine 500 to cut the output image TI. Alternatively, the system 1000 may express the output image TI using a combination of these devices. For example, the system 1000 may use the printer 300 to print the output image TI on a sheet material, and then use the cutting machine 500 to cut the sheet material on which the output image TI is printed along the outline of the decorative character DT included in the output image TI.

[0118] (2) In the above embodiments, the decorative element of the decorative character image OIf is a "flower," but this is not limited to this. The decorative element may be something other than a flower, such as a living thing like a star or a butterfly, or a plant like a fruit. The decorative element may also be a rose, cherry blossom, or other sub-concept of a flower. It is preferable to use different text PT depending on the decorative element. For example, if the decorative element is a rose, the text PT should be "letter decorated with roses." It is also preferable to use a color related to the decorative element as the color of the related color object (e.g., a circle mark RM). For example, if the decorative element is a rose, it is preferable to use a color commonly known as the color of a rose, such as red, white, or yellow, as the color of the circle mark RM.

[0119] (3) In the first embodiment, the image processing server 200 selects one motif image MI from the motif image group 800 to be used to generate the input image IIf based on a user instruction (S105 in FIG. 3). Alternatively, the image processing server 200 may automatically select one motif image MI from the motif image group 800. The automatic selection method may be, for example, a random selection method or a selection method according to a predetermined order.

[0120] In the first embodiment, the motif image MI used to generate the input image IIf is selected from the motif image group 800 stored in advance in the nonvolatile storage device 230 of the image processing server 200. Alternatively, the motif image MI may be selected from images (e.g., photographic images) prepared by the user and stored in the nonvolatile storage device 130 of the terminal device 100.

[0121] (4) In the first embodiment described above, when generating an input image IIf by combining a portion of the motif image MI with the character area TA (S130 in FIG. 3), the portion of the motif image MI to be combined with the character area TA is automatically determined by placing a character mask image MKI at a predetermined position relative to the motif image MI. Alternatively, the image processing server 200 may determine the portion to be combined based on a user instruction acquired via the terminal device 100. In this case, for example, the terminal device 100 displays on the display unit 140 a predetermined UI screen showing the motif image MI and the character area TA superimposed on each other. The user may, for example, move the character area TA on the UI screen to input the relative position of the character area TA with respect to the motif image MI.

[0122] In the first embodiment, the decorative information (information indicating the position where the decorative element is placed relative to the character) is a composite portion to be composited into the character area TA of the motif image MI, and this decorative information is acquired based on user input. As can be seen from the explanations of the above modifications (3) and (4), "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI selected based on instructions input by the user and the position of the composite portion determined based on instructions input by the user. Furthermore, "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI selected based on instructions input by the user and the position of the composite portion determined automatically (e.g., randomly). Furthermore, "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI selected automatically (e.g., randomly) and the position of the composite portion determined automatically (e.g., randomly) when an instruction to start the decorated character generation process is input.

[0123] (5) In the second embodiment, the decoration information associated with the sample decorated character image OIs in the first correspondence information 710 is the motif image MI (e.g., motif images MI1-MI3 in FIG. 5). The decoration information associated with the sample decorated character image OIs is not limited to this, and may be information indicating the type and number of related color objects (e.g., circle marks RM, cross marks CM, and star marks SM) used to generate the sample decorated character image OIs. In this case, for example, when a specific sample decorated character image OIs is selected by the user, the image processing server 200 may generate the input image IIf by arranging the related color objects in the character area TA in accordance with the information associated with the specific sample decorated character image OIs.

[0124] (6) The image generation model 900 in each of the above embodiments is a machine learning model called Stable Diffusion, but is not limited to this. Various models trained to generate images decorated with decorative elements such as flowers may be used as the image generation model 900. For example, when a style image and a content image are input, the image generation model 900 may be an image generation model (e.g., a style transfer model called MST (Multimodal Style Transfer)) that generates an image in which the style of the style image is applied to the content image. In this case, for example, by inputting the input images IIf and IIn of this embodiment as content images and the sample decorated character image of this embodiment (e.g., the sample decorated character image OIs1 in FIG. 8) as a style image into the MST model, a decorated character image in which the characters shown in the input image IIf are decorated can be generated. Even in this case, the input images IIf and IIn may be generated by the processing described in each of the above embodiments. Alternatively, other well-known image generation models such as the Adain model, the StyTR2 model, or the U-GAT-IT model may be used.

[0125] Furthermore, the image generation model 900 may employ a diffusion model different from Stable Diffusion, such as a diffusion model called "VQ-Diffusion" or "GLIDE."

[0126] "VQ-Diffusion" is disclosed in the following paper: Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo, "Vector Quantized Diffusion Model for Text-to-Image Synthesis", arXiv:2111.14822v3, March 3, 2022, https: / / arxiv.org / abs / 2111.14822v3

[0127] "GLIDE" is disclosed in the following paper: Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen, "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models", arXiv:2112.10741v3, March 8, 2022, https: / / arxiv.org / abs / 2112.10741v3

[0128] (7) The decorative character generation process in each of the above embodiments is merely an example and may be modified as appropriate. For example, the character information acquired in S100 of FIG. 3 does not need to include font information indicating the type of font specified by the user. In this case, for example, the font image FI generated in S115 of FIG. 3 is generated using a predetermined font.

[0129] 3, a contour line OL is added to the character motif image TMI in Fig. 4(E) to generate the input image IIf in Fig. 4(F), but the contour line OL does not have to be added. In this case, the image processing server 200 may, for example, fill in the area of ​​the character area TA of the input image IIf that is different from the circle mark RM with a color different from the background area BA.

[0130] 3, n types of decorated character images OIf with different seed values ​​are generated for each character to be processed. Alternatively, only one type of decorated character image OIf may be generated for each character to be processed. In this case, S145 and S150 in FIG. 3 may be omitted.

[0131] 3, an input image IIf is generated for each character to be processed, and a decorated character image OIf is generated for each character. Alternatively, the image processing server 200 may generate an input image including multiple characters and input input information including the input image to the image generation model 900, thereby generating a single decorated character image including multiple decorated characters.

[0132] 10 of the third embodiment, the image processing server 200 may generate the decorated character image OIf by directly using the input image IIn generated in S420, i.e., the input image IIn in which the circle mark RM is placed at the default position. In this case, S420-475 in FIG. 10 may be omitted.

[0133] 10, in the decorative character generation process, a circle mark RM is placed at a default position relative to the character area TA in the input image IIn included in the adjustment instruction input screen WI4 (FIG. 11). Alternatively, the adjustment instruction input screen WI4 may display an input image including only the character area TA without the circle mark RM placed therein. In this case, the image processing server 200 may acquire instructions specifying the placement positions from the user multiple times and place the number of circle marks RM desired by the user at the positions specified by the user, thereby generating the input image IIn in which the circle marks RM are placed.

[0134] 10, the color, position, size, and character color of the circle mark can be changed based on user instructions, but the color, position, size, and part of the character color of the circle mark may not be changeable.

[0135] In the decorative character generation process of Fig. 10, the color of the circle mark RM and the character color can be set to colors similar to each other. Alternatively, the color of the circle mark RM and the character color may not be set to colors similar to each other. In this case, for example, the process for setting the outline MOL to the circle mark RM (S430-S440 in Fig. 10) may be omitted.

[0136] (8) In the system 1000 of each of the above embodiments, the terminal device 100 mainly executes processing related to a screen that functions as a user interface (referred to as a UI screen) and processing for controlling the printer 300. Alternatively, the terminal device 100 may execute part of the image processing executed by the image processing server 200 on behalf of the image processing server 200.

[0137] For example, the terminal device 100 may execute steps S100-S135 and steps S145-S165 of the decorated character generation process in Fig. 3. The image processing server 200 may execute only the process of generating the decorated character image OIf using the image generation model 900 in S140. The same applies to the decorated character generation process in Figs. 7 and 10.

[0138] 3 may be executed by the terminal device 100. In this case, the system 1000 may be configured with only the terminal device 100 and the printer 300, and the image processing server 200 may be omitted.

[0139] (9) In each of the above embodiments, the decorative character image OIf is generated by the user of the terminal device 100 in order to print the decorative character image OIf. However, the generation of the decorative character image OIf may also be performed by, for example, the manufacturer of the printer 300, the sewing machine 400, or the cutting machine 500. In this case, for example, a computer (terminal or server) owned by the manufacturer executes the decorative character generation process shown in FIGS. 3, 7, and 10. In this case, for example, decorative character images OIf are generated for all alphabets, and data of the decorative character images OIf are stored in the memory of the printer 300, the sewing machine 400, or the cutting machine 500 when the printer 300 or the sewing machine 400 is manufactured. The printer 300, the sewing machine 400, or the cutting machine 500 uses the data of the decorative character image OIf to print, embroider, or cut the decorative character image OIf in response to, for example, a user's operation.

[0140] (10) In the above embodiments and modifications, the processor 210 may cause the GPU 260 to execute various calculations. Note that the GPU 260 may be omitted.

[0141] (11) The processing performed by the terminal device 100 in Fig. 1 may be performed by, for example, the printer 300, the sewing machine 400, the cutting machine 500, or an image processing device such as a digital camera or a multifunction peripheral. For example, the screens WI1-WI4 in the above embodiment displayed on the display unit 140 of the terminal device 100 may be displayed on a display unit provided in these image processing devices.

[0142] In each of the above embodiments, a part of the hardware configuration may be replaced with software, and conversely, a part or all of the software configuration may be replaced with hardware. For example, the processing by the image generation model 900 (FIG. 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0143] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and DVD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0144] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof. [Explanation of symbols]

[0145] 1000...system, 100...terminal device, 110...processor, 115...storage device, 120...volatile storage device, 130...non-volatile storage device, 131...computer program, 140...display unit, 150...operation unit, 170...communication interface, 200...image processing server, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...computer program, 260...graphics processing unit, 270...communication interface, 300...printer, 400...sewing machine, 710...first correspondence information, 720...second correspondence information Report,800...motif image group,900...image generation model,910...text encoder,920...image encoder,930...latent variable model,940...image decoder,D1-D3...decorative element,DT...decorative character,Da...decorative element,Da,Db...decorative element,FI...font image,II,IIf,IIn...input image,IT...Internet,LN...local area network,MI...motif image,MKI...character mask image,OI...output image,OIf...decorative character image,OIs...sample decorated character image,OLI...character outline image,TI...output image,TMI...character motif image

Claims

1. 1. An image processing system, comprising: an information acquisition unit that acquires, based on a user's input, character information for specifying characters and decoration information for specifying positions at which decoration elements constituting the decoration are to be placed relative to the characters; an image generation unit that generates a decorated character image by inputting input information using the character information and the decorative information into an image generation model, the decorated character image being an image showing a decorated character that is the character decorated with decoration including the decorative element, and the image generation model being trained to generate an image decorated with decoration including the decorative element based on the input information; a display control unit that displays the decorative character image on a display unit; an image forming unit that prints, embroiders, or cuts the decorative character image displayed on the display unit onto an object when an instruction from a user is received; An image processing system comprising:

2. 2. The image processing system according to claim 1, the input information includes an input image including a character area indicating a character specified by the character information and a related color object arranged so as to overlap the character area; The associated color object is an object having a color associated with the decorative element.

3. 3. The image processing system according to claim 2, the decorative information includes a specific image including a plurality of the related color objects; The image generation unit generating a region image representing the character region based on the character information; synthesizing the specific image with the character region to generate the input image; An image processing system that inputs the input information, including the input image, into the image generation model.

4. 4. The image processing system according to claim 3, The information acquisition unit acquires the specific image by selecting one image from among a plurality of candidate images based on a user instruction.

5. 4. The image processing system according to claim 3, The image generation unit determining a composite portion of the specific image to be composited with the character region based on a user instruction; an image processing system that generates the input image by combining the composite portion of the specific image with the character region;

6. 2. The image processing system according to claim 1, The image processing system further comprises: a sample selection unit that selects one image from a plurality of sample decorated character images based on an instruction from a user, the sample decorated character images including a first decorated character image generated by inputting the input information generated using first decoration information into the image generation model, and a second decorated character image generated by inputting the input information generated using second decoration information into the image generation model; the information acquisition unit acquires the first decoration information when the first decorated character image is selected, and acquires the second decoration information when the second decorated character image is selected; The image generation unit When the first decorated character image is selected, the input information is generated using the first decoration information, and when the second decorated character image is selected, the input information is generated using the second decoration information; An image processing system that inputs the generated input information into the image generation model.

7. 3. The image processing system according to claim 2, the decoration information is information indicating a position where the related color object is to be placed relative to the character area; The image processing system further comprises: an input screen display unit that displays a first input screen including an image showing the character area on the display unit; the information acquisition unit acquires the decoration information by having a user input, via the input screen, a position designation instruction that designates a position of the related color object relative to the character area; The image generation unit generating the input image in which the related color object is arranged at a position designated by the designation instruction relative to the character region; an image processing system that inputs the input information, including the input image to be generated, into the image generation model;

8. 3. The image processing system according to claim 2, The image processing system, wherein the input image includes a first contour line indicating the contour of the character region.

9. 3. The image processing system according to claim 2, the character information includes font information indicating a font type; An image processing system, wherein the shape of the character region has a font shape indicated by font information.

10. 2. The image processing system according to claim 1, the information acquisition unit acquires the character information specifying k characters (k is an integer of 2 or more); The image generation unit generating k pieces of said input information corresponding to k characters; generating k decorated character images by inputting the k pieces of input information one by one into the image generation model; The image forming unit prints or embroiders an image including the k decorative character images.

11. 2. The image processing system according to claim 1, the input information generation unit generates n types of the input information using one piece of the character information and the decoration information; the image generation unit generates the n types of decorative character images by inputting the n types of input information one by one into the image generation model; the display control unit displays the n types of decorative character images on the display unit; The image forming unit prints or embroiders an image selected by a user from among the n types of decorative character images on the object.

12. 2. The image processing system according to claim 1, An image processing system, wherein the decorative information includes information for specifying at least one of the color and size of the decorative element.

13. 8. The image processing system according to claim 7, The image processing system, wherein the information acquisition unit allows input of the position designation instruction that designates a position that overlaps the character area, and does not allow input of the position designation instruction that designates a position that does not overlap the character area.

14. 3. The image processing system according to claim 2, further comprising: a storage area for storing correspondence information that associates the amount of decoration with the number and positions of the related color objects to be arranged in the character area; The information acquisition unit obtaining an amount specification instruction from a user that specifies an amount of decoration; acquiring, as the decoration information, the number and positions of the related color objects associated with the amount of decoration specified by the amount specification instruction, with reference to the target information; The image generation unit generates the input image using the number and positions of the related color objects.

15. 3. The image processing system according to claim 2, The information acquisition unit further acquires, from a user, an instruction to designate at least one of a color of the related color object and a color of the character area; The image generation unit If the color of the related color object and the color of the character area are similar, a second outline having a color different from the color of the related color object is added to the related color object; an image processing system that generates the input image including the associated color object with the contour line added and the text region;

16. A method for generating a decorated character image, which is an image showing a decorated character that is decorated with a decorative element, comprising: a first step of generating an input image showing a character area showing a character and a related color object arranged so as to overlap the character area, the related color object being an object having a color related to the decorative element; A second step, wherein the image generation model generates the decorative character image by inputting input information including the aforementioned input image, the image generation model is trained to generate an image with decorations including the decorative elements based on the input information, A method for generating a decorative character image, including:

Images