Image processing system and method for generating decorative character image

Abstract

The present invention generates an image of a character using a machine learning model. This image processing system comprises: an information acquisition unit that acquires character information for designating a character and decoration information for designating a position where a decorative element constituting a decoration is disposed with respect to the character; and an image generation unit that causes an image generation model to generate a decorative character image by inputting, to the image generation model, input information which uses the character information and the decorative information. The decorative character image is an image showing a decorative character being a character to which a decoration including the decorative element is applied. The image generation model is trained so as to generate, on the basis of the inputted information, an image to which the decoration including the decorative element is applied.

Description

Image processing system and method for generating decorative character images

[0001] The present specification relates to a technique for generating an image showing characters.

[0002] Patent Literature 1 discloses a technique for training a neural network using a group of prototype characters as input patterns and a group of transformed characters as training patterns. In this technique, when an unlearned character is input to the neural network, a transformed character obtained by transforming the unlearned character according to a transformation rule is output.

[0003] Japanese Patent Application Publication No. 5-266226

[0004] Thus, there is a need for a technique for generating images of characters.

[0005] This specification discloses a technique for generating images of characters using machine learning models.

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] An image processing system comprising: an information acquisition unit that acquires, based on user input, character information for specifying characters and decorative information for specifying positions at which decorative elements that constitute the decoration are to be placed relative to the characters; an image generation unit that causes an image generation model to generate a decorated character image by inputting input information using the character information and the decorative information into the image generation model, wherein the decorated character image is an image showing a decorated character, which is the character that has been decorated with decoration including the decorative elements, and the image generation model has been trained to generate an image decorated with decoration including the decorative elements based on the input information; a display control unit that displays the decorated character image on a display unit; and an image forming unit that, when an instruction from a user is acquired, prints, embroiders, or cuts the decorated character image displayed on the display unit onto an object.

[0008] According to the above configuration, a decorative character image is generated by inputting input information using character information and decoration information acquired based on user input into an image generation model. As a result, a decorative character image corresponding to the character information and decoration information can be generated using a machine learning model. For example, a decorative character image in which decorative elements are arranged at positions specified by the decoration information can be generated. Furthermore, the decorative character image is displayed, and when a user's instruction is received, the decorative character image is printed, embroidered, or cut, so that a decorative character image corresponding to the character information and decoration information can be provided to the user.

[0009] [Application Example 2] A method for generating a decorated character image, which is an image showing decorated characters decorated with decorative elements, comprising: a first step of generating an input image showing a character area showing characters and a related color object arranged so as to overlap the character area, wherein the related color object is an object having a color related to the decorative element; and a second step of inputting input information including the input image into an image generation model, thereby causing the image generation model to generate the decorated character image, wherein the image generation model has been trained to generate an image decorated with decorations including the decorative elements based on the input information.

[0010] According to the above configuration, an input image showing a character is generated using character information and decoration information, and a decorated character image is generated by inputting input information including the input image into an image generation model. As a result, a decorated character image according to the character information and decoration information can be generated using a machine learning model. For example, a decorated character image in which a decorative element is arranged at a position specified by the decoration information can be generated.

[0011] The technology disclosed in this specification can be realized in various forms, such as an image processing method and an image processing device, a computer program for realizing the functions of the method or device, a recording medium (e.g., a non-temporary recording medium) on which the computer program is recorded, and the like.

[0012] 1 is a block diagram showing the configuration of a system 1000 according to a first embodiment. A block diagram showing an example of an image generation model 900. A flowchart of decorated character generation processing according to the first embodiment. A diagram showing an example of an image used in decorated character generation processing. An explanatory diagram of a decorated character image OIf. A diagram showing an example of a selection screen and an output screen. A flowchart of decorated character generation processing according to a second embodiment. A diagram showing an example of a sample selection screen WI3 according to the second embodiment. A flowchart of decorated character generation processing according to a third embodiment. A flowchart of input image generation processing. An explanatory diagram of the amount of decoration. A diagram showing an example of an adjustment instruction input screen WI4. A first explanatory diagram of input image generation processing. A second explanatory diagram of input image generation processing.

[0013] A. First Embodiment: A1. Configuration of System 1000 FIG. 1 is a block diagram showing the configuration of system 1000 of the first embodiment. System 1000 includes a terminal device 100, an image processing server 200, and a printer 300. Terminal device 100 and printer 300 are connected to a local area network LN and are communicatively connected to each other. Local area network LN is connected to the Internet IT, and image processing server 200 is connected to the Internet IT. Therefore, terminal device 100 and image processing server 200 are communicatively connected via local area network LN and Internet IT. In a modified example, as described below, a sewing machine 400 or a cutting machine 500 may be provided instead of printer 300.

[0014] The image processing server 200 is, for example, a computer, such as a cloud server, operated by a business that manufactures and sells the printer 300 .

[0015] The image processing server 200 includes a processor 210, a storage device 215, a graphics processing unit 260 (hereinafter referred to as GPU 260), and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0016] The processor 210 is a device configured to process data, such as a central processing unit (CPU). The volatile storage device 220 is, for example, a dynamic random access memory (DRAM), and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores data for a computer program 231, an image generation model 900, and a motif image group 800. The image generation model 900 is a program module that forms a trained machine learning model.

[0017] The computer program 231 and the image generation model 900 are provided, for example, in a form uploaded by a business operator that operates the image processing server 200. The processor 210 executes the computer program 231 to execute the decorated character generation process described below in cooperation with the terminal device 100. The motif image group 800 includes a plurality of motif images used in the decorated character generation process described below.

[0018] The GPU 260 is a computing device configured to execute various numerical calculations such as image processing and machine learning. The GPU 260 executes various calculations in accordance with instructions from the processor 210. In this embodiment, the GPU 260 is used to execute calculations of the image generation model 900. The communication interface 270 is an interface for connecting to the Internet (IT), and is, for example, a wired interface conforming to Ethernet (registered trademark).

[0019] The terminal device 100 is a computer, for example, a mobile terminal such as a smartphone, a tablet computer, etc. In a modified example, the terminal device 100 may be a personal computer.

[0020] The terminal device 100 includes a processor 110, a storage device 115, a display unit 140, an operation unit 150, and a communication interface 170. These elements are connected to one another via a bus. The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130.

[0021] The processor 110 is a device configured to perform data processing, such as a CPU. The volatile storage device 120 is, for example, a DRAM, and the non-volatile storage device 130 is, for example, a flash memory. The non-volatile storage device 130 stores data for a computer program 131.

[0022] The computer program 131 is an application program that is provided, for example, in the form of being downloaded from a server connected via the Internet IT. By executing the computer program 131, the processor 110 functions as an application that executes the decorative character generation process described above in cooperation with the image processing server 200.

[0023] The display unit 140 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 150 is a device configured to receive user operations, such as buttons or a touch panel overlaid on the display unit 140. In this embodiment, the display unit 140 and the operation unit 150 form a so-called touch screen. The user can input various instructions to the image processing server 200 by operating the operation unit 150. The display unit 140 may display operation elements (e.g., buttons, sliders, etc.), and the displayed elements may be operated through operation of the operation unit 150.

[0024] The printer 300 is a device that prints an image on a printing medium such as paper using color materials such as ink or toner.

[0025] A2. Image Generation Model 900 Fig. 2 is a block diagram illustrating an example of an image generation model 900. The image generation model 900 may be any of various models that use input image data to generate output image data based on the input image. In this embodiment, the image generation model 900 is a machine learning model called Stable Diffusion.

[0026] Stable Diffusion is a model that synthesizes high-resolution images using a latent diffusion model. The technology for synthesizing high-resolution images using a latent diffusion model is disclosed, for example, in the following paper: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjoern Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models", arXiv:2112.10752, April 13, 2022, http: / / arxiv.org / abs / 2112.10752

[0027] Data for pre-trained Stable Diffusion models is made publicly available on the Internet by Stability AI. In this embodiment, the data for the publicly available pre-trained model is used as the data for the image generation model 900. The image generation model 900 includes a text encoder 910, an image encoder 920, a latent variable model 930, and an image decoder 940. The text encoder 910 is configured to convert text PT into a vector tv (such a vector tv is also called a text embedding). The image encoder 920 is configured to convert an input image II into a latent variable lvi. The latent variable model 930 is configured to perform a noise addition process that adds noise to the latent variable lvi and a process that outputs a processed latent variable lvo by performing a de-diffusion process that removes noise from the noisy latent variable. The latent variable model 930 includes a neural network called a U-Net (not shown) for performing the de-diffusion process. The image decoder 940 is configured to generate an output image OI using the latent variable lvo. The latent variable model 930 uses the vector tv obtained from the text encoder 910 for conditioning in the de-diffusion process. By executing this de-diffusion process, the latent variable model 930 can generate a latent variable lvo that corresponds to an image conditioned by the text PT. The text encoder 910 is pre-trained so that the vector tv obtained from the text PT and the image represented by the text PT can be associated. As the text encoder 910, an encoder pre-trained using a technology called CLIP is used. CLIP is a technology released by OpenAI.

[0028] The image generation model 900 can generate a new output image OI represented by the text PT by performing a de-diffusion process using randomly generated noise using the vector tv from the text encoder 910. This technique is also called txt2img. Furthermore, the image generation model 900 can generate an output image OI by modifying the input image II according to the text PT using the input image II and the text PT. This technique is also called img2img. In this embodiment, a decorated character image is generated using the img2img technique. For example, as described below, when the image generation model 900 receives the text PT "letter decorated with flowers," it generates an image decorated with "flowers" for the input image II. For this reason, the image generation model 900 can be said to be a model trained to generate decorated images containing decorative elements (e.g., flowers) based on input information (e.g., images or text).

[0029] In this embodiment, the data of the input image II and the output image OI are RGB image data. RGB image data is bitmap data that represents color values ​​using three color components: red (R), green (G), and blue (B). Images II and OI are rectangular images with two sides parallel to a first direction Dx and two sides parallel to a second direction Dy perpendicular to the first direction Dx. Images II and OI are represented by the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy (the color values ​​indicate the gradation values ​​of red (R), green (G), and blue (B) (e.g., values ​​greater than or equal to zero and less than or equal to 255)). The number of pixels in the first direction Dx and the number of pixels in the second direction Dy accepted by the image generation model 900 are each predetermined and common to the input image II and the output image OI. Hereinafter, the above data format of the image data accepted by the image generation model 900 will be referred to as the processed data format.

[0030] A3. Decorative Character Generation Process FIG. 3 is a flowchart of the decorative character generation process of the first embodiment. The decorative character generation process is executed by the terminal device 100 (processor 110), which functions as an application, in cooperation with the image processing server 200 (processor 210). The terminal device 100 (application) mainly executes processes related to a screen functioning as a user interface (referred to as a UI screen) and processes for controlling the printer 300 (or, in a modified example, a sewing machine 400 or a cutting machine 500). The image processing server 200 mainly executes processes for generating a decorative character image based on user instructions acquired via the terminal device 100. In this embodiment, the decorative character image is an image representing decorative characters, which are characters decorated with decorative elements (also called motifs). As described below, in this embodiment, the decorative elements are "flowers." The decorative character generation process is initiated, for example, when a start instruction is input to the terminal device 100 (application) and the image processing server 200 acquires the start instruction via the terminal device 100.

[0031] In S100, the image processing server 200 acquires character information from the user via the terminal device 100. The acquired character information includes information indicating the characters to be processed and font information indicating the font type of the characters. In this embodiment, the characters to be processed are alphabetic characters. The characters to be processed may include some or all of kana characters (hiragana or katakana), kanji characters, numbers, and Arabic characters. The font type is selected, for example, from multiple font candidates prepared in advance. For example, the number of characters to be processed is not limited to one. In other words, the number k of characters to be processed may be an integer equal to or greater than two. In this embodiment, it is assumed that the five-character alphabet "HELLO" is acquired as the characters to be processed (k=5).

[0032] Specifically, the terminal device 100 displays an input screen (not shown) on the display unit 140, which includes an input field for inputting characters to be processed and a list of selectable fonts. On the input screen, the user inputs one or more characters to be processed in the input field and selects one font from the list. The terminal device 100 transmits character information, including information indicating the characters input by the user and information indicating the font selected by the user, to the image processing server 200. As a result, the image processing server 200 acquires the character information.

[0033] In S105, the image processing server 200 acquires one motif image MI from the motif image group 800 based on a user instruction acquired via the terminal device 100. That is, the image processing server 200 acquires one motif image MI from the motif image group 800 based on the character information acquired in S100. FIG. 4 illustrates an example of an image used in the decorative character generation process. FIG. 4A illustrates multiple motif images MI (MI1-MI3) included in the motif image group 800 ( FIG. 1 ). The data for the motif image MI is the RGB image data described above. The motif image MI is an image including objects (also referred to as related color objects) having colors associated with the decorative elements. In this embodiment, the decorative elements are flowers, and therefore the related color objects have colors commonly recognized as flower colors, such as red, yellow, white, blue, and purple. The related color objects can have various shapes and sizes.

[0034] For example, the motif image MI1 in Figure 4(A) includes a star-shaped mark SM (hereinafter also referred to as a star mark) as a related color object. The motif image MI2 includes a circular mark RM (hereinafter also referred to as a circle mark) as a related color object. The motif image MI3 includes a cross-shaped mark CM (hereinafter also referred to as a cross mark) as a related color object. The color, size, and shape of these related color objects affect the color, size, and shape of the decorative elements included in the generated decorative character image.

[0035] The motif images MI included in the motif image group 800 are not limited to the example shown in Fig. 4(A) and may include various images. For example, the motif image MI may be a photographic image including multiple flowers, or an image showing a painting, pattern, or illustration, but preferably includes an associated color object having a color associated with the flower.

[0036] For example, the image processing server 200 transmits thumbnail images of multiple motif images MI included in the motif image group 800 to the terminal device 100. The terminal device 100 displays a selection screen (not shown) including thumbnail images of multiple motif images MI on the display unit 140. The user inputs a selection instruction to select one motif image MI by, for example, tapping a thumbnail image of their choice on the selection screen. The terminal device 100 transmits information indicating the motif image MI selected by the user to the image processing server 200. The image processing server 200 acquires the motif image MI selected by the user in accordance with the information transmitted from the terminal device 100. For example, it is assumed that motif image MI2 in FIG. 4A is acquired as the motif image MI to be used.

[0037] In S110, the image processing server 200 acquires text PTf (prompt) for decorative characters to be input to the image generation model 900. In this embodiment, the text PTf is predetermined text. For this purpose, for example, the image processing server 200 acquires the predetermined text PTf from the non-volatile storage device 230. In this embodiment, the text PTf is, for example, "letter decorated with flowers."

[0038] In S115, the image processing server 200 generates a font image FI. The font image FI is an image representing the character to be processed. The character represented by the font image FI has the shape of the font indicated by the font information. If the character to be processed is multiple characters, the font images FI are generated for each character. For example, if the character to be processed is "HELLO," five font images FI representing each character are generated. The data for each font image FI is generated in a processing data format accepted by the image generation model 900. That is, the font image FI is RGB image data and has the size (number of vertical and horizontal pixels) of the input image II described above. FIG. 4B shows a font image FI representing the character "E" TX. For example, the color of the characters in the font image FI is black, and the color of the parts other than the characters (i.e., the background BG) is white.

[0039] In S120, the image processing server 200 generates a character outline image OLI. The character outline image OLI is an image showing the outlines OL of characters shown in the font image FI. The image processing server 200 generates the character outline image OLI by, for example, performing edge extraction processing on the data of the font image FI. For the edge extraction processing, a known edge extraction processing, for example, the "findContours" function of OpenCV (Open Source Computer Vision Library), can be used. FIG. 4C shows the character outline image OLI showing the outlines OL of the font image FI of FIG. 4B.

[0040] In S125, the image processing server 200 generates a character mask image MKI using the font image FI. The character mask image MKI is a binary image obtained by setting the pixel values ​​constituting the characters of the font image FI to "0" and replacing the pixel values ​​constituting the background of the font image FI with "1." FIG. 4(D) shows a character mask image MKI generated using the font image FI of FIG. 4(B). The character mask image MKI is an image showing the character area TA and background area TB of the font image FI. The shape of the character area TA has the shape of the font indicated by the font information.

[0041] In S130, the image processing server 200 generates a text motif image TMI using the text mask image MKI and the selected motif image MI. The text motif image TMI is an image in which the motif image MI is composited into the text area TA indicated by the text mask image MKI, but the motif image MI is not composited into the background area BA. In other words, the image processing server 200 determines the portion of the multiple motif images MI to be composited into the text area TA based on the user's instructions. The image processing server 200 then composites the portion of the selected motif image MI into the text area TA. Figure 4(E) shows a text motif image TMI generated using the text mask image MKI of Figure 4(D) and the motif image MI of Figure 4(A).

[0042] Specifically, the image processing server 200 determines the relative placement position of the character mask image MKI with respect to the motif image MI to be used, for example, the motif image MI2 in FIG. 4A. The character mask image MKI is positioned at the determined placement position within the motif image MI2 in FIG. 4A, as indicated by a dashed line. In this embodiment, the placement position is automatically determined to a predetermined position. Alternatively, the placement position may be automatically determined to a random position within the motif image MI2. The image processing server 200 places the character mask image MKI at the determined placement position with respect to the motif image MI2. In this state, the image processing server 200 combines the portion of the motif image MI2 that overlaps with the character region TA of the character mask image MKI with the character region TA of the character mask image MKI. The image processing server 200 then paints the background region BA of the character mask image MKI solid white. This generates the character motif image TMI. In this way, the process of determining the relative position of the character mask image MKI with respect to the motif image MI2 can be said to be a process of determining the portion of the motif image MI2 that should be combined with the character area TA.

[0043] In S135, the image processing server 200 generates an input image IIf for decorative characters by combining the character outline image OLI with the character motif image TMI. Fig. 4(F) shows the input image IIf obtained by combining the character outline image OLI of Fig. 4(C) with the character motif image TMI of Fig. 4(E). In the input image IIf of Fig. 4(F), an outline OL is added to the character area TA.

[0044] The above-described processes of S120 to S135 are executed for each of the five font images FI (font images FI for each character of "HELLO") generated in S115. Therefore, at this point, input images IIf for each character of "HELLO" have been generated.

[0045] In S140, the image processing server 200 inputs input information including an input image IIf for decorated characters, text PTf, and a seed value into the image generation model 900 to generate decorated character images OIf. The seed value is a random number for adding variation to the generated image. For example, n different random numbers (n is an integer greater than or equal to 2) are used as the seed value. When there are k characters to be processed, the image processing server 200 generates n decorated character images OIf for each character, for a total of (k×n) decorated character images OIf. The image processing server 200 generates (k×n) decorated character images OIf by inputting the (k×n) pieces of input information one by one into the image generation model 900. When there are five characters to be processed, "HELLO," (5×n) decorated character images OIf are generated.

[0046] For example, n pieces of input information for generating n decorated character images of "E" each include an input image IIf ( FIG. 4(F) ) representing the character "E," text PTf, and one of n types of seed values. The image processing server 200 inputs the n pieces of input information to the image generation model 900 to generate decorated character images OIf representing n decorated character "E" DT. FIG. 5 is an explanatory diagram of the decorated character image OIf. FIG. 5(A) shows partial images OIfa and OIfb of the n decorated character images OIf of "E." As described above, the two decorated character images OIfa and OIfb are generated using input information that differs only in the seed value.

[0047] The decorated character image OIfa is an image representing the decorated character DTa. The decorated character DTa is composed of decorative elements Da (Da1-Da7) representing flowers and elements STa and LFa representing stems and leaves. The decorated character image OIfb is an image representing the decorated character DTb. The decorated character DTb is composed of decorative elements Db (Db1-Db7) representing flowers and elements STb and LFb representing stems and leaves. The decorated character DTa and the decorated character DTb differ in size and detailed shape of the decorative elements Da and Db and the elements STa, STb, LFa, and LFb representing stems and leaves.

[0048] FIG. 5B shows the input image IIf used to generate the decorated character images OIfa and OIfb of FIG. 5A. The input image IIf of FIG. 5B includes a character area TA indicating characters specified by character information and multiple circle marks RM (RM1-RM7) arranged to overlap the character area TA. The decorative elements Da1-Da7 indicating flowers in the decorated character image OIfa of FIG. 5A correspond to the circle marks RM1-RM7 in the input image IIf of FIG. 5B. For example, the positions of the decorative elements Da1-Da7 in the decorated character DTa of the decorated character image OIfa generally coincide with the positions of the circle marks RM1-RM7 in the character area TA of the input image IIf. Similarly, the decorative elements Db1-Db7 indicating flowers in the decorated character image OIfb of FIG. 5A correspond to the circle marks RM1-RM7 in the input image IIf of FIG. 5B. For example, the positions of decorative elements Db1-Db7 in decorated character DTb in decorated character image OIfb generally coincide with the positions of circle marks RM1-RM7 in character area TA of input image IIf.

[0049] In this way, the position of the circle mark RM in the character area TA of the input image IIf can be said to specify the positions at which the decorative elements Da and Db will be placed relative to the characters in the generated decorated character image OIf. In other words, the input image IIf includes information specifying the positions at which the decorative elements Da and Db will be placed relative to the characters.

[0050] Furthermore, the colors of the decorative elements Da1-Da7 representing flowers in the decorative character image OIfa in Fig. 5(A) are similar to the colors of the corresponding circle marks RM1-RM7 in the input image IIf in Fig. 5(B). For example, the color of the decorative element Da1 is similar to the color of the circle mark RM1, and the color of the decorative element Da3 is similar to the color of the circle mark RM3. The same is true for the colors of the decorative elements Db1-Db7 in the decorative character image OIfb in Fig. 5(A).

[0051] In this way, it can be said that the color of the circle mark RM in the character area TA of the input image IIf specifies the colors of the decorative elements Da and Db in the decorative character image OIf to be generated. In other words, the input image IIf includes information specifying the colors of the decorative elements Da and Db.

[0052] Furthermore, the shape of the contour of the decorated character DTa in the decorated character image OIfa in Fig. 5A is formed by, for example, elements STa and LFa that mainly represent stems and leaves. The shape of the contour of the decorated character DTa is similar to the shape of the contour of the character area TA indicated by the contour line OL in the input image IIf. The same is true for the shape of the contour of the decorated character DTb in the decorated character image OIfb.

[0053] 3 , the image processing server 200 displays a selection screen WI1 including n types of decorative character images OIf on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the selection screen WI1 and transmits the generated data to the terminal device 100. The terminal device 100 displays the selection screen WI1 on the display unit 140 using the received data.

[0054] 6A and 6B are diagrams showing examples of a selection screen and an output screen. The selection screen WI1 in Fig. 6A includes n types of generated decorative character images OIf (e.g., OIfa, OIfb), a message MS1, a selection cursor CS, a selection button BT1, and a redo button BT2. The message MS1 is a message that prompts the user to select a desired character from the n types of decorative character images OIf.

[0055] In S150, the image processing server 200 receives a selection instruction from the user via the terminal device 100 to select a decorative character image OIf to be output. Specifically, the user moves the cursor CS on the selection screen WI1 of FIG. 6A to select a desired decorative character image OIf and taps the selection button BT1 to input a selection instruction for the decorative character image OIf. Upon receiving the selection instruction for the decorative character image OIf, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected decorative character image OIf, to the image processing server 200. Note that, if the redo button BT2 on the selection screen WI1 is tapped, the decorative character generation process of FIG. 3 is executed from the beginning, for example.

[0056] In S155, the image processing server 200 displays an output screen WI2 including the decorative character image to be output on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the output screen WI2 including the decorative character image OIf of the type selected by the selection instruction, and transmits the generated data to the terminal device 100. The terminal device 100 displays the output screen WI2 on the display unit 140 using the received data.

[0057] As shown in FIG. 6B , the output screen WI2 includes an output image TI, a print button BT3, and a redo button BT4. The output image TI includes five decorated character images OIf representing the five characters of "HELLO" to be processed. The five decorated character images OIf are images generated using the same seed value as the seed value used to generate the decorated character image OIf selected by the user. When the user checks the output image TI and wishes to print it, they tap the print button BT3 on the output screen WI2. When the print button BT3 is tapped, the process proceeds to S160. That is, the printer 300 prints on paper an image selected by the user from among the n types of decorated character images OIf. When the redo button BT4 is tapped on the output screen WI2, the decorated character generation process of FIG. 3 is executed from the beginning, for example.

[0058] In S160, the image processing server 200 generates a print job for printing an output image TI ( FIG. 6B ) including the decorative character image OIf. In S160, the image processing server 200 transmits the print job to the printer 300, causing the printer 300 to print the output image TI. Specifically, the image processing server 200 may transmit the print job to the printer 300 via the terminal device 100, or may transmit the print job directly to the printer 300 without going through the terminal device 100. Upon receiving the print job, the printer 300 prints the output image TI including the decorative character image OIf on paper in accordance with the print job.

[0059] According to the first embodiment described above, the system 1000 acquires character information for specifying characters based on a start command input by a user (S100 in FIG. 3 ) and acquires a motif image MI as decoration information (S105 in FIG. 3 ). The motif image MI is used to generate an input image IIf and includes a circle mark RM that specifies the position of decorative elements Da and Db that constitute the decoration relative to the characters in the input image IIf (FIG. 4A ). The system 1000 inputs input information into the image generation model 900 using the character information and the motif image MI, causing the image generation model 900 to generate a decorated character image OIf (S115-S140 in FIG. 3 ). The system 1000 displays the decorated character image OIf on the display unit 140 of the terminal device 100 (S155 in FIG. 3 ). The system 1000 prints an output image TI including the decorated character image OIf displayed on the display unit 140 on paper (S160 and S165 in FIG. 3 ). As a result, a decorated character image OIf can be generated using the image generation model 900 in accordance with character information and the input image IIf. For example, a decorated character image OIf can be generated in which decorative elements Da and Db are arranged at positions specified by the input image IIf. Furthermore, by displaying the decorated character image OIfa and forming an output image by printing or embroidery, a decorated character can be provided to the user in accordance with the character information and decorative information. For example, if the font image FI is input directly into the image generation model 900, it is not known where and how the decorative elements will be arranged, and it is therefore impossible to predict what kind of decorated character image will be generated. According to this embodiment, a decorative character image OIf with a good appearance in which the decorative elements Da and Db are appropriately arranged can be generated, compared to when the font image FI is input directly into the image generation model 900.

[0060] Furthermore, according to this embodiment, the input image IIf includes a character area TA and a plurality of circle marks RM (RM1-RM7), which are related color objects arranged so as to overlap the character area TA (FIG. 5B). The circle marks RM are objects having colors related to the decorative elements Da and Db ("flowers" in this embodiment). As a result, for example, the colors of the decorative elements Da and Db can be specified using the color of the circle marks RM. Therefore, a good-looking decorative character image OIf can be generated in which decorative elements Da and Db of appropriate colors are arranged.

[0061] Furthermore, according to this embodiment, the system 1000 generates a character mask image MKI representing the character area TA based on character information (S125 in FIG. 3 ), and then combines the motif image MI with the character area TA identified by the character mask image MKI (S130 in FIG. 3 ) to generate an input image IIf. As a result, an input image IIf is generated in which the motif image MI including the circle mark RM is combined with the character area TA (FIG. 5B ). Therefore, the input image IIf can be generated using the motif image MI, which indicates the positions where the decorative elements Da and Db should be placed using the circle mark RM. Furthermore, the system 1000 automatically determines the portion of the motif image MI to be combined with the character area TA (S130 in FIG. 3 ). As a result, the input image IIf can be easily generated.

[0062] Furthermore, according to this embodiment, the system 1000 selects one motif image MI from among the motif images MI included in the motif image group 800 based on a user instruction (S105 in FIG. 3), and then combines the selected motif image MI with the character area TA to generate an input image IIf (S130 in FIG. 3). As a result, a decorated character image OIf can be generated in accordance with the user's intention.

[0063] Furthermore, according to this embodiment, the input image IIf includes a line (outline OL) indicating the outline of the character area TA (S135 in FIG. 3, FIG. 5B). If the input image IIf did not include the outline OL of the character area TA, the boundary between the character area TA and the background area BA might be unclear. In this case, the outline of the decorated character DT might be unclear in the generated decorated character image OIf, or the outline of the decorated character DT might have a shape that is far removed from the character, which would likely reduce the readability of the decorated character DT. According to this embodiment, the input image IIf includes the outline OL, which makes it easier for the outline of the decorated character DT to maintain the shape of the character, thereby improving the readability of the decorated character DT. For example, in the example of the decorative character images OIfa and OIfb in Figure 5 (A), the outlines of the decorative characters DTa and DTb are formed into the shape of the characters by elements STa, STb, LFa, and LFb, which mainly represent stems and leaves, and the readability of the decorative characters DTa and DTb is high.

[0064] Furthermore, according to this embodiment, the acquired character information includes font information indicating the type of font. The shape of the character area TA of the input image IIf has the shape of the font indicated by the font information ( FIG. 5A ). As a result, the type of font can be specified by the font information, and thus decorated character images OIf representing decorated characters DT of various shapes can be easily generated.

[0065] Furthermore, according to this embodiment, the system 1000 acquires character information specifying k characters (e.g., "HELLO") (where k is an integer greater than or equal to 2) and generates k pieces of input information corresponding to the k characters. The system 1000 generates k decorated character images OIf by inputting the k pieces of input information one by one into the image generation model 900, and prints an output image TI including the k decorated character images OIf (FIG. 6B). The size (vertical or horizontal) of the input image II and output image OI of the image generation model 900 is a predetermined fixed value. Therefore, if a single decorated character image including multiple decorated characters is generated, the number of pixels constituting each decorated character may be reduced, resulting in a decrease in the resolution of the decorated character and a decrease in the image quality of the decorated character. According to this embodiment, such inconveniences are suppressed, and an output image TI showing each decorated character DT with sufficient resolution can be generated and output.

[0066] Furthermore, according to this embodiment, the system 1000 generates n types of input information using one piece of character information and a motif image MI, and generates n types of decorated character images OIf (e.g., OIfa, OIfb) by inputting the n types of input information one by one into the image generation model 900 (FIG. 5A). The system 1000 displays the n types of decorated character images OIf on the display unit 140 of the terminal device 100 (S145 in FIG. 3, FIG. 6A). The system 1000 prints a decorated character image selected by the user from the n types of decorated character images OIf (S155-S165 in FIG. 3, FIG. 6B). As a result, the decorated character image OIf selected by the user from the n types of decorated character images OIf generated using one piece of character information and a motif image MI is printed, making it possible to print a decorated character image OIf that better suits the user's preferences.

[0067] As can be seen from the above description, the motif image MI in this embodiment is an example of decorative information and a specific image. The circle mark RM in this embodiment is an example of a related color object, and the character mask image MKI is an example of an area image. The outline OL in this embodiment is an example of a first outline.

[0068] B. Second Embodiment In the second embodiment, the contents of the decorated character generation process differ from those in the first embodiment. Figure 7 is a flowchart of the decorated character generation process in the second embodiment.

[0069] In S210, the image processing server 200 displays a sample selection screen WI3 including a plurality of sample decorative character images OIs on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the sample selection screen WI3 and transmits the generated data to the terminal device 100. The terminal device 100 displays the sample selection screen WI3 on ​​the display unit 140 using the received data.

[0070] 8 is a diagram showing an example of a sample selection screen WI3 of the second embodiment. The sample selection screen WI3 of FIG. 8 includes a plurality of sample decorative character images OIs (e.g., OIs1-OIs3), a message MS3, a selection cursor CS, and selection buttons BTs. The message MS3 prompts the user to select a desired character from the plurality of sample decorative character images OIs.

[0071] Each of the sample decorative character images OIs corresponds one-to-one to one of the motif images MI included in the motif image group 800. For example, sample decorative character image OIs1 in Fig. 8 corresponds to motif image MI1 in Fig. 4(A). Sample decorative character image OIs2 corresponds to motif image MI2 in Fig. 4(A). Sample decorative character image OIs3 corresponds to motif image MI3 in Fig. 4(A).

[0072] In the second embodiment, data for each sample decorated character image OIs is generated in advance using a corresponding motif image MI1 and stored in the non-volatile storage device 230 of the image processing server 200. Specifically, the data for each sample decorated character image OIs is generated by executing steps S120-S140 of FIG. 3 using a font image FI of a specific character in a specific font ("A" in the example of FIG. 8), a corresponding motif image MI, and predetermined text PTf. The predetermined text PTf is, for example, "letter decorated with flowers," as in the first embodiment. Therefore, for example, the sample decorated character image OIs1 is generated by inputting input information, including an input image IIf generated using the motif image MI1, into the image generation model 900.

[0073] For this reason, the shape, position, size, and color of the decorative elements in the sample decorative character images OIs are influenced by the shape, position, size, and color of the related color objects in the corresponding motif images MI. For example, the shape, size, and color of the decorative element D1 in the sample decorative character image OIs1 are similar to the shape, size, and color of the star mark SM in the corresponding motif image MI1. The shape, size, and color of the decorative element D2 in the sample decorative character image OIs2 are similar to the shape, size, and color of the circle mark RM in the corresponding motif image MI2. The shape, size, and color of the decorative element D3 in the sample decorative character image OIs3 are similar to the shape, size, and color of the cross mark CM in the corresponding motif image MI3. In this way, it can be said that the motif images MI1-MI3 contain information for specifying the shape, size, and color of the decorative elements D1-D3. Furthermore, the color of the entire character in the sample decorative character images OIs (the color of the part excluding the decoration) is similar to the background color of the motif images MI1-MI3.

[0074] In S220, the image processing server 200 selects one sample decorated character based on a user's selection instruction. Specifically, the user moves the cursor CS on the sample selection screen WI3 of FIG. 8 to select a desired sample decorated character image OIs and taps the selection button BTs to input a selection instruction for the sample decorated character image OIs. That is, the image processing server 200 selects one sample decorated character image OIs based on the user's instruction from among a plurality of sample decorated character images OIs, including a first decorated character image OIs1 generated by inputting input information generated using the motif image MI1 into the image generation model 900 and a second decorated character image OIs2 generated by inputting input information generated using the second motif image MI2 into the image generation model 900. When the terminal device 100 receives the selection instruction for the sample decorated character image OIs, it transmits the content of the selection instruction, i.e., information indicating the selected sample decorated character image OIs, to the image processing server 200. The image processing server 200 selects the sample decorative character image OIs designated by the user in accordance with the received information.

[0075] In S230, the image processing server 200 acquires the motif image MI and text PTf corresponding to the selected sample decorated character image OIs. That is, when the first decorated character image OIs1 is selected, the image processing server 200 acquires the first motif image MI1 and text PTf, and when the second decorated character image OIs2 is selected, the image processing server 200 acquires the second motif image MI2 and text PTf. Specifically, in the second embodiment, as shown in FIG. 1 , first correspondence information 710 is pre-stored in the non-volatile storage device 230 of the image processing server 200 in addition to information similar to that in the first embodiment.

[0076] The first correspondence information 710 is information that records each sample decorated character image OIs in association with the motif image MI and text PTf used to generate the sample decorated character image OIs. The image processing server 200 references the first correspondence information 710 to acquire the motif image MI and text PTf (in this embodiment, "letter decorated with flowers") used to generate the selected sample decorated character image OIs.

[0077] In S240, the image processing server 200 acquires character information from the user via the terminal device 100, similar to S100 in FIG.

[0078] 3. In other words, the image processing server 200 generates a decorated character image OIf using the character information acquired in S240 and the motif image MI and text PTf acquired in S230. In other words, if the first decorated character image OIs1 is selected in S220, the image processing server 200 acquires the first motif image MI1 in S230, and if the second decorated character image OIs2 is selected in S220, the image processing server 200 acquires the second motif image MI2 acquired in S230. Furthermore, when the first decorated character image OIs1 is selected in S220, the image processing server 200 generates input information using the character information acquired in S240 and the first motif image MI1 and text PTf acquired in S230, and when the second decorated character image OIs2 is selected in S220, the image processing server 200 generates input information using the character information acquired in S240 and the second motif image MI2 and text PTf acquired in S230. The image processing server 200 then inputs the input information to the image generation model 900. The image processing server 200 displays the generated decorated character image OIf on the display unit 140 of the terminal device 100 or causes the printer 300 to print it.

[0079] According to the second embodiment described above, the system 1000 selects one sample decorated character image OIs based on a user instruction from among a plurality of sample decorated character images OIs, including a first decorated character image OIs1 generated by inputting input information generated using a first motif image MI1 into the image generation model 900, and a second decorated character image OIs2 generated by inputting input information generated using a second motif image MI2 into the image generation model 900 (S220 in FIG. 7). If the first decorated character image OIs1 is selected, the image processing server 200 acquires the first motif image MI1, and if the second decorated character image OIs2 is selected, the image processing server 200 acquires the second motif image MI2 (S230 in FIG. 7). When the first decorative character image OIs1 is selected, the system 1000 generates input information using the first motif image MI1, and when the sample decorative character image OIs2 is selected, the system 1000 generates input information using the second motif image MI2, and inputs the generated input information into the image generation model 900 (S250 in Figure 7).

[0080] As a result, it is possible to generate and print a decorative character image OIf that meets the user's preferences. For example, even when a user looks at the motif images MI (MI1-MI3), the user may be unable to visualize the decorative character image OIf that will be generated. In this case, even if the user is prompted to select a motif image MI to use, the user may be unable to select an appropriate motif image MI that will enable the user to generate the desired decorative character image. In this embodiment, the user simply selects a sample decorative character image OIs of the desired style from among multiple sample decorative character images OIs generated using each of the multiple motif images MI. This allows the user to display and print a decorative character image OIf of the desired style for the desired characters. Therefore, it is possible to easily display and print a decorative character image OIf that meets the user's intentions.

[0081] As can be seen from the above explanation, the first motif image MI1 of this embodiment is an example of first decorative information, and the second motif image MI2 is an example of second decorative information.

[0082] C. Third Embodiment In the third embodiment, the contents of the decorated character generation process differ from those in the first embodiment. Fig. 9 is a flowchart of the decorated character generation process in the third embodiment.

[0083] 3, the image processing server 200 acquires character information from the user via the terminal device 100. The acquired character information includes information indicating the character to be processed and font information indicating the type of font (typeface) of the character.

[0084] 3, the image processing server 200 acquires text PTf (prompt) for decorative characters to be input to the image generation model 900. In this embodiment, the text PTf is predetermined text (e.g., "letter decorated with flowers").

[0085] In S330, the image processing server 200 generates a font image FI (FIG. 4B) using the character information, similar to S115 in Fig. 3. In S340, the image processing server 200 generates a character outline image OLI (FIG. 4C) similar to S120 in Fig. 3.

[0086] In S350, the image processing server 200 executes input image generation processing. Fig. 10 is a flowchart of the input image generation processing. The input image generation processing is processing for generating an input image IIf to be input to the image generation model 900.

[0087] In S405 of Fig. 10, the image processing server 200 receives a selection instruction for selecting the amount of decoration from the user via the terminal device 100. Fig. 11 is an explanatory diagram of the amount of decoration. The upper portions of Figs. 11(A) to 11(C) show input elements VI for inputting a selection instruction for the amount of decoration. The terminal device 100 displays an input screen (not shown) including the input elements VI on the display unit 140.

[0088] The input element VI includes a slide bar SB and a slider SD that can be moved on the slide bar SB in response to a user operation. The position of the slider SD along the longitudinal direction of the slide bar SB corresponds to multiple levels of decoration amount. For example, in this embodiment, the amount of decoration can be specified in five levels, from level 1 to level 5. In the example of FIG. 11 , the position of the slider SD in the input element VI in FIG. 11(A) corresponds to level 1. The position of the slider SD in the input element VI in FIG. 11(B) corresponds to level 3, and the position of the slider SD in the input element VI in FIG. 11(C) corresponds to level 5.

[0089] The user operates the slider SD on the input screen to move the slider SD to a position corresponding to the desired level of decoration amount and taps the confirm button (not shown). This inputs a selection instruction to select one of five levels of decoration amount to the terminal device 100. Upon receiving the selection instruction for the decoration amount, the terminal device 100 transmits the content of the selection instruction, i.e., information indicating the selected decoration amount, to the image processing server 200.

[0090] In S410, the image processing server 200 acquires the number of circle marks RM to be placed in the character area TA and the placement positions of the circle marks RM in the character area TA, which are specified by the selected amount of decoration and the character to be processed. Specifically, in the third embodiment, as shown in Fig. 1, second correspondence information 720 is pre-stored in the non-volatile storage device 230 of the image processing server 200 in addition to information similar to that in the first embodiment. The character to be processed is indicated by the character information acquired in S310 of Fig. 9.

[0091] The second correspondence information 720 is information that records, for each letter of the alphabet, the correspondence between each of the decoration amount levels 1-5, the number of circle marks RM to be placed, and the placement positions of the circle marks RM to be placed. For example, FIG. 11A shows the number of circle marks RM corresponding to decoration amount level 1 and the placement positions of the circle marks RM corresponding to decoration amount level 1 for the letters "H" and "E." At level 1, for example, one circle mark RM is placed near the upper left corner of the alphabet. FIG. 11B shows the number of circle marks RM corresponding to decoration amount level 3 and the placement positions of the circle marks RM corresponding to decoration amount level 3 for the letters "H" and "E." At level 3, for example, three circle marks RM are placed in predetermined positions determined for each letter of the alphabet. FIG. 11C shows the number of circle marks RM corresponding to decoration amount level 5 and the placement positions of the circle marks RM corresponding to decoration amount level 5 for the letters "H" and "E." At level 5, for example, five circle marks RM are distributed at predetermined positions determined for each alphabet. The positions of the circle marks RM are determined to be positions that generate good-looking decorated character images by repeatedly conducting experiments to generate decorated character images using the image generation model 900, for example.

[0092] The image processing server 200 refers to the second correspondence information 720 and obtains, for each character to be processed, information on the number of circle marks RM corresponding to the combination of the selected amount of decoration and the character to be processed, and the placement position of the circle marks RM corresponding to the combination of the selected amount of decoration and the character to be processed.

[0093] In S415, the image processing server 200 generates an input image IIn using the acquired number of circle marks RM and information on the arrangement positions of the circle marks RM. For example, if the characters to be processed are the five characters "HELLO," an input image IIn is generated for each of the five characters. In S420, an adjustment instruction input screen WI4 including the generated input image IIn is displayed on the display unit 140 of the terminal device 100. Specifically, the image processing server 200 generates data for the adjustment instruction input screen WI4 and transmits the generated data to the terminal device 100. The terminal device 100 displays the adjustment instruction input screen WI4 on the display unit 140 using the received data.

[0094] FIG. 12 is a diagram showing an example of the adjustment instruction input screen WI4. The adjustment instruction input screen WI4 includes one or more input images IIn generated in S415. As shown in FIG. 12, the input images IIn (IIn1, IIn2) include character areas TA (TA1, TA2) and circle marks RM arranged so as to overlap the character areas TA. The number of circle marks RM and their positions are determined in S410 according to information acquired from the second correspondence information 720 in accordance with the amount of decoration. An outline OL is added to the character areas TA, and the color of the character areas TA is set to a default color (e.g., black or white).

[0095] The adjustment instruction input screen WI4 further includes messages MS4 and MS5, change buttons BTa-BTc, and a complete button BTf. The message MS4 indicates that the position, color, and size of the circle mark RM and the color of the text can be changed. The message MS5 indicates that the position, color, and size of the circle mark RM will be reflected in the position, color, and size of the decoration of the decorative text image OIfa to be generated.

[0096] For example, the user can input a change instruction to change the position of the circle mark RM relative to the character area TA by touching and moving the circle mark RM with a finger on the adjustment instruction input screen WI4. When the user taps one circle mark RM to select it on the adjustment instruction input screen WI4 and then taps the change button BTa, a UI screen (not shown) for specifying the size of the circle mark RM after the change is displayed as a pop-up. The user can input an instruction to change the size of the circle mark RM via the UI screen. When the user taps one circle mark RM to select it on the adjustment instruction input screen WI4 and then taps the change button BTb, a UI screen (not shown) such as a color palette for specifying the color of the circle mark RM after the change is displayed as a pop-up. The user can input an instruction to change the color of the circle mark RM via the UI screen. When the user taps one character area TA to select it on the adjustment instruction input screen WI4 and then taps the change button BTc, a UI screen (not shown) such as a color palette for specifying the color of the character area TA after the change is displayed as a pop-up. The user can input an instruction to change the color of the character area TA (hereinafter also referred to as character color) via the UI screen. The instruction to change the color input to the terminal device 100 is transmitted to the image processing server 200.

[0097] 10, the image processing server 200 determines whether or not an instruction to change the color of the text or the color of the circle mark RM has been received. If an instruction to change the color of the text or the color of the circle mark RM has not been received (S425: NO), the image processing server 200 proceeds to S450. If an instruction to change the color of the text or the color of the circle mark RM has been received (S425: YES), the image processing server 200 proceeds to S430.

[0098] In S430, when the image processing server 200 changes the color of the character or the color of the circle mark RM in accordance with the change instruction, the image processing server 200 determines whether the color of the character in the input image IIn is similar to the color of the circle mark RM. For example, if an index value indicating the difference between the color of the character and the color of the circle mark RM is less than a predetermined threshold THc, the image processing server 200 determines that the color of the character and the color of the circle mark RM are similar, and if the index value is equal to or greater than the predetermined threshold THc, the image processing server 200 determines that the color of the character and the color of the circle mark RM are not similar. The index value of the difference is, for example, the Euclidean distance between the color of the character and the color of the circle mark RM in a predetermined color space (e.g., RGB color space or Lab color space).

[0099] If the character color and the color of the circle mark RM are similar (S430: YES), the image processing server 200 adds an outline MOL to the circle mark RM having a color similar to the character color in the input image IIn in S435. If the character color and the color of the circle mark RM are not similar (S430: YES), the image processing server 200 does not add an outline MOL to the circle mark RM in the input image IIn in S440.

[0100] In S445, the image processing server 200 changes the color of the text or the color of the circle mark RM in the input image IIn in accordance with the change instruction. When the input image IIn is updated, such as by adding a contour line MOL or changing the text color or the circle mark RM, the updated data is transmitted from the image processing server 200 to the terminal device 100. In other words, the image processing server 200 generates an input image IIn that includes the circle mark RM to which the contour line MOL has been added and the text area TA. In the terminal device 100, the update is reflected in real time on the adjustment instruction input screen WI4 displayed on the display unit 140.

[0101] FIG. 13 is a first explanatory diagram of the input image generation process. In the input image IIna of FIG. 13(A), the character color (the color of the character area TA) and the color of the circle mark RM are not similar. In this case, no outline is added to the circle mark RM in the input image IIna. In the input image IInb of FIG. 13(B), the character color (the color of the character area TA) and the color of the circle mark RM are similar. In this case, an outline MOL is added to the circle mark RM in the input image IInb. A color that is not similar to the color of the character and the circle mark RM is used as the color of the outline MOL. The color of the outline MOL is, for example, predetermined for each expected character color and circle mark RM.

[0102] 10, the image processing server 200 determines whether or not an instruction to change the position or size of the circle mark RM has been acquired. If an instruction to change the position or size of the circle mark RM has not been acquired (S450: NO), the image processing server 200 proceeds to S470. If an instruction to change the position or size of the circle mark RM has been acquired (S450: YES), the image processing server 200 proceeds to S455.

[0103] In S455, the image processing server 200 determines whether the character area TA and the circle mark RM overlap in the input image IIn when the position or size of the circle mark RM is changed in accordance with the change instruction. For example, when a change instruction is acquired to move the circle mark RM in a direction away from the character area TA, the change in the position of the circle mark RM may cause the character area TA and the circle mark RM to no longer overlap. Also, when a change instruction is acquired to reduce the size of the circle mark RM that slightly overlaps the character area TA, the change in size of the circle mark RM may cause the character area TA and the circle mark RM to no longer overlap.

[0104] If the character area TA and the circle mark RM overlap (S455: YES), in S460, the image processing server 200 changes the position or size of the circle mark RM in the input image IIn in accordance with the change instruction. That is, the image processing server 200 allows input specifying the position of the circle mark RM at a position where the character area TA and the circle mark RM overlap. When the input image IIn is updated by changing the position or size of the circle mark RM, the updated data is transmitted from the image processing server 200 to the terminal device 100, and the update is reflected in real time on the adjustment instruction input screen WI4 displayed on the display unit 140.

[0105] If the character area TA and the circle mark RM do not overlap (S455: NO), in S465, the image processing server 200 displays an error message on the adjustment instruction input screen WI4. The error message may include, for example, the phrase "The circle mark cannot be moved to a position where it does not overlap with the text." In other words, the image processing server 200 does not allow an input specifying the position of the circle mark RM at a position where the character area TA and the circle mark RM overlap. For example, the image processing server 200 transmits an instruction to display an error message to the terminal device 100, and the terminal device 100 displays the error message on the adjustment instruction input screen WI4 displayed on the display unit 140 in accordance with the display instruction. Note that when the error message is displayed, the position and size of the circle mark RM are not changed in accordance with the acquired change instruction. As a result, the character area TA and each circle mark RM remain overlapped on the adjustment instruction input screen WI4.

[0106] FIG. 14 is a second explanatory diagram of the input image generation process. The input image IInc in FIG. 14 includes a character area TA and circle marks RMo and RMn. For example, if a change instruction to move the circle mark RMo to position P1 indicated by the dashed line is acquired, the circle mark RMo overlaps the character area TA at the changed position P1. Therefore, in this case, the position of the circle mark RMo is changed to position P1 (YES in S455 of FIG. 10, S460).

[0107] On the other hand, if a change instruction to move the circle mark RMn to position P2 indicated by the broken line is acquired, the circle mark RMn will not overlap the character area TA at the changed position P2. Therefore, in this case, the position of the circle mark RMn is not changed to position P2, and an error message is displayed (NO in S455, S465 in FIG. 10).

[0108] In S470 of Fig. 10, the image processing server 200 determines whether the user has completed adjustments to the color, size, position, etc. of the circle mark RM. When the user has completed these adjustments, the user taps the Done button BTf on the adjustment instruction input screen WI4 of Fig. 12. Therefore, the image processing server 200 determines that the user's adjustments have been completed when the Done button BTf is tapped.

[0109] If the user's adjustments are complete (S470: YES), the image processing server 200 acquires the data of the final input image IIn after the adjustments and ends the input image generation process in S475. If the user's adjustments are complete (S470: NO), the image processing server 200 returns to S425.

[0110] In S380 of Fig. 9 after the input image generation process is completed, the image processing server 200 executes the processes of S140-165 of Fig. 3. That is, the image processing server 200 generates a decorated character image OIf using the final input image IIna acquired in S475 of Fig. 10, the character information, and the text PTf acquired in S330, and then displays or prints the generated decorated character image OIf.

[0111] According to the third embodiment described above, the system 1000 allows the user to input an instruction specifying the position of the circle mark RM relative to the character area TA via the adjustment instruction input screen WI4 (FIGS. 12 and S450 in FIG. 10). The system 1000 generates an input image IIn in which the circle mark RM is placed at the position specified by the instruction relative to the character area TA (S460 and S470 in FIG. 10). The system 1000 inputs input information including the generated input image IIn to the image generation model 900 (S380 in FIG. 9 and S140 in FIG. 3). As a result, by allowing the user to specify the position of the circle mark RM relative to the character area TA on the adjustment instruction input screen WI4, a decorated character image OIf decorated according to the user's intentions can be generated and printed using the image generation model 900.

[0112] Furthermore, according to this embodiment, the system 1000 allows input of an instruction specifying a position of the circle mark RM that overlaps the character area TA, but does not allow input of an instruction specifying a position that does not overlap the character area TA (S450-S465 in FIG. 10 ). If an input image IIn is generated in which the circle mark RM and the character area TA are arranged so that they do not overlap, an appropriate decorated character image OIf may not be generated. For example, decoration may appear in a position unrelated to the characters in the decorated character image OIf, which may reduce the readability of the characters. According to this embodiment, the inconvenience of generating an input image IIn in which the circle mark RM and the character area TA are arranged so that they do not overlap can be suppressed.

[0113] Furthermore, according to this embodiment, the system 1000 includes a storage area (non-volatile storage device 230) for storing second correspondence information 720 that associates the amount of decoration, the number of circle marks RM to be placed in the character area TA, and the positions of the circle marks RM to be placed in the character area TA ( FIG. 1 ). The system 1000 acquires an instruction specifying the amount of decoration from the user ( S405 in FIG. 10 ), and then references the second correspondence information 720 to acquire the number and positions of the circle marks RM associated with the amount of decoration specified by the user's instruction ( S410 in FIG. 10 ). The system 1000 generates an input image IIn using the acquired number and positions of the circle marks RM ( S415 in FIG. 10 ). For example, a user with little knowledge of design may be unable to properly place the circle marks RM in the character area TA. In this case, if the placement of the circle marks RM were left entirely up to the user, it is highly likely that an attractive decorated character image OIf will be generated. According to this embodiment, the user simply specifies the amount of decoration, and an input image IIn is easily generated in which the circle mark RM is positioned in an appropriate position. Therefore, a decorative character image OIf with a good appearance can be easily generated. Furthermore, even when the user specifies the circle mark RM, the circle mark RM can be positioned in an appropriate initial position relative to the character area TA, so the user can easily adjust the position of the circle mark RM according to their own preferences. Therefore, the burden on the user for generating the decorative character image OIf can be reduced.

[0114] Furthermore, according to this embodiment, the system 1000 receives an instruction from the user specifying at least one of the color of the circle mark RM and the color (character color) of the character area TA (S425 in FIG. 10 ). If the color of the circle mark RM and the color of the character are similar (YES in S430 in FIG. 10 ), the system 1000 adds an outline MOL to the circle mark RM, the outline MOL being a different color from the color of the circle mark RM (S435 in FIG. 10 ). The system 1000 generates an input image IIn including the circle mark RM with the outline MOL added and the character area TA (S475 in FIG. 10 ). As a result, the user can easily distinguish and visually recognize the character area TA and the circle mark RM on the adjustment instruction input screen WI4. Therefore, the user can confirm the position of the circle mark RM relative to the character area TA and set the position and size of the circle mark RM so that an appropriate input image IIn is generated. Furthermore, when generating a decorated character image OIf, an input image IIn that clearly distinguishably represents the character area TA and the circle mark RM is input to the image generation model 900. Therefore, it is possible to increase the likelihood that a decorative element (e.g., a flower) corresponding to the circle mark RM will be appropriately added to the decorated character in the generated decorated character image OIf, and therefore a decorative character image OIf that looks good can be generated.

[0115] As can be seen from the above explanation, in this embodiment, the information on the number and positions of the circle marks RM acquired from the second correspondence information 720, and the instructions on the position, color, and size of the circle marks RM acquired from the user are examples of decorative information. Also, the outline MOL of the circle mark RM in this embodiment is an example of a second outline.

[0116] D. Modifications: (1) As shown in FIG. 1 , the system 1000 may include a sewing machine 400 and a cutting machine 500. The sewing machine 400 is a device that forms an image on fabric by embroidering the fabric with thread according to embroidery data. The cutting machine 500 is a device that expresses an image in the shape of a sheet-like material by cutting the sheet-like material with a blade according to cut data. The sheet-like material may be, for example, paper, cloth, a resin sheet such as urethane foam, or leather.

[0117] For example, in the above embodiment, the system 1000 causes the printer 300 to print the output image TI including the decorative character image OIf. Alternatively, the system 1000 may cause the sewing machine 400 to embroider the output image TI. The system 1000 may also cause the cutting machine 500 to cut the output image TI. Alternatively, the system 1000 may express the output image TI using a combination of these devices. For example, the system 1000 may use the printer 300 to print the output image TI on a sheet material, and then use the cutting machine 500 to cut the sheet material on which the output image TI is printed along the outline of the decorative character DT included in the output image TI.

[0118] (2) In the above embodiments, the decorative element of the decorative character image OIf is a "flower," but this is not limited thereto. The decorative element may be something other than a flower, such as a living thing like a star or a butterfly, or a plant like a fruit. The decorative element may also be a rose, cherry blossom, or other sub-concept of a flower. It is preferable to use different text PT depending on the decorative element. For example, if the decorative element is a rose, the text PT should read "letter decorated with roses." It is also preferable to use a color associated with the decorative element for the related color object (e.g., a circle mark RM). For example, if the decorative element is a rose, it is preferable to use a color commonly known as the color of a rose, such as red, white, or yellow, for the circle mark RM.

[0119] (3) In the first embodiment described above, the image processing server 200 selects one motif image MI from the motif image group 800 to be used to generate the input image IIf based on a user instruction (S105 in FIG. 3). Alternatively, the image processing server 200 may automatically select one motif image MI from the motif image group 800. The automatic selection method may be, for example, a random selection method or a selection method according to a predetermined order.

[0120] In the first embodiment, the motif image MI used to generate the input image IIf is selected from the motif image group 800 stored in advance in the non-volatile storage device 230 of the image processing server 200. Alternatively, the motif image MI may be selected from images (e.g., photographic images) prepared by the user and stored in the non-volatile storage device 130 of the terminal device 100.

[0121] (4) In the first embodiment, when generating an input image IIf by combining a portion of the motif image MI with the character area TA (S130 in FIG. 3), the portion of the motif image MI to be combined with the character area TA is automatically determined by placing the character mask image MKI at a predetermined position relative to the motif image MI. Alternatively, the image processing server 200 may determine the portion to be combined based on a user instruction acquired via the terminal device 100. In this case, for example, the terminal device 100 displays on the display unit 140 a predetermined UI screen showing the motif image MI and the character area TA superimposed. The user may, for example, move the character area TA on the UI screen to input the relative position of the character area TA with respect to the motif image MI.

[0122] In the first embodiment, the decorative information (information indicating the position where the decorative element is placed relative to the character) is a composite portion to be composited into the character area TA of the motif image MI, and this decorative information is acquired based on user input. As can be seen from the explanations of the above variants (3) and (4), "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI selected based on instructions input by the user and the position of the composite portion determined based on instructions input by the user. Furthermore, "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI selected based on instructions input by the user and the position of the composite portion automatically (e.g., randomly) determined. Furthermore, "the decorative information is acquired based on the user's input" includes acquiring the decorative information in accordance with the motif image MI automatically (e.g., randomly) selected and the position of the composite portion automatically (e.g., randomly) determined when an instruction to start the decorated character generation process is input.

[0123] (5) In the second embodiment, the decorative information associated with the sample decorated character image OIs in the first correspondence information 710 is the motif image MI (e.g., motif images MI1-MI3 in FIG. 5 ). The decorative information associated with the sample decorated character image OIs is not limited to this, and may be information indicating the type and number of related color objects (e.g., circle marks RM, cross marks CM, and star marks SM) used to generate the sample decorated character image OIs. In this case, for example, when a specific sample decorated character image OIs is selected by the user, the image processing server 200 may generate the input image IIf by arranging the related color objects in the character area TA in accordance with the information associated with the specific sample decorated character image OIs.

[0124] (6) The image generation model 900 in each of the above embodiments is a machine learning model called Stable Diffusion, but is not limited to this. Various models trained to generate images decorated with decorative elements such as flowers may be used as the image generation model 900. For example, when a style image and a content image are input, the image generation model 900 may be an image generation model (e.g., a style transfer model called MST (Multimodal Style Transfer)) that generates an image in which the style of the style image is applied to the content image. In this case, for example, by inputting the input images IIf and IIn of this embodiment as content images and the sample decorated character image of this embodiment (e.g., the sample decorated character image OIs1 in FIG. 8 ) as a style image into the MST model, a decorated character image can be generated in which decoration is applied to the characters shown in the input image IIf. Even in this case, the input images IIf and IIn may be generated by the processing described in each of the above embodiments. Alternatively, other known image generation models such as the Adain model, the StyTR2 model, and the U-GAT-IT model may be employed.

[0125] Furthermore, the image generation model 900 may employ a diffusion model other than Stable Diffusion, such as a diffusion model called "VQ-Diffusion" or "GLIDE."

[0126] "VQ-Diffusion" is disclosed in the following paper: Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo, "Vector Quantized Diffusion Model for Text-to-Image Synthesis", arXiv:2111.14822v3, March 3, 2022, https: / / arxiv.org / abs / 2111.14822v3

[0127] "GLIDE" is disclosed in the following paper: Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen, "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models", arXiv:2112.10741v3, March 8, 2022, https: / / arxiv.org / abs / 2112.10741v3

[0128] (7) The decorative character generation process in each of the above embodiments is merely an example and may be modified as appropriate. For example, the character information acquired in S100 of FIG. 3 does not need to include font information indicating the type of font specified by the user. In this case, for example, the font image FI generated in S115 of FIG. 3 is generated using a predetermined font.

[0129] 3, a contour line OL is added to the character motif image TMI of FIG. 4(E) to generate the input image IIf of FIG. 4(F), but the contour line OL does not have to be added. In this case, the image processing server 200 may, for example, fill in the area of ​​the character area TA of the input image IIf other than the circle mark RM with a color different from the background area BA.

[0130] 3, n types of decorated character images OIf with different seed values ​​are generated for each character to be processed. Alternatively, only one type of decorated character image OIf may be generated for each character to be processed. In this case, steps S145 and S150 in FIG. 3 may be omitted.

[0131] 3, an input image IIf is generated for each character to be processed, and a decorated character image OIf is generated for each character. Alternatively, the image processing server 200 may generate an input image including multiple characters and input input information including the input image to the image generation model 900, thereby generating a single decorated character image including multiple decorated characters.

[0132] 10 of the third embodiment, the image processing server 200 may generate a decorated character image OIf by directly using the input image IIn generated in S420, i.e., the input image IIn in which the circle mark RM is placed in the default position. In this case, steps S420 to S475 of FIG. 10 may be omitted.

[0133] 10 , in the decorative character generation process, the input image IIn included in the adjustment instruction input screen WI4 ( FIG. 11 ) has a circle mark RM placed at a default position relative to the character area TA. Alternatively, the adjustment instruction input screen WI4 may display an input image including only the character area TA without the circle mark RM. In this case, the image processing server 200 may receive instructions from the user specifying the placement positions multiple times and place the number of circle marks RM desired by the user at the positions specified by the user, thereby generating the input image IIn in which the circle marks RM are placed.

[0134] 10, the color, position, size, and character color of the circle mark can be changed based on a user instruction, but the color, position, size, and part of the character color of the circle mark may not be changeable.

[0135] In the decorative character generation process of Fig. 10, the color of the circle mark RM and the color of the text can be set to colors similar to each other. Alternatively, the color of the circle mark RM and the color of the text may not be set to colors similar to each other. In this case, for example, the process for setting the outline MOL for the circle mark RM (S430-S440 in Fig. 10) may be omitted.

[0136] (8) In the system 1000 of each of the above embodiments, the terminal device 100 mainly executes processing related to a screen that functions as a user interface (referred to as a UI screen) and processing for controlling the printer 300. Alternatively, the terminal device 100 may execute part of the image processing executed by the image processing server 200 on behalf of the image processing server 200.

[0137] For example, the terminal device 100 may execute the processes of S100-S135 and S145-S165 of the decorated character generation process of Fig. 3. The image processing server 200 may execute only the process of generating the decorated character image OIf using the image generation model 900 of S140. The same applies to the decorated character generation process of Figs. 7 and 10.

[0138] 3 may be executed by the terminal device 100. In this case, the system 1000 may be configured with only the terminal device 100 and the printer 300, and the image processing server 200 may be omitted.

[0139] (9) In each of the above embodiments, the decorative character images OIf are generated by the user of the terminal device 100 in order to print the decorative character images OIf. However, the generation of the decorative character images OIf may also be performed by, for example, the manufacturer of the printer 300, sewing machine 400, or cutting machine 500. In this case, for example, a computer (terminal or server) owned by the manufacturer executes the decorative character generation process shown in FIGS. 3 , 7 , and 10 . In this case, decorative character images OIf are generated for, for example, all letters of the alphabet, and data for the decorative character images OIf is stored in the memory of the printer 300, sewing machine 400, or cutting machine 500 when the printer 300 or sewing machine 400 is manufactured. The printer 300, sewing machine 400, or cutting machine 500 uses the data for the decorative character images OIf to print, embroider, or cut the decorative character images OIf in response to, for example, a user's operation.

[0140] (10) In the above embodiments and modifications, the processor 210 may cause the GPU 260 to execute various operations. Note that the GPU 260 may be omitted.

[0141] (11) The processing executed by the terminal device 100 in Fig. 1 may be executed by, for example, the printer 300, the sewing machine 400, the cutting machine 500, or an image processing device such as a digital camera or a multifunction peripheral. For example, the screens WI1-WI4 in the above embodiment displayed on the display unit 140 of the terminal device 100 may be displayed on the display unit of these image processing devices.

[0142] In each of the above embodiments, a part of the hardware configuration may be replaced with software, and conversely, a part or all of the software configuration may be replaced with hardware. For example, the processing by the image generation model 900 (FIG. 2) may be performed by a dedicated hardware circuit such as an Application Specific Integrated Circuit (ASIC).

[0143] Furthermore, when some or all of the functions of the present disclosure are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and DVD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0144] The above-described examples and modifications can be combined as appropriate. The above-described examples and modifications are provided to facilitate understanding of the present disclosure and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0145] 1000...system, 100...terminal device, 110...processor, 115...storage device, 120...volatile storage device, 130...non-volatile storage device, 131...computer program, 140...display unit, 150...operation unit, 170...communication interface, 200...image processing server, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...computer program, 260...graphics processing unit, 270...communication interface, 300...printer, 400...sewing machine, 710...first correspondence information, 720...second correspondence information Information, 800...motif image group, 900...image generation model, 910...text encoder, 920...image encoder, 930...latent variable model, 940...image decoder, D1-D3...decorative elements, DT...decorative characters, Da...decorative elements, Da, Db...decorative elements, FI...font image, II, IIf, IIn...input image, IT...Internet, LN...local area network, MI...motif image, MKI...character mask image, OI...output image, OIf...decorative character image, OIs...sample decorative character image, OLI...character outline image, TI...output image, TMI...character motif image

Claims

1. An image processing system comprising: an information acquisition unit that acquires, based on user input, character information for specifying characters and decoration information for specifying positions at which decorative elements that constitute decoration are to be placed relative to the characters; an image generation unit that causes an image generation model to generate a decorated character image by inputting input information using the character information and the decoration information into the image generation model, wherein the decorated character image is an image showing a decorated character, that is, the character that has been decorated with decoration including the decorative elements, and the image generation model has been trained to generate an image decorated with decoration including the decorative elements based on the input information; a display control unit that displays the decorated character image on a display unit; and an image forming unit that, when an instruction from a user is received, prints, embroiders, or cuts the decorated character image displayed on the display unit onto an object.

2. An image processing system according to claim 1, wherein the input information includes an input image including a character area indicating a character specified by the character information and a related color object arranged to overlap the character area, and the related color object is an object having a color related to the decorative element.

3. An image processing system as described in claim 2, wherein the decorative information includes a specific image including a plurality of the related color objects, and the image generation unit generates an area image indicating the character area based on the character information, synthesizes the specific image with the character area to generate the input image, and inputs the input information including the input image into the image generation model.

4. An image processing system according to claim 3, wherein the information acquisition unit acquires the specific image by selecting one image from among a plurality of candidate images based on a user instruction.

5. An image processing system as described in claim 3, wherein the image generation unit determines a composite portion of the specific image to be composited with the character area based on a user instruction, and composites the composite portion of the specific image with the character area to generate the input image.

6. An image processing system according to claim 1, further comprising: a sample selection unit that selects one image based on a user instruction from a plurality of sample decorated character images, including a first decorated character image generated by inputting the input information generated using first decorated information into the image generation model, and a second decorated character image generated by inputting the input information generated using second decorated information into the image generation model; wherein the information acquisition unit acquires the first decorated information when the first decorated character image is selected, and acquires the second decorated information when the second decorated character image is selected; and wherein the image generation unit generates the input information using the first decorated information when the first decorated character image is selected, and generates the input information using the second decorated information when the second decorated character image is selected; and inputs the generated input information into the image generation model.

7. An image processing system as described in claim 2, wherein the decorative information is information indicating the position at which the related color object is to be placed relative to the character area, the image processing system further comprising: an input screen display unit that displays a first input screen including an image showing the character area on the display unit; the information acquisition unit acquires the decorative information by having a user input, via the input screen, a position designation instruction that designates the position of the related color object relative to the character area; and the image generation unit generates the input image in which the related color object is placed at the position designated by the designation instruction relative to the character area, and inputs the input information including the generated input image into the image generation model.

8. An image processing system according to claim 2, wherein the input image includes a first contour line indicating the contour of the character region.

9. An image processing system according to claim 2, wherein the character information includes font information indicating a type of font, and the shape of the character area has the shape of the font indicated by the font information.

10. An image processing system as defined in claim 1, wherein the information acquisition unit acquires the character information specifying k characters (k is an integer of 2 or more), the image generation unit generates k pieces of input information corresponding to the k characters, and generates the k decorative character images by inputting the k pieces of input information one by one into the image generation model, and the image formation unit prints or embroiders an image including the k decorative character images.

11. An image processing system as described in claim 1, wherein the input information generation unit generates n types of input information using one piece of character information and one piece of decorative information; the image generation unit generates n types of decorative character images by inputting the n types of input information one by one into the image generation model; the display control unit displays the n types of decorative character images on the display unit; and the image forming unit prints or embroiders an image selected by a user from the n types of decorative character images onto the object.

12. An image processing system according to claim 1, wherein the decorative information includes information for specifying at least one of the color and size of the decorative element.

13. An image processing system as described in claim 7, wherein the information acquisition unit allows input of the position designation instruction that designates a position that overlaps with the character area, and does not allow input of the position designation instruction that designates a position that does not overlap with the character area.

14. An image processing system as described in claim 2, further comprising: a memory area for storing correspondence information that associates the amount of decoration with the number and positions of the related color objects to be placed in the character area; the information acquisition unit acquires from a user an amount designation instruction that designates the amount of decoration; and, by referring to the target information, acquires, as the decoration information, the number and positions of the related color objects that correspond to the amount of decoration designated by the amount designation instruction; and the image generation unit generates the input image using the number and positions of the related color objects.

15. An image processing system as described in claim 2, wherein the information acquisition unit further acquires an instruction from a user specifying at least one of the color of the related color object and the color of the character area, and the image generation unit, if the color of the related color object and the color of the character area are similar, adds a second contour line to the related color object, the second contour line having a color different from the color of the related color object, and generates the input image including the related color object with the second contour line added and the character area.

16. A method for generating a decorated character image, which is an image showing decorated characters decorated with decorative elements, comprising: a first step of generating an input image showing a character area showing characters and a related color object arranged to overlap the character area, wherein the related color object is an object having a color related to the decorative element; and a second step of inputting input information including the input image into an image generation model, thereby causing the image generation model to generate the decorated character image, wherein the image generation model has been trained to generate an image decorated with decorations including the decorative elements based on the input information.

Images