Method and apparatus for transforming an input image based on target style and target color information

Abstract

To provide an image transformation method to obtain a result image that represents a texture corresponding to a target style input by a user and a color corresponding to target color information from an input image, using a trained transformation model.SOLUTION: An image transformation method, which is executed by using a pretrained transformation model, includes: receiving an input image 205, a target style and target color information 210 for transformation; generating a texture-transformed image obtained by transforming the input image to a texture corresponding to the target style and a color-transformed image obtained by transforming the input image to a color corresponding to the target color information, using the pretrained transformation model 250; and synthesizing the texture-transformed image and the color-transformed image to generate a result image 260 formed by transforming the input image according to the target style and the target color information.SELECTED DRAWING: Figure 2B

Description

[Technical Field]

[0001] The present disclosure relates to a method and apparatus for transforming an input image according to input target style and target color information, and more particularly to a method and apparatus for generating a resultant image from an input image, exhibiting a texture corresponding to the target style and a color corresponding to the target color information, using a pre-trained transformation model. [Background technology]

[0002] There has been a great deal of interest in services that provide online content including images or videos, such as manga, cartoon, and webtoon services. The images or videos included in such content have a particular texture (or style) that reflects the unique artistic style of the artist or content creator.

[0003] To efficiently generate images and videos for content, content creators often convert general images, such as photographs, into images that reflect the specific style of the content. In particular, when creating complex content images, such as backgrounds, content creators often convert general images into the style of the content rather than drawing the background directly. Similarly, users who consume content may want to obtain their own images in a way that reflects the specific style of the content for secondary consumption.

[0004] At this time, it is required not only to convert the image input by the user into an image that reflects the specific style of the content, but also to reflect the color information (or color sense) and degree of abstraction input by the user, so that a resulting image that meets the user's detailed requirements is obtained.

[0005] Patent Document 1 (published on October 28, 2009) discloses a method and device for separating an image into a foreground and a background, a method and device for separating an image into a foreground and a background and substituting the background, and a computer-readable recording medium on which a program for executing the method is recorded.

[0006] The information above is for ease of understanding only and may include material that does not form part of the prior art, or that the prior art would present to one skilled in the art. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent Publication No. 10-2009-0111939 Summary of the Invention [Problem to be solved by the invention]

[0008] One embodiment provides an image conversion method that receives an input image and target style and target color information for conversion, uses a pre-trained conversion model to generate a texture-converted image that converts the input image to the texture of the target style and a color-converted image that converts the input image to the color of the target color information, and combines the texture-converted image and the color-converted image to generate a resultant image in which the input image is converted according to the target style and target color information. [Means for solving the problem]

[0009] In one aspect, an image conversion method executed by a computer system is provided, comprising the steps of receiving an input image, a target style to which the input image is to be converted, and target color information to which the input image is to be converted; generating a texture-converted image by converting the input image to a texture corresponding to the target style using a pre-trained conversion model, and a color-converted image by converting the input image to a color corresponding to the target color information; and combining the texture-converted image and the color-converted image to generate a resultant image in which the input image is converted according to the target style and the target color information.

[0010] The conversion model may be pre-trained to convert an input image to represent a texture corresponding to the target style and a color corresponding to input target color information without inputting a reference image indicating the target style.

[0011] The target color information may include a palette containing a group of different colors determined based on the color distribution of the input image.

[0012] Each of the plurality of colors may be configured to be selectable by the user.

[0013] The target style may indicate content including an image or a video, and the texture corresponding to the target style may be the texture of the image or video included in the content.

[0014] The receiving step may include receiving an abstraction level for setting a degree of abstraction represented by the resultant image, and the generating step may generate the texture-converted image by converting the input image into a texture corresponding to the target style that reflects the degree of abstraction represented by the abstraction level.

[0015] The image conversion method may further include, as a step of pre-processing the input image, converting the input image into a first converted image which is an image in Lab color space, generating a first color map by simplifying the input image, generating a second color map by converting colors of the first color map based on the target color information, and converting the second color map into a second converted image which is an image in Lab color space, wherein the first converted image and the second converted image are input to the conversion model, and the generating step may involve the conversion model generating the texture converted image based on the first converted image and generating the color converted image based on the second converted image.

[0016] The generating step may include the steps of: encoding the first transformed image using an encoder; generating the texture transformed image by decoding the encoded first transformed image using a texture decoder; and generating the color transformed image by decoding the encoded first transformed image based on the second transformed image using a color decoder.

[0017] The texture-converted image may be an image in Lab color space that is an image of an L component, and the color-converted image may be an image in Lab color space that is an image of an ab component, and the combining step may include generating a combined image by combining the L component image and the ab component image, and converting the combined image into an image in RGB color space.

[0018] The conversion model may include an abstraction level reflection module configured with a plurality of layers, each layer generating an output value reflecting a degree of abstraction indicated by each abstraction level, and the generating step may include converting the input image into a texture corresponding to the target style reflecting the degree of abstraction indicated by the received abstraction level using an output value of a layer of the abstraction level reflection module corresponding to the received abstraction level, thereby generating the texture-converted image.

[0019] The conversion model may be pre-trained using first training data generated by data augmenting at least one first image containing a texture corresponding to the target style, and second training data generated by data augmenting at least one second image that is an image to be converted to the target style and color converted.

[0020] The first training data may include an image that has undergone at least one of a resizing process and a resolution change process so as to replicate a predetermined degree of abstraction processing on the first image.

[0021] The second learning data may include a converted learning image obtained by converting the second image into an arbitrary color.

[0022] The training converted image is generated by a conversion step, which may include converting at least one of the second image and a color map of the second image generated by simplifying the second image into an image in HSV color space, and converting the image in HSV color space into an arbitrary color.

[0023] The converting step may include converting at least one of the second image and the color map of the second image into an image in Lab color space, extracting L information from the image in La color space, converting the image in HSV color space converted to an arbitrary color into an image in Lab color space, and replacing the L information of the converted image in Lab color space with the extracted L information.

[0024] The conversion model may be one that has been trained in advance using an image obtained by converting the first training data into an image of the L component of Lab color space, an image obtained by converting the second image into an image of Lab color space, an image obtained by converting the second image into an arbitrary color and then converting it into an image of the ab component of Lab color space, and an image obtained by converting the color map of the second image into an image in Lab color space.

[0025] In another aspect, a computer system for executing an image conversion method is provided, the computer system including at least one processor implemented to execute instructions readable by the computer system, wherein the at least one processor receives an input image, a target style to which the input image is to be converted, and target color information to which the image is to be converted, and uses a pre-trained conversion model to combine the texture-converted image and the color-converted image to generate a texture-converted image in which the input image is converted to a texture corresponding to the target style, and a resultant image converted in accordance with color conversion information in which the input image is converted to a color corresponding to the target color information.

[0026] In yet another aspect, there is provided a method for training a conversion model for converting an image, which is executed by a computer system, comprising the steps of: acquiring training data obtained by data-extending at least one first image having a texture corresponding to a target style and at least one second image that is an image to be converted to the target style and color-converted; and training the conversion model using the training data to generate a texture-converted image in which the second image is converted to a texture corresponding to the target style and a color-converted image in which the second image is converted to a color corresponding to target color information; wherein the texture-converted image and the color-converted image are combined to generate a resultant image in which the second image is converted according to the target style and the target color information; and the trained conversion model converts an input image to represent a texture corresponding to the target style and a color corresponding to the input target color information without inputting a reference image indicating the target style. [Effects of the Invention]

[0027] Using the trained transformation model, a resultant image can be obtained from an input image, which exhibits texture corresponding to the target style and color corresponding to the target color information entered by the user.

[0028] Using the trained transformation model, a resultant image can be obtained that represents the texture of the target style and the color of the target color information, while reflecting the abstraction level input by the user to set the degree of abstraction that the resultant image will represent.

[0029] The result image can be obtained without inputting a reference image showing the target style by using a pre-trained transformation model to convert the input image to represent a texture corresponding to the target style and a color corresponding to the input target color information, without inputting a reference image.

[0030] By using a trained conversion model to obtain and use a resultant image that reflects user-controllable target color information and abstraction level from an input image, it is possible to reduce the time and resources required to create background parts excluding foreground parts such as characters when creating content such as webtoons, cartoons, and animations. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 2 illustrates an image transformation method for transforming an input image according to target style and target color information to generate a resultant image in one embodiment. [Figure 2] FIG. 2a illustrates a computer system for performing an image transformation method in one embodiment, and FIG. 2b illustrates an image transformation method performed using a pre-trained transformation model in one embodiment. [Figure 3] 1 is a flowchart illustrating an image transformation method for transforming an input image according to a target style and the target color information to generate a resultant image, in one embodiment. [Figure 4] 4a and 4b illustrate a method for pre-processing an input image in one embodiment. [Figure 5] FIG. 1 illustrates a method for generating texture-transformed and color-transformed images from an input image using a pre-trained transformation model in one example. [Figure 6] 1 is a flowchart illustrating a method for generating a result image in one example. [Figure 7] FIG. 2 illustrates a transformation model that transforms an input image according to a target style and the target color information to generate a resultant image in one example. [Figure 8] FIG. 1 illustrates a method for training a transformation model in one example. [Figure 9] 9a and 9b are a flowchart illustrating a method for obtaining training data for a transformation model in one example. [Figure 10] 10a and 10b illustrate a method for obtaining training data for a transformation model in one example. [Figure 11] FIG. 10 is a diagram illustrating a structure of an abstraction level reflection module included in a conversion model in an example. [Figure 12] FIG. 1 illustrates the structure of an identifier network in one example. [Figure 13] FIG. 10 illustrates a resultant image, in one example, generated based on an input image and reflecting target style and target color information. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0033] FIG. 1 illustrates an image transformation method for transforming an input image according to target style and target color information to generate a resultant image in one embodiment.

[0034] Referring to Figure 1, we will explain a method for converting an image (hereinafter referred to as an input image) input from a user, a target style, target color information, and an abstraction level into a result image that reflects the target style, target color information, and abstraction level (i.e., a method for generating a result image).

[0035] An image may include a photograph or other image, while in embodiments, an image may mean a picture, but should not be limited to this, and may be interpreted to include a video.

[0036] The screen 50 may be a screen of a user terminal that displays an input image input by a user and a resultant image obtained by converting the input image according to settings made by the user. That is, the screen 50 may be an execution screen of software, a program, or an application that executes the image conversion method of the embodiment.

[0037] The user terminal may be a smart device such as a smartphone, a personal computer (PC), a laptop computer, a tablet, an Internet of Things device, or a wearable computer.

[0038] As shown in the figure, the screen 50 may include, for example, a first area 10 displaying an input image input by a user and a resultant image resulting from the conversion of the input image, a second area 20 including a user interface for editing the image displayed in the first area 10, and third areas 30, 40 including UIs for the user to set at least one of a target style, target color information, and abstraction level. Here, the third areas 30, 40 may include an area 30 including a UI for setting the target style into which the input image will be converted and a UI for setting the abstraction level for setting the degree of abstraction shown in the resultant image. The third areas 30, 40 may also include an area 40 including a UI for setting target color information into which the input image will be converted.

[0039] As an example, assume that an input image input by a user is displayed in the first area 10. The user may load an image stored in the user terminal or an image captured using the camera of the user terminal to display the input image in the first area 10. The input image is an image before conversion and may be called an "original image." The input image may be converted into a "result image" by the image conversion method of the embodiment, and the result image may also be displayed in the first area 10. The result image may be displayed in the first area 10 instead of the input image.

[0040] The second area 20 is a UI for editing the input image or the resultant image displayed in the first area 10, and may include editing tools such as a brush and / or an eraser. The second area 20 may also include a UI for undoing and redoing an action. Meanwhile, the second area 20 may also include a UI for switching between displaying the input image and displaying the resultant image in the first area 10.

[0041] Area 30 of the third areas 30 and 40 may include a UI for setting a target style. The target style may indicate content including an image or video. That is, the target style may be expressed by such a "content name." The content may be, for example, a comic, cartoon, or webtoon. The target style may indicate a texture to which an input image is converted. That is, the image conversion method of the embodiment converts the texture of the input image into a texture corresponding to the target style. In this case, the texture corresponding to the target style may be the texture of an image or video included in the content. The texture may include the style, design, or other features of the image or video included in the content to visually distinguish the content from other content. For example, when a user selects the name of a specific webtoon as the target style in area 30, the image conversion method of the embodiment may convert the texture of the input image into the texture of the selected webtoon.

[0042] Additionally, area 30 may include a UI for setting an abstraction level indicating the degree of abstraction of the resulting image. "Abstraction" may indicate the degree of detail included in the resulting image, the thickness of the lines representing the resulting image, etc. Alternatively, "abstraction" may indicate the degree of simplification of the resulting image. For example, a higher level of abstraction may result in the resulting image containing less detail, and the lines of the resulting image may be less clear or softly expressed. Conversely, a higher level of abstraction may result in the resulting image containing more detail, and the resulting image may be more precisely expressed. In an embodiment, by setting the abstraction level in area 30, the degree of abstraction of the resulting image may be adjusted according to the set abstraction level.

[0043] Of the third areas 30 and 40, area 40 may include a UI for setting target color information. The target color information may indicate a color to which the input image is to be converted. That is, the image conversion method of the embodiment converts the color of the input image to a color corresponding to the target color information. The color corresponding to the target color information may refer to the color of the resulting image. For example, when a user selects a specific color (or color tone) (hue and saturation) in area 40, the image conversion method of the embodiment may convert the color of the input image to the selected color. As a result, the resulting image will have the same color as the target color information selected in area 40.

[0044] Therefore, according to the image conversion method of the embodiment, the input image can be converted into a result image that reflects the set target style, the set target color information, and the set abstraction level according to the information set in the third area 30, 40.

[0045] The image conversion method of the embodiment may be executed by a user terminal that outputs the screen 50. Alternatively, the image conversion method may be executed by a computer system that communicates with the user terminal but is separate from the user terminal. For example, the image conversion method may be executed by a server.

[0046] In the following detailed description, a computer system that implements such an image conversion method will be referred to as computer system 100.

[0047] In an embodiment, the computer system 100 may convert an input image into a result image using a pre-trained conversion model. In this case, the computer system 100 may convert the texture of the input image into a texture corresponding to a set target style (1) and convert the color of the input image into a color corresponding to set target color information (2). The conversion model may perform the texture conversion of the input image (1) and the color conversion of the input image (2) as separate processes. As a result, the computer system 100 may use the conversion model to generate a texture-converted image by converting the input image into a texture corresponding to the target style, generate a color-converted image by converting the input image into a color corresponding to the target color information, and generate a result image by combining the texture-converted image and the color-converted image.

[0048] The specific manner in which the transformation model is used to perform texture transformation of an input image and color transformation of an input image, and the specific manner in which the resulting image is generated, will be described in more detail with reference to FIGS.

[0049] FIG. 2a illustrates a computer system that implements the image transformation method in one embodiment.

[0050] The computer system 100 may be any computing device that performs the tasks necessary to carry out the image conversion method of the embodiments.

[0051] The computer system 100 may be configured to include at least one computing device. The computer system 100 may perform texture transformation and color transformation of an input image using a pre-trained transformation model to generate a resultant image that reflects settings from a user.

[0052] The computer system 100 may be the user terminal described above, or may be another computer device or server that communicates with the user terminal.

[0053] Computer system 100 may include memory 130, processor 120, communication section 110, and input / output interface 140 as shown.

[0054] Memory 130 is a computer-readable recording medium and may include random access memory (RAM), read-only memory (ROM), and a persistent mass storage device such as a disk drive. Here, a persistent mass storage device such as a ROM or a disk drive may be included as a separate persistent storage device separate from memory 130. An operating system and at least one program code may also be stored in memory 130. Such software components may be loaded from a computer-readable recording medium separate from memory 130. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In another embodiment, software components may be loaded into memory 130 via communication unit 110, which is not a computer-readable recording medium.

[0055] Processor 120 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to processor 120 by memory 130 or by communication unit 110. For example, processor 120 may be configured to execute instructions received according to program code loaded into memory 130.

[0056] The communication unit 110 may be a component that allows the computer system 100 to communicate with other devices (such as a user terminal or another server). That is, the communication unit 110 may be a hardware module, such as an antenna, a data bus, a network interface card, a network interface chip, or a networking interface port of the computer system 100, that transmits and receives data and / or information to and from other devices, or a software module, such as a network device driver or a networking program.

[0057] The input / output interface 140 may be a means for interfacing with input devices such as a keyboard, a mouse, etc., and output devices such as a display, a speaker, etc.

[0058] The processor 120 may manage the components of the computer system 100, may execute programs or applications for performing the above-described preprocessing, similar cut pair determination, and matching, and may process operations necessary for executing the programs or applications and processing data, etc. The processor 120 may be at least one processor (such as a CPU or GPU) of the computer system 100 or at least one core within a processor.

[0059] In other embodiments, the computer device 100 and the processor 120 may include fewer or more components than those shown in the figures. For example, the processor 120 may include a configuration that performs functions for training a transformation model and using the trained transformation model to execute the image conversion method of an embodiment. Such a configuration of the processor 120 may be a part of the processor 120 or may be a function realized by the processor 120. The configuration included in the processor 120 may represent different functions that the processor 120 executes in response to control instructions from the operating system code and the code of at least one computer program.

[0060] The specific manner in which the computer system 100 uses the transformation model to perform texture transformation of the input image and color transformation of the input image, and the specific manner in which the resulting image is generated, will be described in more detail with reference to FIGS. 2b-13.

[0061] In this regard, FIG. 2b illustrates an image transformation method performed using a pre-trained transformation model in one embodiment.

[0062] FIG. 2b illustrates how a pre-trained transformation model 250 is used to generate a result image 260 based on an input image 205 (i.e., how the input image 205 is transformed into the result image 260).

[0063] The input data pre-processing unit 230 and the conversion model 250 may be implemented using the processor 120. That is, the operations performed by the input data pre-processing unit 230 and the conversion model 250 may be operations performed by the processor 120.

[0064] As shown in the figure, computer system 100 may receive input of an input image 205 and target color information 210. Computer system 100 may also receive input of a target style and an abstraction level indicating the degree of abstraction of the resulting image 260. As shown in the figure, target color information 210 may be configured as a palette containing a group of different colors. Alternatively, unlike what is shown in the figure, target color information 210 may indicate a single selected color.

[0065] The palette included in the target color information 210 may include a group of multiple different colors determined based on the color distribution of the input image 205. For example, the number of colors included in the palette and / or the type of each color may be determined based on the color distribution of the input image 205. The palette of the target color information 210 shown in the figure indicates a group of colors that reflects the color distribution of the current input image 205. In other words, the palette shown in the figure may be a source palette. The number and / or type of colors included in the palette may be determined based on the number and / or type of colors used in the input image 205.

[0066] For example, the user can select a palette containing a different group of colors (e.g., a group of colors that generally represent red tones) using the UI in the above-described area 40 to change the target color information 210. Alternatively, each of the multiple colors included in the palette may be configured to be selectable by the user, allowing the user to change each color in the palette.

[0067] The input data preprocessor 230 may preprocess the input image 205 based on the target color information 210 and generate data for input to the transformation model 250. The method for preprocessing the input image 205 is described in more detail with reference to FIG.

[0068] The data preprocessed by the input data preprocessing unit 230 may be input to the conversion model 250. The conversion model 250 may include a shared encoder 252 for encoding the preprocessed data of the input image 205. The conversion model 250 may include a texture decoder 256 for performing decoding for texture conversion on the encoded data and a color decoder 254 for performing decoding for color conversion on the encoded data. A texture-converted image may be generated by converting the input image 205 into a texture corresponding to the target style through decoding by the texture decoder 256, and a texture-converted image may be generated by converting the input image 205 into a color corresponding to the target color information through decoding by the color decoder 254. The texture conversion result by the conversion model 250 may reflect a degree of abstraction corresponding to the target style and abstraction level input by the user. The detailed structure of the conversion model 250 will be described in more detail with reference to FIG. 7.

[0069] The computer system 100 may generate a resultant image 260 by combining the texture conversion result and the color conversion result obtained by the conversion model 250. Therefore, the generated resultant image 260 reflects the texture of the input target style and the degree of abstraction corresponding to the abstraction level, and the color of the resultant image 260 is the same as the input target color information 210.

[0070] In this regard, FIG. 13 shows a resultant image, in one example, generated based on an input image and reflecting target style and target color information.

[0071] FIG. 13 shows an example in which an input image on the left side is converted into a resultant image with a texture corresponding to a target style. The resultant image may have a higher level of abstraction set toward the right. That is, the resultant image on the right side contains less detail and is softer. As shown in the figure, the resultant image may show colors corresponding to the input target color information 1310 to 1340. Each target color information is illustrated as a palette containing multiple color groups. The resultant image may be expressed using the colors indicated by these palettes.

[0072] In this way, the image conversion method of the embodiment converts the input image into a resultant image that reflects the input target style, target color information, and abstraction level.

[0073] The above description of the technical features explained with reference to FIG. 1 can also be applied to FIGS. 2 and 13, so duplicated description will be omitted.

[0074] The following describes in more detail how the above-described transformation model 250 is used to transform the input image 205 into the result image 260.

[0075] In the following detailed description, operations performed by computer system 100, or processor 120, or components thereof, are described as operations performed by computer system 100 for convenience of explanation.

[0076] FIG. 3 is a flow chart illustrating an image transformation method for transforming an input image according to target style and target color information to generate a resultant image, in one embodiment.

[0077] In step 310, the computer system 100 may receive an input image 205, a target style to which the input image 205 is to be transformed, and target color information 210 to which the input image 205 is to be transformed. For example, the computer system 100 may receive the input image 205, the target style, and the target color information 210 input by a user using a user terminal.

[0078] In step 320, the computer system 100 may use the pre-trained transformation model 250 to generate a texture-transformed image by transforming the input image 205 into a texture corresponding to the target style, and may generate a color-transformed image by transforming the input image 205 into a color corresponding to the target color information 210.

[0079] Meanwhile, the computer system 100 may further receive an abstraction level for setting the degree of abstraction indicated by the resultant image 260. As a result, the computer system 100 can generate a texture-converted image by converting the input image 205 into a texture corresponding to a target style that reflects the degree of abstraction indicated by the received abstraction level.

[0080] At step 330, computer system 100 may combine the texture-transformed image and the color-transformed image to generate a resultant image 260 that transforms input image 205 according to received target style and target color information 210. For example, computer system 100 may generate resultant image 260 by coloring the texture-transformed image with the color of the color-transformed image.

[0081] In this manner, the transformation model 250 of the embodiment can separately perform texture transformation on the input image 205 and color transformation on the input image 205. The transformation model 250 may be trained by separately calculating loss functions for the texture-transformed image and the color-transformed image before the resultant image 260 is generated.

[0082] At step 340, the computer system 100 may output the resultant image 260 on the user terminal (eg, the first area 10 of the screen 50).

[0083] The transformation model 250 of the embodiment may be pre-trained to transform an input image to represent a texture corresponding to the target style and a color corresponding to the input target color information 210, without inputting a reference image representing the target style. That is, the transformation model 250 can generate the result image 260 based on the input image 205 without needing to input another image having a texture of the target style in addition to the input image 205.

[0084] Meanwhile, the input image 205 received in step 310 may be preprocessed before being input to the transformation model 250. That is, in step 315, the computer system 100 may preprocess the input image 205 to generate data suitable for input to the transformation model 250. The method for preprocessing the input image 205 will be described in more detail with reference to FIG.

[0085] The technical features explained above with reference to FIGS. 1, 2, and 13 can also be applied to FIG. 3, so duplicated explanations will be omitted.

[0086] 4a and 4b illustrate a method for pre-processing an input image in one example.

[0087] In Figure 4b, input image 440 may correspond to input image 205 described above. Input image 440 may be, for example, an image in RGB color space, such as I src RGB It may be expressed as:

[0088] The computer system 100 receives an input image 440I src RGBThe input image 440 converted to an image in the Lab color space (RGB2Lab) may be converted to an image in the Lab color space. The Lab color space may be a color space expressed by a combination of brightness (L) and red, green, and yellow, blue (a Channel, b Channel). In other words, the L component of the Lab color space can represent brightness, and the ab component of the Lab color space can represent color. The input image 440 converted to an image in the Lab color space is I src RGB It can be expressed as I src RGB may be input to the transformation model 250.

[0089] The computer system 100 also receives an input image 440I src RGB A color map 445 corresponding to the input image 440 may be generated by performing an image simplification process on C. src RGB The image simplification process may be performed, for example, based on the superpixel algorithm.

[0090] The color map 445 may be recolored based on the target color information 455 entered by the user. The computer system 100 may also generate a source palette of the input image 440 based on the color distribution of the input image 440. (outside 1) You can generate JPEG0007765098000001.jpg1326. Source Palette (outside 2) JPEG0007765098000002.jpg1326 may be generated based on, for example, a K-means clustering algorithm. The source palette may also be called an initial palette. The computer system 100 may be configured to generate the source palette by the user. (Outside 3) By editing JPEG0007765098000003.jpg1326, the palette corresponding to target color information 455 is (outside 4) For example, the source palette (outside 5) At least one of the colors constituting JPEG0007765098000005.jpg1326 is changed by user input, and a palette corresponding to target color information 455 is created. (outside 6) JPEG0007765098000006.jpg1523 may be generated. A palette corresponding to the target color information 455 (outer 7) The color map 445 is recolored based on JPEG0007765098000007.jpg1523 to produce a color map 465 that reflects the target color information 455. (outside 8) JPEG0007765098000008.jpg1426 may be generated. (outer 9) JPEG0007765098000009.jpg1426 may be converted to an image in Lab color space, and the color map converted to such an image in Lab color space is (Outside 10) JPEG0007765098000010.jpg1326. Meanwhile, before the recoloring is performed, the unedited color regions are represented as color map 445C src RGB To ensure that the input image 440I remains the same as src RGB is the source palette (Outside 11) A color segmentation mask (M) 460 may be calculated for each color of JPEG0007765098000011.jpg1326, and masking using such a mask may be performed.

[0091] For example, input image 440 and source palette (Outside 12) The image mask (M) may be generated based on JPEG0007765098000012.jpg1326. The computer system 100 may extract all pixels in the source palette from the input image 440. (Outside 13) The mask may be generated by identifying the closest color in JPEG0007765098000013.jpg1326 and filling in the mask area corresponding to the identified color. (Outside 14) JPEG0007765098000014.jpg1523 and source palette (Outside 15) Using JPEG0007765098000015.jpg1326, the color map 445 may be recolored. At this time, for every pixel in the color map 445, the computer system 100 may identify which color in the generated mask each pixel corresponds to, and then match the identified colors with the palette corresponding to the target color information 455. (Outside 16) The color map 445 may be recolored by comparing the colors of JPEG0007765098000016.jpg1523.

[0092] The color map converted to the final generated image in Lab color space. (Outside 17) JPEG0007765098000017.jpg1326 may be the input to the transformation model 250.

[0093] 4a, the method for pre-processing the input image 440 will be described again. In step 405, the computer system 100 may convert the input image 440 into a first transformed image, which is an image in the Lab color space. The first transformed image is the I src Lab which may be an input to the transformation model 250.

[0094] In step 410, the computer system 100 may generate a first color map by simplifying the input image 440. For example, the simplification may be based on a superpixel algorithm or another image quantization algorithm. The first color map may be the color map 445C described above. src RGB may correspond to.

[0095] In step 420, the computer system 100 may generate a second color map by converting the colors of the first color map based on the received target color information. For example, the first color map may be recolored with colors corresponding to the target color information, and a second color map may be generated according to such recoloring. The second color map may be the color map 465 described above. (Outside 18) JPEG0007765098000018.jpg1426 may be supported.

[0096] In step 430, the computer system 100 may convert the second color map into a second transformed image, which is an image in the Lab color space. The second transformed image produced by such a conversion may be a second transformed image, which is an image in the Lab color space, as described above. (Outside 19) JPEG0007765098000019.jpg1326, which may be the input to the transformation model 250.

[0097] The transformation model 250 is a first transformed image I src Lab The texture transformed image can be generated based on the second transformed image (outside 20) The above-described color-converted image can be generated based on JPEG0007765098000020.jpg1326.

[0098] The technical features explained above with reference to FIGS. 1 to 3 and 13 can also be applied to FIG. 4, and therefore a duplicated explanation will be omitted.

[0099] FIG. 5 is a diagram illustrating a method for generating a texture-transformed image and a color-transformed image from an input image using a pre-trained transformation model in one example.

[0100] The method for generating the texture-transformed and color-transformed images described above using the transformation model 250 will now be described in more detail with reference to steps 510-530.

[0101] As described with reference to FIG. 2b, the transformation model 250 may include a shared encoder 252 for encoding preprocessed data of the input image 440, a texture decoder 256 for performing decoding for texture transformation on the encoded data, and a color decoder 254 for performing decoding for color transformation on the encoded data.

[0102] At step 510, the transformation model 250 may use the encoder 252 to encode the first transformed image generated based on the pre-processing of the input image 440 described above.

[0103] In step 520, the transformation model 250 may generate a texture-transformed image by decoding the first transformed image encoded by the encoder 252 using the texture decoder 254. The texture-transformed image may be one in which the texture of the input image 440 has been transformed into a texture corresponding to the target style.

[0104] In step 530, the transformation model 250 may generate a color-transformed image using the color decoder 254 based on the second transformed image generated based on the pre-processing of the input image 440, for example, by decoding the encoded first transformed image using the second transformed image as condition information. By inputting the second transformed image to the color decoder 254, a color corresponding to the target color information may be reflected in the color-transformed image.

[0105] The computer system 100 can generate a final result image by combining the texture-transformed image generated in step 520 and the color-transformed image generated in step 530 .

[0106] The technical features explained above with reference to FIGS. 1 to 4 and 13 can also be applied to FIG. 5, and therefore overlapping explanations will be omitted.

[0107] FIG. 6 is a flow chart illustrating a method for generating a result image in one embodiment.

[0108] The texture-transformed image described with reference to FIG. 5 may be an image in Lab color space and an image with an L component. That is, the texture-transformed image does not need to have an ab component. The color-transformed image may be an image in Lab color space and an ab component. The color-transformed image may be a color map of the ab channel to generate the result image.

[0109] In step 610, the computer system 100 may generate a composite image by combining the image of the L component of the texture-transformed image and the image of the ab component of the color-transformed image. The composite image may be an image in Lab color space.

[0110] In step 620, the computer system 100 may convert the composite image, which is an image in Lab color space, to an image in RGB color space. The image finally converted to an image in RGB color space may be the result image described above.

[0111] The steps 610 and 620 described above may be operations performed by the transformation model 250 .

[0112] The technical features explained above with reference to FIGS. 1 to 5 and 13 can also be applied to FIG. 6, and therefore overlapping explanations will be omitted.

[0113] FIG. 7 illustrates a transformation model for generating a resultant image by transforming an input image according to a target style and the target color information in one example.

[0114] Referring now to FIG. 7, the method for generating the texture-transformed and color-transformed images described with reference to FIGS. 5 and 6, and the method for generating the resultant image, will be described in more detail.

[0115] 7 may be a representation of the above-described conversion model 250. Such a conversion model may be an artificial intelligence (AI)-based model, such as a deep learning-based model. The conversion model may include a pre-trained VGG model (e.g., a VGG19 network).

[0116] The transformation model may include a shared encoder 730 , a texture decoder 740 , and a color decoder 750 .

[0117] The shared encoder 730 converts the input image 440 into a first transformed image I src Lab 710. The joint encoder 730 may consist of multiple convolutional layers and residual blocks.

[0118] The texture decoder 740 may decode the encoding result from the shared encoder 730 to generate a texture-transformed image 770. That is, the texture decoder 740 may generate an image reflecting a texture corresponding to the received target style. The generated texture-transformed image 770 may be an image of the L component of the Lab color space. Furthermore, the texture-transformed image 770 generated by the texture decoder 740 may reflect the degree of abstraction indicated by the received abstraction level.

[0119] To this end, the transform module 250 may include an abstraction level reflecting module 760 configured with multiple layers, including layers that generate output values ​​reflecting the degree of abstraction indicated by the respective abstraction levels. The abstraction level reflecting module 760 may be included in the texture decoder 740. That is, the texture decoder 740 may include the abstraction level reflecting module 760 and multiple convolution layers.

[0120] The abstraction level reflection module 760 will be described in more detail below with reference to FIG.

[0121] In relation to this, FIG. 11 is a diagram showing the structure of an abstraction level reflection module included in a conversion model in one example.

[0122] The illustrated abstraction level reflection module 1100 is a more detailed representation of the abstraction level reflection module 760. As illustrated, the abstraction level reflection module 1100 may include multiple layers (or residual blocks) and a gating module. Each layer may be configured to generate an output value that reflects the abstraction level indicated by the respective abstraction level. The abstraction level reflection module 1100 may also be referred to as a continuous abstraction module (CAM).

[0123] The transformation module 250 (the texture decoder 740 of the transformation module 250) may use the output value of the layer of the abstraction level reflection module 1100 corresponding to the abstraction level received from the user to convert the input image 440 into a texture corresponding to the target style reflecting the abstraction level indicated by the received abstraction level, thereby generating a texture-converted image 770.

[0124] In the illustrated example, alpha α may represent the received abstraction level. The abstraction level reflection module 1100 may use the output value of the layer that corresponds to the received abstraction level α to generate the texture-converted image 770.

[0125] As an example, the abstraction level reflection module 1100 may determine which layer (residual block) of feature maps to use based on the received abstraction level α, and may use the determined feature maps to calculate one final feature map using a sum operation to generate the texture-converted image 770.

[0126] This allows the generated texture-transformed image 770 to reflect not only the texture of the received target style, but also the degree of abstraction indicated by the received abstraction level.

[0127] The color decoder 750 may be composed of a residual block and multiple convolution layers. The color decoder 750 decodes the encoding result of the shared encoder 730, but does not decode the second transformed image. (outside 21) JPEG0007765098000021.jpg1326720 may be used as an additional input to perform decoding and generate a color-transformed image 780 that reflects colors corresponding to the target color information. That is, the color decoder 750 can generate an image that reflects colors (color sense, palette, etc.) corresponding to the received target color information. The generated color-transformed image 780 may be an image of the ab component in the Lab color space.

[0128] The texture-transformed image 770 and the color-transformed image 780 may be combined to generate a resultant image 790. The combined image obtained by combining the texture-transformed image 770 and the color-transformed image 780 may be (outside 22) It may also be called JPEG0007765098000022.jpg1529. Composite image (outside 23) JPEG0007765098000023.jpg1529 may be an image in Lab color space, and the composite image (outside 24) JPEG0007765098000024.jpg1529 may be converted to an image in the RGB color space. The converted image in the RGB color space becomes the resultant image 790, which is (Outside 25) It may also be called JPEG0007765098000025.jpg1627.

[0129] As described above, the transformation module 250 can use the texture decoder 740 and color decoder 750 to separately generate the texture-transformed image 770 and the color-transformed image 780, which can then be combined to generate the result image 790.

[0130] Therefore, the final result image 790 reflects the target style, target color information, and degree of abstraction received from the user.

[0131] The technical features explained above with reference to FIGS. 1 to 6 and 13 can also be applied to FIGS. 7 to 11, and therefore, redundant explanations will be omitted.

[0132] A method for constructing a pre-trained transformation model 250 by training the above-described transformation model 250 will be described in more detail below with reference to FIGS.

[0133] FIG. 8 illustrates a method for training a transformation model in one example.

[0134] The computer system that trains the conversion model 250 may be a device or server separate from the above-described computer system 100. For ease of explanation, the following description of the embodiment will be given assuming that the computer system 100 and the computer system that trains the conversion model 250 are the same.

[0135] In step 810, the computer system 100 may acquire training data obtained by data-augmenting at least one first image including a texture corresponding to a target style and at least one second image that is an image to be converted into the target style and color-converted. The first image may be, for example, an image including content indicated by the target style. For example, if the content is a webtoon, the first image may be an image extracted from the webtoon and include a scene. The second image may be an image to be converted into the texture indicated by the first image. The second image may include a photograph or other image.

[0136] The first image and the second image may each include a plurality of images and may be processed and expanded into a form suitable for training the conversion model 250. The training data may refer to data that has been processed and expanded into a form suitable for training the conversion model 250.

[0137] For example, the conversion model 250 may be trained using first training data generated by data augmenting at least one first image containing a texture corresponding to a predetermined target style, and second training data generated by data augmenting at least one second image that is an image to be converted to this target style and color converted (using any target color information), and the above-mentioned pre-trained conversion model 250 may have been pre-trained using such first training data and second training data.

[0138] The method of generating the first training data and the second training data by data augmenting the first image and the second image will be described in more detail with reference to FIGS. 9 and 10.

[0139] In step 820, computer system 100 may train conversion model 250 to generate a texture-converted image in which the second image is converted into a texture corresponding to the target style and a color-converted image in which the second image is converted into a color corresponding to the target color information, using the acquired training data. The generated texture-converted image and color-converted image may be combined to generate a resultant image in which the second image is converted in accordance with the target style and the target color information.

[0140] The transformation model 250 trained by step 820 can transform an input image (input from the user) to represent a texture corresponding to the target style and a color corresponding to target color information (optionally input by the user) without the input of a reference image showing the target style.

[0141] The technical features explained above with reference to FIGS. 1 to 7, 11 and 13 can also be applied to FIG. 8, and therefore overlapping explanations will be omitted.

[0142] 9a and 9b are a flowchart illustrating a method for obtaining training data for a transformation model in one example.

[0143] 10a and 10b illustrate an example method for obtaining training data for a transformation model, with Fig. 10a illustrating the steps described with reference to Fig. 9.

[0144] As described above, the conversion model 250 may be trained using first training data generated by data augmenting at least one first image containing texture corresponding to a predetermined target style, and second training data generated by data augmenting at least one second image that is an image to be converted to this target style and color converted (using any target color information).

[0145] A method for generating the first training data will be described in more detail with reference to FIG. 9a.

[0146] The first training data must include training image(s) for generating a result image that reflects the degree of abstraction indicated by the abstraction level input to the conversion model 250. That is, the first training data includes training image(s) that replicate a predetermined degree of abstraction processing on the first image (i.e., processing that reflects the degree of abstraction indicated by an arbitrary abstraction level). Such first training data may be generated by a resizing process and / or a resolution change process on the first image. That is, the first training data may include an image that has been subjected to at least one of a resizing process and a resolution change process so as to replicate a predetermined degree of abstraction processing on the first image.

[0147] To generate such first training data, computer system 100 may perform a resizing process and a resolution change process on the first image to generate the first training data in step 910. Computer system 100 may perform the resizing process and the resolution change process on the first image taking into account any level of abstraction and various sizes and / or resolutions, thereby generating first training data including multiple images.

[0148] In step 920, the computer system 100 may convert the generated first training data into an image in the Lab color space. For example, the computer system 100 may convert the first training data into an image of the L component in the Lab color space. The converted image may become the final first training data.

[0149] Referring to FIG. 10a, a method for generating the final first training data will be described. At least one first image containing a texture corresponding to a predetermined target style is generated as I tgt RGB The first image I tgt RGB may perform a resizing and / or resolution change process so as to replicate the abstraction process of the degree of abstraction indicated by an arbitrary abstraction level α input by the user. The image generated by the resizing and / or resolution change process may be an image in the RGB color space, and such an image in the RGB color space may be converted into an image of the L component in the Lab color space. The first training data finally generated is I tgt L The first training data I tgt L may be input to the transformation model 250 described with reference to FIG. 7 and used to calculate a loss function for implementing the transformation model 250 to mimic the texture of the target style.

[0150] A method for generating the second training data will be described in more detail with reference to FIG. 9b.

[0151] The second training data may include a training transformed image obtained by color-transforming the second image to an arbitrary color. The second training data must include a training image(s) for generating a result image that reflects a color corresponding to the target color information input to the transformation model 250. That is, the second training data includes a training transformed image(s) that replicates an arbitrary color change to the second image (i.e., a process that reflects a color corresponding to the arbitrary target color information). The second training data including such a training transformed image may be generated by the transformation steps 930 to 980 described below.

[0152] In step 930, the computer system 100 may convert at least one of the second image and the color map of the second image generated by simplifying the second image into an image in the HSV color space. The image simplification may be performed based on, for example, a superpixel algorithm. The computer system 100 may convert both the second image and the color map of the second image into an image in the HSV color space. The HSV color space may be a color space expressed by specifying a specific color using coordinates of hue, saturation, and value.

[0153] In step 950, the computer system 100 may convert the converted HSV color space image into an arbitrary color. Here, "any color" may correspond to any target color information that the user can input. The arbitrary color may be an arbitrarily selected color. The color conversion in step 950 may generate the training converted image(s) described above.

[0154] In step 980, the computer system 100 may convert the training transformed image(s) generated in step 950 into an image in RGB color space.

[0155] Meanwhile, in one embodiment, second training data more suitable for training the conversion model 250 may be generated by correcting the brightness information L of the training converted image(s) generated in step 950. For example, in step 940, the computer system 100 may further convert at least one of the second image and the color map of the second image generated by simplifying the second image into an image in the Lab color space. The computer system 100 may convert both the second image and the color map of the second image into an image in the Lab color space. In step 945, the computer system 100 may extract L information (L component) from such a converted image in the Lab color space. Such extracted L information may be used, for example, to correct the L information of the color-converted image generated in step 950. Extracting such L information may be referred to as caching L information. That is, in step 960, the computer system 100 may convert the image in the HSV color space converted to an arbitrary color in step 950 into an image in the Lab color space. In step 970, the computer system 100 may replace the L information of the Lab color space image converted in step 960 with the L information extracted in step 945 (i.e., the cached L information). In step 980, the computer system 100 may convert the Lab color space image resulting from step 970 into an RGB color space image. This allows the training converted image(s) generated in step 950 to have the L information corrected to a more appropriate value.

[0156] In this regard, FIG. 10b shows the difference between an image in which the L information is corrected in steps 940-970 and an image in which it is not. For example, (c) shows an image in HSV color space, and (d) shows an image in HSV color space in which the L information is corrected using the above-described method. It can be seen that the image in (d) shows a more natural brightness. That is, the above-described method can suppress sudden changes in brightness, thereby eliminating the problem of loss of brightness information during image conversion to HSV color space. Therefore, it can be said that the training converted image in which the L information is corrected is more suitable for training the conversion model 250.

[0157] Therefore, through steps 930 to 980, the second training data suitable for training the conversion model 250 can be acquired.

[0158] Referring to FIG. 10a, a method for generating final second training data will be described. At least one second image to be converted into a target style and color converted into target color information is generated from I src RGB The second image I src RGB is the color map C src RGB The second image I src RGB and colormap C src RGB may be converted into training transformed images 1040 and 1050 by an HSV transformation-based data augmentation process 1030. The HSV transformation-based data augmentation process 1030 may be performed by the above-described steps 930 to 980. The training transformed image 1040 is src RGB may be augmented by any color change process. The training transformed image 1050 is a color map C src RGBmay be extended by any color change process. The training transformed images 1040 and 1050 may be images in the RGB color space. The training transformed image 1040 may be converted into an image of the ab component in the Lab color space. On the other hand, the training transformed image 1050 may be converted into an image in the Lab color space. The second training data that is finally generated is generated from the training transformed image 1040. (outside 26) Generated from JPEG0007765098000026.jpg1527 and training transformed image 1050 (outside 27) It may also be called JPEG0007765098000027.jpg1627. Second training data (outside 28) JPEG0007765098000028.jpg1627 may be input to the transformation model 250 described with reference to FIG. 7, and may be input to the color decoder 760 of the transformation model 250. (outside 29) JPEG0007765098000029.jpg1527 may be input to the conversion model 250 and may be used to calculate a loss function to implement the conversion model 250 so as to generate a resultant image that reflects the colors of the target color information.

[0159] Meanwhile, the second image I src RGB The second image I may also serve as training data for the transformation model 250. src RGB may be converted to an image in the Lab color space, and the converted image I src Lab may be input to the transformation model 250.

[0160] That is, the conversion model 250 converts the first learning data into an image of the L component of the Lab color space (i.e., the final first learning data) I tgt L , Image I, which is the second image converted to an image in Lab color spacesrc Lab , an image obtained by converting the second image to any color and then converting it to an image of the ab component of the Lab color space (Outside 30) JPEG0007765098000030.jpg1527 and the image obtained by converting the color map of the second image to an image in Lab color space (Outside 31) JPEG0007765098000031.jpg1627 can be used for learning. The term "image" used in the learning data and the embodiments is used to collectively mean data representing an image.

[0161] The technical features explained above with reference to FIGS. 1 to 8, 11, and 13 can also be applied to FIGS. 9 and 10, and therefore overlapping explanations will be omitted.

[0162] FIG. 12 illustrates the structure of an identifier network in one example.

[0163] The discriminator network 1200 may be a discriminator included in the conversion model 250. The conversion model 250 may be realized by adversarial learning using the discriminator network 1200.

[0164] The transformation model 250 may be trained using five loss functions.

[0165] The five loss functions are: L content vgg , L texture vgg , L color vgg , L texture adv , L color adv It may be called.

[0166] Of these, L texture adv , L color advmay denote the adversarial loss function of the classifier network 1200. The adversarial loss function may be used for the texture decoder 740 and color decoder 750 described above. Meanwhile, L content vgg , Ltexture vgg , L color vgg may be a loss function that is compared against the input image and / or the target image (an image with texture in the target style).

[0167] Also, L content vgg , L texture vgg , L texture adv is the loss function associated with the texture transformation, and L color vgg , L color adv may be a loss function associated with the color transformation.

[0168] The result generated by the texture decoder 740 may be passed through a VGG network, after which a restoration loss function for the image (e.g., the second image described above) may be calculated, and the feature map passed through the VGG network may have a Gram matrix calculated, after which a loss function with the Gram matrix corresponding to the image corresponding to the target style may be calculated.

[0169] The result generated by the color decoder 750 is an image that has passed through the VGG network and undergone a color data expansion process. (Outside 32) A loss function may be calculated for JPEG0007765098000032.jpg1527. Using this loss function, a result image can be generated that reflects colors corresponding to various target color information input by the user.

[0170] Below we explain each loss function in more detail.

[0171] First, the loss function of the texture decoder 740 will be described.

[0172] L texture adv may denote the adversarial loss for making the conversion model 250 mimic the texture corresponding to the target style (e.g., texture in the cartoon domain), and may be expressed as Equation (1).

[0173]

number

[0174] G may denote a network for texture transformation (i.e., a cartoonization network), which is a network for texture transformation with a given abstraction factor (abstraction level) α. (Outside 33) JPEG0007765098000034.jpg1127 may be a multi-abstraction discriminator.

[0175] L content vgg may represent the content loss required to enable a texture-transformed image (i.e., a cartoonized image) to preserve the semantic information of the original image, and may be expressed as Equation (2).

[0176]

number

[0177] The conversion model 250 may use the conv4_4 layer of a pre-trained VGG19 network to compute the Gram matrix.

[0178] In addition, the network for texture conversion (i.e., cartoonization network) may be configured to learn high-level texture surfaces using the Gram-based loss expressed in Equation (3).

[0179]

number

[0180] On the other hand, the final variation loss for imparting spatial smoothness to the output may be expressed as in Equation (4).

[0181]

number

[0182] Balancing Parameters (Outside 34) For JPEG0007765098000038.jpg1727, the final loss function of the text decoder 740 may be expressed as in Mathematical Formula (5).

[0183]

number

[0184] Next, the loss function of the color decoder 750 will be described.

[0185] Adversarial loss and perceptual color loss may be used. color adv is the same as that described in connection with the texture decoder 740, but with the color map (Outside 35) The only difference is that JPEG0007765098000040.jpg1326 is injected as condition information. The adversarial loss can be expressed as Equation (6).

[0186]

number

[0187] D may be a discriminator network for color transformation. G may denote a network for color transformation.

[0188] The perceptual loss (perceptual color loss) can be a plain VGG-based distance. The generated image and the Ground-Truth (GT) image are in the ab color space, so they must be converted back to the RGB color space before being input to the VGG network. The ab channel image is a brightness image extracted from the original image. src L The output from the color decoder 750 may be (Outside 36) It is often called JPEG0007765098000042.jpg2026, (Outside 37) JPEG0007765098000043.jpg2187. Perceptual color loss L color vgg may be calculated as in equation (7).

[0189]

number

[0190] The final loss function of the color decoder 750 may be expressed as equation (8).

[0191]

number

[0192] Lambda may denote a balancing parameter.

[0193] The overall framework of the transformation model 250 may be trained in an end-to-end manner using a final loss. The overall loss is (Outside 38) It can be JPEG0007765098000046.jpg1987.

[0194] The technical features explained above with reference to FIGS. 1 to 11 and 13 can also be applied to FIG. 12, and therefore overlapping explanations will be omitted.

[0195] The above-described devices may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also access, record, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, a single processing device may be described. However, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0196] Software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or may independently or collectively instruct the processing device. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0197] Methods according to embodiments may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record a computer-executable program or may temporarily record the program for execution or download. The medium may be various recording or storage means in the form of a single piece of hardware or multiple pieces of hardware combined together. It may be a medium directly connected to a computer system or distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to record program instructions, such as ROMs, RAMs, and flash memories. Other examples of media include recording media or storage media managed by app stores that distribute applications, websites that provide and distribute various software, and servers.

[0198] Although the embodiments have been described above based on limited examples and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the above description. For example, the described techniques may be performed in an order different from that described, and / or the described system, structure, device, circuit, or other element may be coupled or combined in a manner different from that described, or may be substituted or replaced by other elements or equivalents, and still achieve suitable results.

[0199] Therefore, different embodiments are within the scope of the appended claims, provided that they are equivalent to the claims. [Explanation of symbols]

[0200] 100: Computer Systems 110: Communications Department 120: Processor 130: Memory 140: Input / output interface

Claims

1. 1. A method of image conversion performed by a computer system, comprising: receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style using a pre-trained transformation model, and a color-transformed image by converting the input image into a color corresponding to the target color information; and combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; 1. A method of image transformation comprising: The image conversion method includes: generating a first transformed image from the input image that is input to the transformation model used to generate the texture transformed image; generating a second transformed image using a color map generated based on the input image and the target color information, the second transformed image being used to generate the color-transformed image; further comprising The generating step includes generating the texture-transformed image, in which a texture corresponding to the target style is reflected, based on the first transformed image using a texture decoder of the transformation model, and generating the color-transformed image, in which a color corresponding to the target color information is reflected, based on the second transformed image using a color decoder of the transformation model, to generate the texture-transformed image and the color-transformed image, respectively.

2. The transformation model is Without inputting a reference image showing the target style, The input image is pre-trained to convert the input image to represent a texture corresponding to the target style and a color corresponding to the input target color information.

2. The image transformation method of claim 1.

3. the target color information includes a palette including a group of a plurality of different colors determined based on the color distribution of the input image; 2. The image transformation method of claim 1.

4. Each of the plurality of colors is configured to be selectable by a user.

4. The image transformation method of claim 3.

5. the target style indicates content including images or videos; The texture corresponding to the target style is the texture of an image or video included in the content.

2. The image transformation method of claim 1.

6. The receiving step includes: receiving an abstraction level for setting the degree of abstraction shown in the resulting image; Including, The generating step includes: converting the input image into a texture corresponding to the target style that reflects the degree of abstraction indicated by the abstraction level to generate the texture-converted image; 2. The image transformation method of claim 1.

7. 1. A method of image conversion performed by a computer system, comprising: receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style using a pre-trained transformation model, and a color-transformed image by converting the input image into a color corresponding to the target color information; and combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; pre-processing the input image, converting the input image into a first converted image that is an image in Lab color space; generating a first color map by simplifying the input image; generating a second color map by transforming the colors of the first color map based on the target color information; and converting the second color map into a second converted image that is an image in Lab color space; 1. A method of image transformation comprising: the first transformed image and the second transformed image are input to the transformation model; The generating step includes: The image transformation method, wherein the transformation model generates the texture transformed image based on the first transformed image and generates the color transformed image based on the second transformed image.

8. The generating step includes: The transformation model is encoding the first transformed image using an encoder; generating the texture transformed image by decoding the encoded first transformed image using a texture decoder; and generating the color-transformed image by decoding the first transformed image, which has been encoded based on the second transformed image, using a color decoder; 8. The image transformation method of claim 7, comprising:

9. the texture transformed image is an image of the L component as an image in Lab color space, The color transformation image is an image of ab components as an image of Lab color space, The synthesizing step comprises: generating a composite image by combining the L component image and the ab component image; and converting the composite image to an image in RGB color space; The image transformation method of claim 1 , comprising:

10. 1. A method of image conversion performed by a computer system, comprising: receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style using a pre-trained transformation model, and a color-transformed image by converting the input image into a color corresponding to the target color information; and combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; 1. A method of image transformation comprising: The receiving step includes: receiving an abstraction level for setting a degree of abstraction of the resulting image; The generating step includes: generating the texture-converted image by converting the input image into a texture corresponding to the target style that reflects the degree of abstraction indicated by the abstraction level; The transformation model is an abstraction level reflecting module configured with a plurality of layers, each layer generating an output value reflecting an abstraction level indicated by the respective abstraction level; The generating step includes: An image conversion method, comprising: converting the input image into a texture corresponding to the target style that reflects the abstraction level indicated by the received abstraction level using an output value of the layer of the abstraction level reflection module corresponding to the received abstraction level, thereby generating the texture-converted image.

11. The transformation model is First training data generated by data augmenting at least one first image including a texture corresponding to the target style; and Second learning data generated by data augmenting at least one second image that is an image to be converted into the target style and color converted. It has been previously trained using 2. The image transformation method of claim 1.

12. 1. A method of image conversion performed by a computer system, comprising: receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style using a pre-trained transformation model, and a color-transformed image by converting the input image into a color corresponding to the target color information; and combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; 1. A method of image transformation comprising: The transformation model is First training data generated by data augmenting at least one first image including a texture corresponding to the target style; and Second learning data generated by data augmenting at least one second image that is an image to be converted into the target style and color converted. It has been pre-trained using The first learning data is A method of image transformation comprising an image that has undergone at least one of a resizing operation and a resolution change operation to replicate a predetermined degree of abstraction of the first image.

13. The second learning data is a training converted image obtained by converting the second image into an arbitrary color; 12. The image transformation method of claim 11.

14. The training transformed image is generated by a transformation step, The converting step comprises: converting at least one of the second image and a color map of the second image generated by simplifying the second image into an image in HSV color space; and converting the image in the HSV color space into an arbitrary color; 14. The image transformation method of claim 13, comprising:

15. 1. A method of image conversion performed by a computer system, comprising: receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style using a pre-trained transformation model, and a color-transformed image by converting the input image into a color corresponding to the target color information; and combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; 1. A method of image transformation comprising: The transformation model is First training data generated by data augmenting at least one first image including a texture corresponding to the target style; and Second learning data generated by data augmenting at least one second image that is an image to be converted into the target style and color converted. It has been pre-trained using The second learning data is a training converted image obtained by converting the second image into an arbitrary color; The training transformed image is generated by a transformation step, The converting step comprises: converting at least one of the second image and a color map of the second image generated by simplifying the second image into an image in HSV color space; and converting the image in the HSV color space into an arbitrary color; Including, The converting step comprises: converting at least one of the second image and the color map of the second image to an image in Lab color space; extracting L information from the image in Lab color space; converting the image in the HSV color space, which has been color-converted to a given color, into an image in the Lab color space; and replacing the L information of the converted Lab color space image with the extracted L information; An image transformation method, including:

16. The transformation model is an image obtained by converting the first learning data into an image of the L component of the Lab color space; an image obtained by converting the second image into an image in Lab color space; an image obtained by converting the second image into an arbitrary color and then converting the converted image into an image of ab components in the Lab color space; and an image obtained by converting the color map of the second image into an image in Lab color space; It has been previously trained using 14. The image transformation method of claim 13.

17. A program recorded on a computer-readable recording medium for causing the computer system to execute the method of claim 1.

18. 1. A computer system for executing an image transformation method, comprising: at least one processor implemented to execute instructions readable by said computer system; Including, The at least one processor receiving an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed; generating a texture-transformed image by converting the input image into a texture corresponding to the target style and a color-transformed image by converting the input image into a color corresponding to the target color information using a pre-trained transformation model; combining the texture-transformed image and the color-transformed image to generate a resultant image that transforms the input image according to the target style and the target color information; and the at least one processor further generates a first transformed image from the input image that is input to the transformation model used to generate the texture-transformed image; and generates a second transformed image that is used to generate the color-transformed image using a color map generated based on the input image and the target color information; the generating step includes generating the texture-transformed image, in which a texture corresponding to the target style is reflected, based on the first transformed image using a texture decoder of the transformation model, and generating the color-transformed image, in which a color corresponding to the target color information is reflected, based on the second transformed image using a color decoder of the transformation model, in order to generate the texture-transformed image and the color-transformed image, respectively. Computer system.

19. 1. A method for training a transformation model for transforming an image, executed by a computer system, comprising: acquiring training data by data-augmenting at least one first image including a texture corresponding to a target style and at least one second image that is an image to be converted into the target style and color converted; training the transformation model to generate a texture-transformed image by transforming the second image into a texture corresponding to the target style and a color-transformed image by transforming the second image into a color corresponding to target color information using the training data; Including, the texture-transformed image and the color-transformed image are combined to generate a resultant image of the second image transformed in accordance with the target style and the target color information; The learned transformation model transforms an input image to represent a texture corresponding to the target style and a color corresponding to input target color information without inputting a reference image indicating the target style; The learned transformation model further generates a first transformed image from the input image that is input to the transformation model and used to generate the texture-transformed image, and generates a second transformed image that is used to generate the color-transformed image using a color map generated based on the input image and the target color information; The trained transformation model includes generating the texture-transformed image, in which a texture corresponding to the target style is reflected, based on the first transformation image using a texture decoder of the transformation model, and generating the color-transformed image, in which a color corresponding to the target color information is reflected, based on the second transformation image using a color decoder of the transformation model, in order to generate the texture-transformed image and the color-transformed image, respectively. Image conversion method.

Images