Image processing device, image display device, image processing method, image display method and program

Abstract

To provide service or a device that can reuse information used in converting a style of an image.SOLUTION: An image processing device includes at least one memory, and at least one processor. The at least one processor executes generating a second image obtained by converting a style of a first image on the basis of a style vector and a conversion model. The style vector is selected from one or more style vectors according to an instruction of a user. The selected style vector is used for generating a fourth image obtained by converting a third image based on the conversion model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image display device, an image processing method, an image display method, and a program. [Background technology]

[0002] Image style conversion techniques have been proposed that convert the style of a live-action image into a painting style, an illustration style, an animation style, etc. For example, Patent Document 1 discloses an image style conversion device that converts an input image into an image having a style (e.g., a painting style) based on the style of a reference image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-132855 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide a service or device that can reuse information used when converting the style of an image. [Means for solving the problem]

[0005] An image processing device according to one aspect of the present disclosure includes at least one memory and at least one processor, wherein the at least one processor executes the steps of: generating a second image by converting the style of a first image based on a style vector and a transformation model; the style vector is a style vector selected from one or more style vectors in response to a user instruction; and the selected style vector is a style vector used to generate a fourth image by converting the style of a third image based on the transformation model. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 10 is a diagram illustrating an example of a style conversion process. [Figure 2] FIG. 10 is a diagram illustrating an example of style conversion of an image. [Figure 3] FIG. 2 illustrates an example of a functional configuration of an image processing apparatus. [Figure 4] FIG. 10 is a diagram illustrating an example of a home screen. [Figure 5] FIG. 10 is a diagram showing an example of a pickup screen. [Figure 6] FIG. 10 is a diagram illustrating an example of an image search screen. [Figure 7] FIG. 10 is a diagram showing an example of an image detail screen. [Figure 8] FIG. 10 is a diagram illustrating an example of a preset management screen. [Figure 9] FIG. 10 is a diagram illustrating an example of an image editing screen. [Figure 10] FIG. 10 is a diagram illustrating an example of a style conversion screen. [Figure 11] FIG. 10 is a diagram illustrating an example of a style conversion screen. [Figure 12] FIG. 10 is a diagram illustrating an example of the operation of a parameter adjustment area. [Figure 13] FIG. 10 is a diagram illustrating an example of a processing procedure of an image processing method. [Figure 14] FIG. 1 illustrates an example of a hardware configuration of an image processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0008] [Image processing device overview] One embodiment of the present disclosure is an image processing device that provides an image processing service that converts the style of an image. The image in this embodiment may be a real-life image or a computer graphics (CG) image. The style in this embodiment may be, for example, a painting style, an illustration style, an animation style, or a combination of these, and the combination may be continuously adjustable.

[0009] For example, the image processing service of this embodiment can convert an image into a style that combines a typical oil painting style and a typical watercolor painting style, and the degree of influence of each style can be adjusted as desired (for example, about 80% watercolor painting style and about 20% oil painting style).

[0010] Style conversion uses parameters that characterize the style of an image (hereinafter also referred to as a "style vector"), such as color, contrast, texture, or paint type.

[0011] The style vector is, for example, a 32-dimensional real vector. However, in the style conversion, it is not necessary to use all components of the style vector, and only components that significantly characterize the style may be used. Each component of the style vector does not need to be associated with a feature that is easy for humans to understand. In this case, style conversion is repeated while adjusting each component of the style vector by trial and error. Note that, in the image processing service of this embodiment, for example, only 20 dimensions out of the 32 dimensions are used.

[0012] In the image processing service of this embodiment, when the style of an image is converted, the style-converted image and the style vector used for the style conversion are stored in association with each other. As a method for associating the style-converted image and the style vector, for example, the style-converted image and the style vector may be stored as a set, information that allows one piece of information to be acquired from the other piece of information may be stored, or one piece of information and identification information for the other piece of information may be stored as a set. In this way, the image processing service of this embodiment can convert the style of an image using the style vector used for style conversion of another image.

[0013] The image processing service of this embodiment also has a function of presenting to the user the style vector used in the style conversion of a certain image. The user may set the value of the style vector to be used by themselves while referring to the presented style vector, or may copy the entire presented style vector as their own style vector. In this way, the image processing service of this embodiment allows the style conversion of a desired image to be performed using the style vector used in the style conversion of another image.

[0014] <Model training and style conversion> Model learning and style conversion in this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an example of style conversion processing in this embodiment.

[0015] The style vector in this embodiment is generated by a trained extraction model 20. As shown in FIG.

[0016] The feature extractor 21 is trained to classify images using a plurality of training images 10. The feature extractor 21 extracts image features 23 from the training images 10. The structure of the feature extractor 21 may be a neural network (for example, a convolutional neural network (CNN)). For example, the image features 23 are a 512-dimensional feature vector.

[0017] The dimension reducer 22 is trained to reduce the dimension of the image features 23. The dimension reducer 22 reduces the dimension of the image features 23 to generate a style vector 30. The dimension reducer 22 may have a neural network structure (for example, an encoder of a variational auto-encoder (VAE)). For example, the style vector 30 is a 32-dimensional feature vector. Note that the style vector may be extracted by any method as long as it reflects information that characterizes the style of the image.

[0018] The style conversion in this embodiment is performed by a trained conversion model 50. As shown in FIG. 1, the conversion model 50 in this embodiment includes, for example, an encoder 51, a modulation layer 52, and a decoder 53.

[0019] The encoder 51 generates a first intermediate representation based on the input image 40. The encoder 51 may be a neural network. The modulation layer 52 generates a second intermediate representation based on the first intermediate representation and the style vector 30. The modulation layer 52 may be a neural network. The decoder 53 generates a transformed image 60, in which the style of the input image 40 has been transformed, based on the second intermediate representation. The decoder 53 may be a neural network.

[0020] The modulation layer 52 generates the second intermediate representation by, for example, adaptive instance normalization (AdaIn). Adaptive instance normalization is a process that performs instance normalization and affine transformation. The modulation layer 52 determines transformation parameters of the affine transformation based on the style vector 30.

[0021] The conversion model 50 is trained as a generator in a conditional generative adversarial network (cGAN). Specifically, the conversion model 50 is input to a classifier that distinguishes between an image style-converted using the style vector 30 and a true image of a style corresponding to the style vector 30, and based on the classification result by the classifier, the conversion model 50 is trained to generate an image that the classifier will misclassify. Note that the conversion model is not limited to the configuration shown in FIG. 1. The image processing device in this embodiment can use any conversion model that can generate an image obtained by converting the original image into a style represented by the style vector, based on the original image and style vector.

[0022] Fig. 2 is a diagram showing an example of style conversion of an image. Fig. 2(A) is a real-life image that is the source of conversion, and Fig. 2(B) is an image obtained by performing style conversion on the real-life image shown in Fig. 2(A). As shown in Fig. 2, the style conversion in this embodiment makes it possible to generate an image in which the style of the real-life image has been converted.

[0023] [Functional configuration of image processing device] A functional configuration of an image processing device according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of an image processing device according to this embodiment.

[0024] As shown in FIG. 3, the image processing device 100 in this embodiment includes an image conversion unit 101, an image extraction unit 102, a style adjustment unit 103, a point management unit 104, a model storage unit 110, an image information storage unit 111, a style storage unit 112, and a user information storage unit 113.

[0025] <Model Memory> The model storage unit 110 stores trained conversion models. The conversion models are machine learning models that perform predetermined image conversion processing. The conversion model in this embodiment is a style conversion model that receives an image and a style vector as input, converts the image into a style represented by the style vector, and outputs the converted image. The conversion model may be a neural network.

[0026] The model storage unit 110 may store multiple conversion models. The multiple conversion models may be prepared according to the styles that can be realized by the conversion. For example, a first conversion model may realize an animation or illustration style, a second conversion model may realize a Western painting style, and a third conversion model may realize all the styles that can be realized by the other conversion models. Furthermore, the multiple conversion models may be prepared according to the number of dimensions or format of the style vector. For example, the first conversion model may use a style vector that includes a component that indicates hue, and the second conversion model may use a style vector that includes a binary component that specifies monochrome.

[0027] In the image processing device according to this embodiment, the style vectors available for a certain transformation model may be limited. In this case, a style vector corresponding to a certain transformation model may not be available for other transformation models. The model storage unit 110 may store identification information for each transformation model.

[0028] <Image information storage unit> The image information storage unit 111 stores image information related to images generated by the image processing device 100. The image information includes an image before style conversion, an image after style conversion, metadata assigned to the image, identification information for identifying a style vector used when generating the image, and identification information for identifying a user who generated the image.

[0029] The metadata assigned to an image may include, for example, a title, a description, and tags. The metadata may be arbitrarily set by the user who generated the image. Alternatively, the image processing device 100 may automatically set the metadata based on the image.

[0030] The identification information of a style vector is, for example, a name set in the style vector, an identifier (ID) assigned when the style vector is generated, etc. The identification information of a user is, for example, a user name set by the user, an ID assigned when the user starts using the image processing service, etc.

[0031] <Style Memory> The style storage unit 112 stores style information related to a style vector. The style information includes the style vector, a thumbnail (reduced image) representing the style vector, identification information identifying a conversion model that can use the style vector, identification information identifying a user who generated the style vector, and the like. The style storage unit 112 may store information linking a style vector to a conversion model that can use the style vector. As an example, the style information may store identification information for a conversion model that can be used, link information to the conversion model that can be used, and the like. Furthermore, a style vector and a conversion model that can use the style vector may be stored as a set in the style storage unit 112.

[0032] A thumbnail of a style vector is a reduced image obtained by style-converting a sample image using the style vector. The identification information of the converted model includes, for example, the name set for the discriminative model, the version number of the discriminative model, and the ID assigned when the discriminative model was generated.

[0033] Hereinafter, the style vectors stored in the style storage unit 112 are also referred to as "presets." Presets include style vectors that are pre-stored by an administrator of the image processing service, and style vectors that are generated by a user adjusting an existing style vector. Pre-stored presets are also referred to as "system presets," and presets generated by a user are also referred to as "user presets."

[0034] <User information storage unit> User information relating to users of the image processing service is stored in the user information storage unit 113. The user information includes identification information for identifying the user, contract information indicating the fee type to which the user has subscribed, favorite information saved by the user, and the like.

[0035] The fee structure in this embodiment includes a free plan and a paid plan. The paid plan may include a plurality of plans with different usage fees. The paid plan in this embodiment includes a standard plan with a low usage fee and a premium plan with a high usage fee.

[0036] The favorite information in this embodiment includes information related to style vectors saved by the user, other users followed by the user, and images created by other users and saved by the user.

[0037] The available functions may be restricted based on the fee type to which the user has subscribed. In this embodiment, adjustment of style vectors and saving of style vectors to favorite information are restricted to users who have subscribed to a paid plan. However, the function restrictions based on the fee type are not limited to this and can be applied to any function.

[0038] <Image conversion section> The image conversion unit 101 converts the style of an input image using a conversion model stored in the model storage unit 110. Specifically, the image conversion unit 101 first acquires a style vector in accordance with a user instruction. The style vector acquired by the image conversion unit 101 may be a style vector (i.e., a preset) stored in the style storage unit 112, or may be a style vector output from the style adjustment unit 103.

[0039] Next, the image conversion unit 101 reads out the conversion model specified by the user from the model storage unit 110. Subsequently, the image conversion unit 101 inputs the input image and the style vector into the conversion model, thereby generating an image in which the style of the input image has been converted (hereinafter also referred to as a "converted image").

[0040] After generating the converted image, the image conversion unit 101 stores image information related to the converted image in the image information storage unit 111. When the style vector is a style vector output from the style adjustment unit 103, the image conversion unit 101 stores style information related to the style vector in the style storage unit 112.

[0041] <Image extraction section> The image extraction unit 102 extracts image information stored in the image information storage unit 111. The image extraction unit 102 extracts image information based on, for example, the user who generated the image, the model used when generating the image, a style vector associated with the image, etc. The image extraction unit 102 may extract image information based on search conditions specified by the user.

[0042] In user-based image extraction, for example, image information about images generated by other users followed by the user is extracted. In model-based image extraction, for example, image information about images generated using a transformation model selected by the user is extracted. In style vector-based image extraction, for example, image information about images generated using a style vector that is the same as or similar to the style vector selected by the user is extracted.

[0043] <Style Adjustment Section> The style adjustment unit 103 adjusts each component (hereinafter also referred to as "parameter") of the style vector in response to a user instruction. When adjusting the style vector, the style adjustment unit 103 creates a thumbnail (reduced image) to be displayed on a screen for user operation.

[0044] The reduced image is generated by converting a portion of the image to be style converted based on the current style vector. Two reduced images are generated for each parameter of the style vector. The first reduced image is generated based on a style vector in which the parameter is set to a first value (for example, the minimum value) while the other parameters are fixed. The second reduced image is generated based on a style vector in which the parameter is set to a second value (for example, the maximum value) while the other parameters are fixed. Note that the change range of the parameter is just an example, and it is sufficient that the second value is greater than the first value.

[0045] Therefore, the style adjustment unit 103 generates a maximum of 40 reduced images (20 dimensions x 2 images) in one screen display. However, if there are reduced images that have already been generated, the style adjustment unit 103 may reuse those reduced images to reduce the number of reduced images to be generated.

[0046] <Points Management Department> The point management unit 104 manages points held by the user. In this embodiment, points are consumed every time an image style transformation and a style vector adjustment are performed. Points may be given to the user over time. Points may also be purchased through a predetermined payment method.

[0047] The points given to a user may vary depending on the user's fee structure. For example, a user of a free plan may be given points that allow two to three style changes and ten to twenty style adjustments per day. A user of a standard plan may be given points that allow 10 to 30 style changes and 100 to 200 style adjustments per day. A user of a premium plan may be given points that allow 100 or more style changes per day. Note that a premium plan user may be configured to be able to make unlimited style adjustments.

[0048] [Image processing device user interface] A user interface of an image processing device according to an embodiment of the present disclosure will be described with reference to Fig. 3 to Fig. 13. The user interface may be realized, for example, as an operation screen displayed on a display device by a program installed in the image processing device 100.

[0049] <Home screen> 4 is a diagram showing an example of a home screen in this embodiment. The home screen is the screen that is first displayed after a user logs in to the image processing service. The home screen is the starting point for the user to start each function of the image processing service.

[0050] As shown in FIG. 4, the home screen 1000 has an image input field 1010 and an image display area 1020.

[0051] An image to be processed is input into the image input field 1010. When the user drags and drops an image file into the image input field 1010, the image represented in the image file is input.

[0052] Input into image input field 1010 may be any operation that can identify an image. For example, clicking image input field 1010 may display a file selection screen, and image input may be accepted by selecting an image file on the file selection screen. Alternatively, for example, image input may be accepted by entering an absolute path or URL (Uniform Resource Locator) to the image file as text into image input field 1010.

[0053] Images that the user has previously styled, images that the user has published using users that the user follows, and the like are randomly displayed in the image display area 1020. The image title and user name may be displayed on the image displayed in the image display area 1020. The image title and user name may be displayed as a pop-up when the mouse is hovered over the image.

[0054] When the user selects any image in the image display area 1020, an image detail screen for that image is displayed.

[0055] <Pickup Screen> 5 is a diagram showing an example of a pick-up screen in this embodiment. The pick-up screen is a screen for viewing images published in the image processing service. The pick-up screen can be opened from the main menu displayed in common on each screen.

[0056] As shown in FIG. 5, a pickup screen 1100 includes a category selection field 1110 and an image display area 1120 .

[0057] The category selection field 1110 displays tabs indicating the category of the image to be viewed. In this embodiment, tabs corresponding to featured images (pickup), long-term popular images (trend), short-term popular images (upcoming), and latest posted images (recent) are displayed. Featured images are images selected by an administrator of the image processing service, etc. Long-term popular images are images that have remained popular over a long period of time. Short-term popular images are images that have become increasingly popular in a short period of time. Latest posted images are images that have been published within a predetermined period of time.

[0058] When the user presses a tab in the category selection field 1110, images corresponding to the category corresponding to the tab are displayed in the image display area 1120. As with the home screen 1000, the image displayed in the image display area 1120 may display the title of the image and the user name.

[0059] When the user selects any image in the image display area 1120, an image detail screen for that image is displayed.

[0060] <Image search screen> 6 is a diagram showing an example of an image search screen in this embodiment. The image search screen is a screen for searching for images published in an image processing service. The image search screen can be opened from the main menu displayed in common on each screen.

[0061] As shown in FIG. 6, an image search screen 1200 has a search condition input field 1210 and a search result display area 1220 .

[0062] When a user inputs search criteria into a search criteria input field 1210, images that match the search criteria are displayed in a search result display area 1220. The search targets titles, descriptions, tags, etc., that are assigned to the images. As with the home screen 1000, the images displayed in the search result display area 1220 may also display the titles and user names of the images.

[0063] When the user selects any image in the search result display area 1220, an image details screen for that image is displayed.

[0064] <<Image details screen>> 7 is a diagram showing an example of an image detail screen in this embodiment. The image detail screen is a screen for referring to detailed information about an image. The image detail screen is launched when the user selects any image displayed on the home screen 1000, the pick-up screen 1100, the image search screen 1200, or the like.

[0065] 7, the image detail screen 1300 has an image display area 1310, an image information display area 1320, a style information display area 1330, and a related image display area 1340. The style information display area 1330 has an add button 1331.

[0066] An image selected by the user is displayed in the image display area 1310. The image display area 1310 may be divided into two areas, and the image before style conversion and the image after style conversion may be displayed in a manner that allows comparison. In this embodiment, the image display area 1310 is divided into two areas, left and right, based on the position of the pointer, and the image before style conversion is displayed in the left area, and the image after style conversion is displayed in the right area. In this case, when the user moves the position of the pointer, the boundary line of the areas moves accordingly, and the display ratio of the image before style conversion and the image after style conversion changes. Note that the image display area 1310 may be divided into the pre-conversion and post-conversion areas in any manner based on the position of the pointer, such as by dividing into two areas, top and bottom.

[0067] Detailed information about the image is displayed in the image information display area 1320. The information displayed in the image information display area 1320 includes, for example, the user who created the image, attribute information of the image (such as Exif (Exchangeable image file format) information), and metadata (title, description, and tags) assigned to the image. The title, description, and tags are displayed in an editable format.

[0068] Information about the style vector used to convert the style of the image is displayed in style information display area 1330. The information displayed in style information display area 1330 includes, for example, the name of the style vector, a reduced image of the style vector, the name of the conversion model corresponding to the style vector, and an image visualizing each parameter of the style vector.

[0069] When the user presses the add button 1331, the style vector displayed in the style information display area 1330 is saved in the user's favorite information.

[0070] Images related to the image in question are randomly displayed in the related image display area 1340. Images displayed in the related image display area 1340 are images that have a user, tag, or style similar to that of the image in question. The image displayed in the related image display area 1340 may display the title and user name of the image, similar to the home screen 1000. The title and user name of the image may be displayed as a pop-up when the mouse is hovered over the image.

[0071] When the user selects any image in the related image display area 1340, the information displayed on the image detail screen 1300 is updated to information about that image.

[0072] <Preset Management Screen> 8 is a diagram showing an example of a preset management screen in this embodiment. The preset management screen is a screen for managing presets saved by the user as favorite information. The preset management screen can be opened from the main menu displayed in common on each screen.

[0073] 8, the preset management screen 1400 has one or more preset display areas 1410 (1410-1 to 1410-2). The preset display area 1410 has a thumbnail image display field 1411, a name display field 1412, a parameter display field 1413, a user display field 1414, and a model display field 1415.

[0074] The same number of presets as the user has saved in the favorite information are displayed in the preset display area 1410. If a large number of presets are saved in the favorite information, the preset display area 1410 may display only as many presets as can be displayed on the display device, and additional preset display areas 1410 may be added as the user scrolls the preset management screen 1400 downward.

[0075] The preset display area 1410 may be displayed in any order on the preset management screen 1400. For example, the order of the preset display area 1410 may be determined based on the date and time when the presets were saved in the favorite information, the order in which the presets are sorted based on their names, the number of times they have been used for style conversion, or the like.

[0076] The thumbnail image display field 1411 displays a thumbnail image of the preset. The thumbnail image display field 1411 is divided into two areas, similar to the image display area 1310 of the image detail screen 1300, with one area displaying a thumbnail image before style conversion and the other area displaying a thumbnail image after style conversion. As an example, the thumbnail image display field 1411 in this embodiment is divided into two areas, left and right, by a diagonal line passing through the center, with the left area displaying a thumbnail image before style conversion and the right area displaying a thumbnail image after style conversion. Note that the pre-conversion area and the post-conversion area may be divided in any direction, such as vertically.

[0077] The name display field 1412 displays the name of the preset. The parameter display field 1413 displays an image that visualizes each parameter of the preset. In the image displayed in the parameter display field 1413, each parameter may be normalized within a predetermined range. For example, the predetermined range may be -2 to +2, or may be a range of -100 to +100. The user display field 1414 displays the name of the user who created the preset. The model display field 1415 displays the name of the conversion model corresponding to the preset.

[0078] <Image editing screen> 9 is a diagram showing an example of an image editing screen in this embodiment. The image editing screen is a screen for editing an image. The image editing screen is displayed after an image input is accepted on the home screen 1000. The image editing screen may be displayed when the user selects one of the images from a screen that displays a list of images input by the user.

[0079] As shown in FIG. 9, an image editing screen 1500 has an image display area 1510, an editing history display area 1520, a publish button 1530, and a style conversion button 1540.

[0080] The image to be edited is displayed in image display area 1510. The editing history display area 1520 displays the editing history of the image in chronological order. When the user selects one of the editing histories from editing history display area 1520, the information displayed on image editing screen 1500 is updated to information about the image at the time of the selected editing history. Note that when editing is performed on an image selected from editing history display area 1520, the editing history after the selected image is deleted.

[0081] When the user presses the publish button 1530, the image is made public. The published image can be referenced by other users of the image processing service. When the user presses the style conversion button 1540, a style conversion screen for the image is displayed.

[0082] <Style conversion screen> Fig. 10 is a diagram showing an example of a style conversion screen in this embodiment. The style conversion screen is a screen for converting the style of an image. Fig. 10 shows an example of the style conversion screen when converting a style using a preset.

[0083] As shown in FIG. 10, a style conversion screen 1600 when a preset is used has an image display area 1610, a preset display area 1620, a model selection field 1630, a preview display area 1640, and an apply button 1650.

[0084] The image to be converted is displayed in image display area 1610. Image display area 1610 includes a preview reference area 1611. Preview reference area 1611 is an area for acquiring a reduced image to be displayed in preview display area 1640. At the time of initial display, preview reference area 1611 is located in the center of image display area 1610. However, preview reference area 1611 can be moved to any position in image display area 1610 by a user operation.

[0085] A list of available presets is displayed in the preset display area 1620. The available presets are user presets saved in favorite information by the user and system presets. If there are multiple available conversion models, only the presets corresponding to the conversion model selected in the model selection field 1630 are displayed in the preset display area 1620. Note that the preset corresponding to the conversion model may be extracted based on information linking the conversion model and style vector stored in the style storage unit 112.

[0086] The model selection field 1630 displays a list of available conversion models in a selectable manner. The model selection field 1630 can display the list of conversion models in a drop-down list, for example, and allow the user to select a conversion model through operation. When there is only one available conversion model, the model selection field 1630 may be fixed or may be hidden.

[0087] When the user selects a conversion model in the model selection field 1630, the presets displayed in the preset display area 1620 are updated to the presets corresponding to the selected conversion model.

[0088] When the user selects a preset in preset display area 1620, the image to be converted is converted to a low resolution and the style converted using that preset is displayed in image display area 1610. At the same time, a reduced image before style conversion using that preset and a reduced image after style conversion using that preset are displayed in preview display area 1640. The reduced image displayed in preview display area 1640 is an image included in the area indicated by preview reference area 1611.

[0089] The conversion model in this embodiment may input an image of the same size as the image to be converted, and output a style-converted image of the same size as the input image. For example, if the image to be converted is an image of 1024 x 1024 pixels, the 1024 x 1024 pixel image is input to the conversion model, and a style-converted image of 1024 x 1024 pixels is output. Therefore, by converting the image to be converted to a low resolution, it is possible to reduce the amount of calculation during style conversion.

[0090] The process of converting the image to be converted to a low resolution may be performed in advance. As an example, when the user inputs the image to be converted on the home screen 1000, the process of converting the image to a low resolution may be performed. The image converted to a low resolution may be stored in the image information storage unit 111 or the like together with the image to be converted. By preparing the low resolution image in advance, the process of converting the image to a low resolution can be omitted when displaying a preview.

[0091] When the user presses the apply button 1650, the style of the entire image to be converted, displayed in the image display area 1610, is converted using the preset selected in the preset display area 1620. When the style conversion is complete, image information related to the converted image is stored in the image information storage unit 111.

[0092] Fig. 11 is a diagram showing an example of a style conversion screen in this embodiment, which shows the style conversion screen when converting a style by adjusting preset parameters.

[0093] As shown in FIG. 11, a style conversion screen 1700 for adjusting a style differs from the style conversion screen 1600 shown in FIG. 10 in that it has a plurality of parameter adjustment areas 1710 (1710-1 to 1710-6) instead of the preset display area 1620.

[0094] The number of parameter adjustment areas 1710 displayed is the same as the number of parameters used in style conversion among the parameters of the style vector. That is, as an example, up to 32 parameter adjustment areas 1710 can be displayed. In this embodiment, 20 dimensions of the style vector are used in style conversion, so 20 parameter adjustment areas 1710 are displayed.

[0095] The parameter adjustment area 1710 displays, in an adjustable form, each parameter of the preset selected in the preset display area 1620 of the style conversion screen 1600. The parameter adjustment area 1710 displays, for each parameter of the preset, two reduced images corresponding to that parameter. The first reduced image is a reduced image that has been style-converted using a style vector in which the value of that parameter has been changed to a first value (for example, the minimum value) while keeping the other parameters fixed. The second reduced image is a reduced image that has been style-converted using a style vector in which the value of that parameter has been changed to a second value (for example, the maximum value) while keeping the other parameters fixed.

[0096] Fig. 12 is a diagram showing an example of the operation of the parameter adjustment area, Fig. 12(A) is a diagram showing the parameter adjustment area 1710 before the parameter adjustment, and Fig. 12(B) is a diagram showing the parameter adjustment area 1710 after the parameter adjustment.

[0097] As shown in FIG. 12, a parameter adjustment area 1710-n (n=1, . . . , 6) has a parameter change field 1711-n, a first thumbnail image display field 1712-n, and a second thumbnail image display field 1713-n.

[0098] The parameter change field 1711-n displays the current value of the style vector in a changeable manner. The style vector value displayed in the parameter change field 1711-n may be normalized within a predetermined numerical range. As an example, the value may be normalized within a range from -2 to +2. The parameter change field 1711-n in this embodiment is configured as a slider bar arranged horizontally and continuously adjustable from the left end indicating the minimum value (-2 in this embodiment) to the right end indicating the maximum value (+2 in this embodiment).

[0099] The parameter change field 1711-n may be configured as a slider bar that can be adjusted in a predetermined number of steps, a text box in which a numerical value can be directly input, a text box in which a numerical value can be changed using a plus button and a minus button, a list box in which a numerical value can be selected such as a select box or a radio button, a dial in which a numerical value increases or decreases depending on the direction of rotation, etc. In other words, any configuration can be used as long as it is possible to visually confirm the relationship between the current value and the value after change for each parameter of the style vector.

[0100] The first reduced image display field 1712-n displays a reduced image obtained by style-converting the image in the area indicated in the preview reference area 1611 using a style vector in which the parameter in question is set to its minimum value while keeping other parameters fixed. The second reduced image display field 1713-n displays a reduced image obtained by style-converting the image in the area indicated in the preview reference area 1611 using a style vector in which the parameter in question is set to its maximum value while keeping other parameters fixed.

[0101] The first thumbnail display field 1712-n and the second thumbnail display field 1713-n show the effect on the style of the parameter corresponding to the parameter adjustment area 1710-n. Therefore, by comparing the first thumbnail display field 1712-n and the second thumbnail display field 1713-n, it is possible to understand the characteristics of the style that will be affected by adjusting the parameter.

[0102] As shown in FIG. 12, when one of the parameter change fields 1711 (parameter change field 1711-1 in the example of FIG. 12) is adjusted, the first thumbnail image display field 1712 and the second thumbnail image display field 1713 corresponding to the other parameter change fields 1711 (parameter change fields 1711-2 to 1711-6 in the example of FIG. 12) are updated based on the adjusted parameters. In addition, the image to be converted is converted to a low resolution, and the image that has been style-converted based on the adjusted parameters is displayed in the image display area 1610. The user can adjust the parameters while comparing the thumbnail image displayed in the preview display area 1640 with the thumbnail images displayed in each parameter adjustment area 1710, thereby efficiently searching for a style vector that is close to the desired style.

[0103] [Image processing procedure] The processing procedure of the image processing method executed by the image processing device 100 according to an embodiment of the present disclosure will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the processing procedure of the image processing method.

[0104] In step S1, the image extraction unit 102 of the image processing device 100 extracts image information stored in the image information storage unit 111. The image information extracted by the image extraction unit 102 varies depending on the screen displayed on the display device. For example, when the home screen 1000 is displayed on the display device, the image extraction unit 102 extracts image information related to images generated by the user and users followed by the user. Furthermore, for example, when the image search screen 1200 is displayed on the display device, the image extraction unit 102 extracts image information that matches search criteria specified by the user.

[0105] In step S2, the image extraction unit 102 of the image processing device 100 saves the style vector in favorite information in response to a user instruction. The image extraction unit 102 first selects an image in response to the user instruction. Next, the image extraction unit 102 outputs image information related to the selected image. The output image information includes at least the image and the style vector used to generate the image.

[0106] Next, the image extraction unit 102 selects a style vector in response to a user instruction. Next, the image extraction unit 102 stores identification information of the selected style vector in the user information storage unit 113. The identification information of the style vector is saved in favorite information of the user information related to the user.

[0107] In this embodiment, the style vector can be saved only if the user has subscribed to a paid plan. Therefore, the image extraction unit 102 determines whether the user's fee plan is a paid plan or a free plan based on the contract information stored in the user information storage unit 113. If the user has subscribed to the free plan, step S2 is not executed.

[0108] In step S3, the image conversion unit 101 of the image processing device 100 receives an input of an image in response to an instruction from the user. The image conversion unit 101 stores the received input image in the image information storage unit 111.

[0109] In step S4, the image conversion unit 101 of the image processing device 100 selects a conversion model in response to a user instruction. If there is only one conversion model stored in the model storage unit 110, the selection of the conversion model is omitted. Next, the image conversion unit 101 acquires, from the user information storage unit 113, identification information of the style vector stored in the user's favorite information.

[0110] Based on the acquired identification information, the image conversion unit 101 reads out the user presets stored in the style storage unit 112. At the same time, the image conversion unit 101 reads out the system presets stored in the style storage unit 112. Note that when a conversion model is selected, the image conversion unit 101 reads out only the user presets and system presets corresponding to the selected conversion model.

[0111] Then, the image conversion unit 101 selects a preset to be used for style conversion from among the system presets or the user presets in response to an instruction from the user.

[0112] In step S5, the style adjustment unit 103 of the image processing device 100 adjusts each parameter of the preset in accordance with an instruction from the user. Every time the style adjustment unit 103 adjusts each parameter of the preset, it generates a reduced image corresponding to each parameter using the selected conversion model.

[0113] When the reduced image is generated, the point management unit 104 subtracts a predetermined number of points from the points stored in the user information storage unit 113 .

[0114] Note that the user can arbitrarily choose whether or not to adjust the preset. If the user wants to use the preset selected by the user as is for style conversion, step S5 does not need to be executed.

[0115] In this embodiment, preset adjustment can be performed only if the user has subscribed to a paid plan. Therefore, the style adjustment unit 103 determines whether the user's fee plan is a paid plan or a free plan based on the contract information stored in the user information storage unit 113. If the user has subscribed to the free plan, step S5 is not performed.

[0116] In step S6, the image conversion unit 101 of the image processing device 100 acquires a style vector. If step S5 is executed, the image conversion unit 101 acquires the style vector output by the style adjustment unit 103. If step S5 is not executed, the image conversion unit 101 acquires a preset selected in response to a user instruction.

[0117] The image conversion unit 101 reads out a conversion model selected by a user from the model storage unit 110. Next, the image conversion unit 101 inputs the input image and the style vector into the conversion model to generate a converted image in which the style of the input image has been converted. The image conversion unit 101 displays the converted image on a display device.

[0118] When the converted image is generated, the point management unit 104 subtracts a predetermined number of points from the points stored in the user information storage unit 113 .

[0119] In step S7, the image conversion unit 101 of the image processing device 100, in response to a user instruction, stores image information related to the converted image in the image information storage unit 111. When the image conversion unit 101 generates a converted image using a style vector adjusted by the style adjustment unit 103, the image conversion unit 101 stores style information related to the style vector in the style storage unit 112.

[0120] [supplement] In the above embodiment, only users who have subscribed to a paid plan can save and adjust style vectors. Users who have subscribed to a free plan may be provided with a function that allows them to randomly change style vectors. This function allows users to repeatedly change style vectors randomly, potentially resulting in a style vector that can be converted into a desired style.

[0121] Image processing device 100 may display style vectors and information associated with the style vectors on a display device, and select a desired style vector based on a user instruction. The information displayed by image processing device 100 on a display device may be either the style vector or information associated with the style vector, or may be both. One piece of information or multiple pieces of information may be associated with one style vector.

[0122] When displaying style vectors and information associated with the style vectors on a display device, the image processing device 100 may display at least a plurality of style vectors or a plurality of pieces of information associated with the style vectors on one screen at a time. The image processing device 100 may display at least each style vector or each piece of information associated with the style vector on a different screen. The image processing device 100 may divide at least a plurality of style vectors or a plurality of pieces of information associated with the style vectors into a predetermined number and display them on different screens.

[0123] An example of information associated with a style vector is one or more images style-transformed using the style vector. Note that the one or more images may be one image or multiple images. However, the information associated with a style vector is not limited to images style-transformed using the style vector.

[0124] The information associated with the style vector may be, for example, information such as the creator, creation date, user, and use date of the style vector, or may be descriptive information of the style vector (e.g., "converted to ~ style"). The descriptive information may be defined by the creator or user of the style vector.

[0125] For example, the information associated with the style vector may be an image that has been style-converted using the style vector, as well as an image that has undergone a predetermined process, such as changing the resolution or size.

[0126] The selection of a style vector may be performed based on a user's designation of the style vector, or based on a user's designation of information associated with the style vector.

[0127] In the above embodiment, whether or not the style vector adjustment and style vector storage are available is determined based on contract information indicating the fee type to which the user has subscribed. Whether or not the style vector adjustment and style vector storage are available may also be determined based on user attribute information. User attribute information may include, for example, the user's contract information, the organization to which the user belongs, information that distinguishes the user, information that identifies the user's usage status, etc. The user's usage status may include, for example, the number of points owned by the user, the user's usage history, etc.

[0128] In the above embodiment, the user selects a style vector to be used for style conversion by specifying one style vector from multiple style vectors. Alternatively, the user may be able to specify one style vector from one or more style vectors. The one or more style vectors may be one style vector or multiple style vectors.

[0129] In the above embodiment, the user can select a conversion model to be used for style conversion by specifying one conversion model from a plurality of conversion models. The user may also be able to specify one conversion model from one or more conversion models. The one or more conversion models may be one conversion model or multiple conversion models.

[0130] The image processing device 100 in this embodiment may be configured with at least one memory and at least one processor. In this case, the at least one processor can control the storage of various data in the at least one memory and the acquisition of various data from the at least one memory. The at least one processor may also control the screen displayed on the display device.

[0131] The image processing device 100 in this embodiment may be realized by at least one server and at least one user terminal (hereinafter also referred to as an "image display device"). The user terminal may be, for example, a desktop or laptop personal computer, a smartphone, a tablet terminal, etc. When realized by multiple user terminals, each user terminal may be owned by a different user.

[0132] When the image processing device 100 is realized by a server and a user terminal, some or all of the information stored in a storage device of one device may be stored in a storage device of the other device. Also, some or all of the information stored in a storage device of either the server or the user terminal may be stored in an external storage device.

[0133] When the image processing device 100 is realized by a server and a user terminal, it can be arbitrarily determined whether each process performed by the image processing device 100 is performed by the server or the user terminal. As an example, the user terminal may be configured to display various information and accept instructions from a user, while the server performs other processes. Examples of user instructions include selecting a style vector or a conversion model, adjusting a style vector, etc. Examples of processes performed by the server include generating a style-converted image, storing the conversion model, style vector, user information, etc.

[0134] When the image processing device 100 is realized by a server and a user terminal, the selection of a style vector or a conversion model in the user terminal includes the selection of a style vector or a conversion model by an operation on the user terminal, and the selection of a style vector or a conversion model by the server based on information from the user terminal.

[0135] When the image processing device 100 is realized by a server and a user terminal, the display device that displays the style vector and the information associated with the style vector may be a display or the like provided in the server or the user terminal.

[0136] The hardware configuration of the server and the user terminal may be the same as that of the image processing device 100. An example of the hardware configuration of the image processing device 100 will be described later.

[0137] [summary] According to this embodiment, it is possible to provide a device that can reuse information used when converting the style of an image. Furthermore, by using the image processing device 100 of this embodiment, it is possible to provide a service that can reuse information used when converting the style of an image.

[0138] The image processing device 100 in this embodiment stores a style-converted image and a style vector used for the style conversion in association with each other, and can use a style vector associated with another image when converting the style of an input image. Therefore, the image processing device 100 in this embodiment can reuse the style vector used when converting the style of an image.

[0139] The image processing device 100 of this embodiment displays the style-converted image in association with the style vector used for the style conversion, and converts the style of the input image using the style vector selected by the user. Therefore, the image processing device 100 of this embodiment allows the user to easily understand the style realized by the style vector and easily select a style vector that is close to the desired style.

[0140] The image processing device 100 in this embodiment can save style vectors generated by other users in favorite information and use them when converting the style of an input image. Therefore, the image processing device 100 in this embodiment allows style vectors to be shared between users.

[0141] When multiple conversion models are available, the image processing device 100 of this embodiment displays only the style vector corresponding to the conversion model selected by the user on the style conversion screen. Therefore, the image processing device 100 of this embodiment can select an appropriate style vector depending on the conversion model to be used.

[0142] The image processing device 100 of this embodiment can arbitrarily adjust each parameter of a style vector. Therefore, the image processing device 100 of this embodiment allows a user to generate a style vector that can achieve a desired style.

[0143] When adjusting a style vector, the image processing device 100 of this embodiment displays an image whose style has been converted using a style vector in which, for each parameter, the parameter is set to its maximum value while keeping the other parameters fixed, and an image whose style has been converted using a style vector in which the parameter is set to its minimum value. Thus, the image processing device 100 of this embodiment allows the user to adjust the style vector while checking the effect of each parameter on the style.

[0144] In the image processing device 100 of this embodiment, only users who have subscribed to a paid plan can save and adjust style vectors. When adjusting a style vector, points are consumed each time a parameter is adjusted. Adjusting a style vector is a computationally intensive process that involves performing multiple style conversions. Therefore, the image processing device 100 of this embodiment can efficiently allocate computational resources to users who are highly motivated to use the system.

[0145] The image processing device 100 according to this embodiment can generate a second image by converting the style of a first image based on a style vector and a conversion model. The style vector can be selected from one or more style vectors in response to a user instruction. The selected style vector can be the style vector used to generate a fourth image by converting the style of a third image based on the conversion model.

[0146] The image processing device 100 of this embodiment displays at least one or more style vectors or information associated with each of the one or more style vectors on a display device before generating a second image, and then generates the second image based on a style vector selected in response to a user instruction. At this time, the associated information includes one or more images generated using each of the one or more style vectors.

[0147] The image processing device 100 in this embodiment can generate a fourth image by converting the style of a third image based on a style vector and a conversion model, in response to an instruction from the user or another user.

[0148] The image processing device 100 in this embodiment can select a conversion model in accordance with a user's instruction and display at least one or more style vectors corresponding to the conversion model or information associated with each of the one or more style vectors on a display device.

[0149] After selecting a style vector, the image processing device 100 according to this embodiment can generate a second image based on the style vector adjusted in accordance with a user's instruction and a transformation model. At this time, it can determine whether the selected style vector is adjustable in accordance with the user's attribute information. Furthermore, when the style vector is adjusted in accordance with a user's instruction, points held by the user can be deducted.

[0150] The image processing device 100 in this embodiment generates a sixth image by converting the style of the fifth image based on a style vector in which the parameter is set to a first value for each parameter of the selected style vector, and a seventh image by converting the style of the fifth image based on a style vector in which the parameter is set to a second value, and can display the sixth image and the seventh image on a display device. In this case, the fifth image can be a partial area included in the first image. Furthermore, when the parameter value is changed by a user instruction, the sixth image and the seventh image can be regenerated for each parameter of the style vector.

[0151] The user terminal in this embodiment acquires a first image to be converted, displays at least one or more style vectors or information associated with each of the one or more style vectors on a display device, selects a style vector from the one or more style vectors in response to a user instruction, acquires a second image obtained by converting the style of the first image, and displays the second image on the display device. The selected style vector may be a style vector used to generate a fourth image obtained by converting the style of a third image based on a conversion model. The second image may be an image generated based on the conversion model and the selected style vector.

[0152] The user terminal in this embodiment can select a conversion model from one or more conversion models in response to a user instruction. The fourth image can be an image generated based on an instruction from the user or another user. The associated information can include one or more images generated using one or more style vectors, respectively.

[0153] After selecting a style vector, the user terminal in this embodiment can generate a second image based on the style vector and transformation model adjusted according to a user instruction.

[0154] In the user terminal of this embodiment, the user can adjust the style vector by operating a slider bar corresponding to each parameter of the style vector displayed on the display device.

[0155] [Hardware configuration of information processing device] Some or all of the devices (image processing devices 100) in the above-described embodiments may be configured as hardware, or may be configured as software (program) information processing executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or the like. When configured as software information processing, software that realizes at least some of the functions of each device in the above-described embodiments may be stored on a non-transitory storage medium (non-transitory computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or a USB (Universal Serial Bus) memory, and the software information processing may be executed by loading the software into a computer. The software may also be downloaded via a communication network. Furthermore, all or part of the software processing may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), thereby allowing the software information processing to be executed by hardware.

[0156] The storage medium that stores the software may be a removable medium such as an optical disk, or a fixed medium such as a hard disk, memory, etc. The storage medium may be provided inside the computer (main storage device, auxiliary storage device, etc.) or outside the computer.

[0157] 14 is a block diagram showing an example of the hardware configuration of each device (image processing device 100) in the above-described embodiment. Each device may be realized as a computer 7 including, for example, a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76.

[0158] Although the computer 7 in FIG. 14 includes one of each component, it may also include multiple of the same component. Also, while FIG. 14 shows one computer 7, the software may be installed on multiple computers, and each of the multiple computers may execute the same or different parts of the software. In this case, a distributed computing configuration may be used in which each computer communicates with the other computers via a network interface 74 or the like to execute the processing. In other words, each device (image processing device 100) in the above-described embodiment may be configured as a system in which one or more computers execute instructions stored in one or more storage devices to realize its functions. Furthermore, the system may be configured such that information transmitted from a terminal is processed by one or more computers provided on a cloud, and the processing results are transmitted to the terminal.

[0159] The various calculations of each device (image processing device 100) in the above-described embodiment may be executed in parallel using one or more processors, or using multiple computers via a network. Furthermore, the various calculations may be distributed to multiple processing cores within a processor and executed in parallel. Furthermore, some or all of the processes, means, etc. of the present disclosure may be realized by at least one of a processor and a storage device provided on a cloud that can communicate with a computer 7 via a network. Thus, each device in the above-described embodiment may be implemented in the form of parallel computing using one or more computers.

[0160] The processor 71 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that at least controls a computer or performs calculations. The processor 71 may be a general-purpose processor, a dedicated processing circuit designed to perform a specific calculation, or a semiconductor device that includes both a general-purpose processor and a dedicated processing circuit. The processor 71 may also include an optical circuit or a calculation function based on quantum computing.

[0161] The processor 71 may perform arithmetic processing based on data or software input from each device or the like configured inside the computer 7, and may output the calculation results or control signals to each device or the like. The processor 71 may control each component constituting the computer 7 by executing the OS (Operating System) of the computer 7, applications, etc.

[0162] Each device (image processing device 100) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, the electronic circuits may communicate with each other via wire or wirelessly.

[0163] The main memory device 72 may store instructions executed by the processor 71, various data, etc., and information stored in the main memory device 72 may be read by the processor 71. The auxiliary memory device 73 is a memory device other than the main memory device 72. Note that these memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data, etc. in each device (image processing device 100) in the above-described embodiment may be realized by the main memory device 72 or the auxiliary memory device 73, or may be realized by an internal memory built into the processor 71. For example, each memory unit (model memory unit 110, image information memory unit 111, style memory unit 112, and user information memory unit 113) in the above-described embodiment may be realized by the main memory device 72 or the auxiliary memory device 73.

[0164] When each device (image processing device 100) in the above-described embodiment is configured with at least one storage device (memory) and at least one processor connected (coupled) to this at least one storage device, at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Also, a configuration in which at least one processor out of multiple processors is connected to at least one storage device out of multiple storage devices may be included. Also, this configuration may be realized by storage devices and processors included in multiple computers. Furthermore, a configuration in which a storage device is integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

[0165] The network interface 74 is an interface for connecting to the communication network 8 wirelessly or via a wire. The network interface 74 may be an appropriate interface, such as one that conforms to an existing communication standard. Information may be exchanged with an external device 9A connected via the communication network 8 via the network interface 74. The communication network 8 may be any one of a WAN (Wide Area Network), a LAN (Local Area Network), a PAN (Personal Area Network), etc., or a combination thereof, as long as information is exchanged between the computer 7 and the external device 9A. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark) or NFC (Near Field Communication), etc.

[0166] The device interface 75 is an interface such as a USB that directly connects to the external device 9B.

[0167] The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device connected directly to the computer 7.

[0168] For example, the external device 9A or the external device 9B may be an input device. The input device may be a device such as a camera, a microphone, a motion capture device, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the computer 7. Alternatively, the external device 9A or the external device 9B may be a device equipped with an input unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0169] Furthermore, the external device 9A or the external device B may be, for example, an output device. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or a speaker that outputs sound or the like. Alternatively, the output device may be a device including an output unit, a memory, and a processor, such as a personal computer, a tablet terminal, or a smartphone.

[0170] Furthermore, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

[0171] Furthermore, the external device 9A or the external device 9B may be a device having some of the functions of the components of each device (image processing device 100) in the above-described embodiment. That is, the computer 7 may transmit some or all of the processing results to the external device 9A or the external device 9B, or may receive some or all of the processing results from the external device 9A or the external device 9B.

[0172] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as aa, abb, aabbcc, etc. Furthermore, it also includes the addition of elements other than the enumerated elements (a, b, and c), such as having d, as in abcd.

[0173] In this specification (including claims), when expressions such as "using data as input / based on / according to / in response to data" (including similar expressions) are used, unless otherwise specified, this includes cases where the data itself is used, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of data, etc.) is used. Furthermore, when a statement is made that a result is obtained "using data as input / based on / according to / in response to data" (including similar expressions), this includes cases where the result is obtained based solely on the data, or where the result is influenced by other data, factors, conditions, and / or states other than the data itself, unless otherwise specified. Furthermore, when a statement is made that "data is output" (including similar expressions), this includes cases where the data itself is used as output, or where data that has been processed in some way (e.g., data with noise added, normalized data, features extracted from data, intermediate representations of various data, etc.) is used as output, unless otherwise specified.

[0174] When the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that encompass any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately according to the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted as being included in these terms without limitation.

[0175] In this specification (including the claims), the expression "A configured to B" may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Also, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure, etc. of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0176] Whenever words implying containing or possessing (e.g., "comprising / including," "having," etc.) are used in this specification (including the claims), they are intended to be open-ended terms that include the inclusion or possession of things other than the object designated by the object of the term. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., expressions using the articles "a" or "an"), the expression should be construed as not being limited to a specific number.

[0177] In this specification (including the claims), even if expressions such as "one or more" and "at least one" are used in some places and expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or that imply a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0178] In this specification, when a particular advantage / result is described as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or circumstances, and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or circumstances are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0179] When terms such as "maximize" and "maximization" are used in this specification (including the claims), they include finding a global maximum, finding an approximation of a global maximum, finding a local maximum, and finding an approximation of a local maximum, and should be interpreted appropriately according to the context in which the term is used. They also include finding approximations of these maxima probabilistically or heuristically. Similarly, when terms such as "minimize" and "minimization" are used, they include finding a global minimum, finding an approximation of a global minimum, finding a local minimum, and finding an approximation of a local minimum, and should be interpreted appropriately according to the context in which the term is used. They also include finding approximations of these minima probabilistically or heuristically. Similarly, when terms such as "optimize" and "optimization" are used, they include finding a global optimum, finding an approximation of a global optimum, finding a local optimum, and finding an approximation of a local optimum, and should be interpreted appropriately according to the context in which the term is used. It also includes finding approximations of these optimum values ​​probabilistically or heuristically.

[0180] In this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" (including similar expressions) is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include electronic circuits, devices including electronic circuits, etc.

[0181] In this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices may store only a portion of the data, or may store the entire data. Also, a configuration in which only some of the multiple storage devices store data may be included.

[0182] In this specification (including the claims), terms such as "first," "second," etc. are used merely as a way of distinguishing between two or more elements, and are not necessarily intended to impose technical meanings such as temporal aspect, spatial aspect, sequence, quantity, etc. Thus, for example, a reference to a first element and a second element does not necessarily mean that only two elements can be employed therein, that the first element must precede the second element, that the first element must be present in order for the second element to be present, etc.

[0183] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention, which is derived from the content defined in the claims and their equivalents. For example, when numerical values ​​or formulas are used in the above-described embodiments, they are shown for illustrative purposes and do not limit the scope of the present disclosure. Furthermore, the order of each operation shown in the embodiments is also illustrative and does not limit the scope of the present disclosure. [Explanation of symbols]

[0184] 100 Image processing device 101 Image conversion unit 102 Image extraction unit 103 Style Adjustment Section 104 Points Management Department 110 Model memory section 111 Image information storage unit 112 Style Memory 113 User information storage unit

Claims

1. at least one memory; at least one processor; The at least one processor generating a second image by transforming the style of the first image based on the style vector and the transformation model; the style vector is selected from one or more style vectors in response to a user instruction, the selected style vector is a style vector used to generate a fourth image obtained by converting the style of the third image based on the conversion model. Image processing device.

2. before generating the second image, at least one of the one or more style vectors or information associated with each of the one or more style vectors is displayed on a display device; The at least one processor After the display, the second image is generated based on the style vector selected in response to an instruction from the user. The image processing device according to claim 1 .

3. the associated information includes one or more images generated using each of the one or more style vectors. The image processing device according to claim 2 .

4. The at least one processor further comprises: generating the fourth image by converting the style of the third image based on the style vector and the conversion model in accordance with an instruction from the user or another user; The image processing device according to claim 1 .

5. The at least one processor selecting the transformation model in response to an instruction from the user; At least one of the one or more style vectors corresponding to the transformation model or information associated with each of the one or more style vectors is displayed on the display device. The image processing device according to claim 2 .

6. The at least one processor generating the second image based on the style vector and the transformation model adjusted in accordance with an instruction from the user after the selection; 6. The image processing device according to claim 1.

7. The at least one processor determining whether the selected style vector is adjustable according to attribute information of the user; The image processing device according to claim 6 .

8. The at least one processor When the style vector is adjusted in accordance with an instruction from the user, points held by the user are subtracted. The image processing device according to claim 7 .

9. The at least one processor generating a sixth image by converting the style of the fifth image based on the style vector in which the parameter is set to a first value, and a seventh image by converting the style of the fifth image based on the style vector in which the parameter is set to a second value, for each parameter of the selected style vector; the sixth image and the seventh image are displayed on a display device. The image processing device according to claim 6 .

10. the fifth image is a partial area included in the first image; The image processing device according to claim 9 .

11. The at least one processor regenerating the sixth image and the seventh image for each parameter of the style vector when the value of the parameter is changed in response to an instruction from the user; The image processing device according to claim 9 .

12. a display device; at least one memory; at least one processor; The at least one processor obtaining a first image to be transformed; displaying on the display device either at least one or more style vectors or information associated with each of the one or more style vectors; selecting a style vector from the one or more style vectors in response to a user instruction; obtaining a second image that is a style conversion of the first image; displaying the second image on the display device; Run the selected style vector is a style vector used to generate a fourth image by converting the style of the third image based on a transformation model; the second image is an image generated based on the transformation model and the selected style vector. Image display device.

13. The at least one processor selecting the transformation model from one or more transformation models in response to an instruction from the user; The image display device according to claim 12.

14. the fourth image is an image generated based on an instruction from the user or another user; The image display device according to claim 12.

15. the associated information includes one or more images generated using each of the one or more style vectors. The image display device according to claim 12.

16. The at least one processor generating the second image based on the style vector and the transformation model adjusted in accordance with an instruction from the user after the selection; 16. The image display device according to claim 12.

17. the style vector is adjusted by the user operating a slider bar corresponding to each parameter of the style vector displayed on the display device; The image display device according to claim 16.

18. At least one processor generating a second image by transforming the style of the first image based on the style vector and the transformation model; the style vector is selected from one or more style vectors in response to a user instruction, the selected style vector is a style vector used to generate a fourth image obtained by converting the style of the third image based on the conversion model. Image processing methods.

19. At least one processor obtaining a first image to be transformed; displaying either at least one or more style vectors or information associated with each of said one or more style vectors on a display device; selecting a style vector from the one or more style vectors in response to a user instruction; obtaining a second image that is a style conversion of the first image; displaying the second image on the display device; Run the selected style vector is a style vector used to generate a fourth image by converting the style of the third image based on a transformation model; the second image is an image generated based on the transformation model and the selected style vector. Image display method.

20. At least one processor obtaining a first image to be transformed; displaying either at least one or more style vectors or information associated with each of said one or more style vectors on a display device; selecting a style vector from the one or more style vectors in response to a user instruction; obtaining a second image that is a style conversion of the first image; displaying the second image on the display device; Execute the selected style vector is a style vector used to generate a fourth image by converting the style of the third image based on a transformation model; the second image is an image generated based on the transformation model and the selected style vector. program.

Images