Information processing device, information processing program, and information processing method

Abstract

An object of the present invention is to provide an information processing device that can automatically perform gradation processing on image data that is the subject of gradation processing. [Solution] The system includes a gradation processing target image data acquisition unit that acquires image data to be gradation processed, and a gradation processing unit that performs gradation processing on the image data to be gradation processed based on a learned model that has been pre-trained using training data regarding gradation processing for the image data to be gradation processed.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing program, and an information processing method that can automatically perform gradation processing on image data. [Background technology]

[0002] In recent years, machine learning using multi-layered neural networks known as deep learning has been applied to a variety of fields, including image processing such as image recognition and image generation, where it has produced remarkable results.

[0003] For example, Non-Patent Document 1 describes an automatic coloring process for black and white photographs that is realized by a deep network, and the coloring process for black and white photographs is realized by machine learning.

[0004] In addition to the need to apply color processing to image data as described in Non-Patent Document 1, there is also a need to apply gradation processing to image data such as line art data that does not contain gradation information, or to adjust the gradation of image data that already contains gradation information. Gradation processing here refers to shadow color processing that adds shading, diagonal shading that applies diagonal lines to shadow areas, toning processing that adds tones used in manga and the like, highlighting processing that changes parts of existing shading, tone conversion processing that changes toned areas of a toned image to a grayscale representation, and special printing processing that generates data for special printing such as embossing and gold leaf processing. Traditionally, the process of finishing images by applying gradation processing to image data has been performed by skilled workers, but manual work has presented the problem of being unable to handle large numbers of pages. Therefore, a system capable of performing gradation processing automatically has been needed. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Automatic colorization of black-and-white photographs by learning global and local features using deep networks Satoshi Iizuka, Edgar Simosera, Hiroshi Ishikawa (http: / / hi.cs.waseda.ac.jp / ~iizuka / projects / colorization / ja / ) Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an information processing device, an information processing program, and an information processing method that are capable of automatically performing gradation processing. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by comprising a gradation processing target image data acquisition unit that acquires gradation processing target image data, and a gradation processing unit that performs gradation processing on the gradation processing target image data based on a pre-trained model.

[0008] The information processing device of the present invention is characterized by comprising a gradation processing target image data acquisition unit that acquires gradation processing target image data, and a gradation processing unit that outputs at least one mask channel for performing gradation processing on the gradation processing target image data based on a pre-trained model.

[0009] In addition, the information processing device of the present invention is characterized in that it includes a processing content designation receiving unit that receives one or more user specifications regarding which gradation processing content to select from a plurality of gradation processing contents, and the gradation processing unit performs gradation processing on the image data to be gradated based on a trained model corresponding to the gradation processing content designated by the processing content designation receiving unit from a plurality of trained models that have been trained in advance.

[0010] In addition, the information processing device of the present invention is characterized in that it includes a processing content designation receiving unit that receives one or more user specifications regarding which gradation processing content to select from a plurality of gradation processing contents, and the gradation processing unit outputs at least one or more mask channels for applying the specified gradation processing to the image data to be gradation processed based on a trained model corresponding to the gradation processing content designated by the processing content designation receiving unit from a plurality of trained models that have been trained in advance.

[0011] In addition, the information processing device of the present invention is characterized in that it includes a hint information acquisition unit that acquires hint information for gradation processing for at least one location of the image data to be gradation processed, and the gradation processing unit performs processing based on the image data to be gradation processed and the hint information.

[0012] In addition, the information processing device of the present invention is characterized in that the learned model is one that has been trained based on both training data that does not provide hint information for gradation processing and training data that provides hint information for gradation processing for at least one location on the image data to be gradation processed.

[0013] In addition, the information processing device of the present invention is characterized in that it includes a light source position information acquisition unit that acquires at least one user specification of light source position information for specifying the position of a light source for the image data to be gradation processed, and the gradation processing unit performs processing based on the image data to be gradation processed and the light source position information.

[0014] In addition, the information processing device of the present invention is characterized in that the learned model is one that has been trained based on both training data that does not provide light source position information and training data that provides light source position information for at least one location in the image data to be subjected to gradation processing.

[0015] The information processing program of the present invention is characterized in that it causes a computer to realize a gradation processing target image data acquisition function that acquires image data to be subjected to gradation processing, and a gradation processing function that performs gradation processing on the image data to be subjected to gradation processing based on a pre-trained model.

[0016] The information processing method according to the present invention is characterized in that it includes a gradation processing target image data acquisition procedure for acquiring gradation processing target image data, and a gradation processing procedure for performing gradation processing on the gradation processing target image data based on a pre-trained model.

[0017] The information processing apparatus according to the present invention comprises: a gradation processing target image data input form display unit that displays a form area for a user to input gradation processing target image data on a display screen; a gradation processing target image display unit that displays an image represented by the input gradation processing target image data in a gradation processing target image display area provided on the display screen; a gradation-processed image display unit that displays an image represented by gradation-processed image data obtained by performing gradation processing on the gradation-processing target image data based on a trained model that has been trained in advance, in a gradation-processed image display area provided on the display screen; The present invention is characterized by comprising:

[0018] In addition, the information processing device of the present invention includes a hint information input tool display unit that displays on the display screen a hint information input tool for specifying areas where gradation processing should be performed on an image represented by the gradation processing target image data displayed in the gradation processing target image display area, and accepts input of hint information, and the gradation processed image display unit displays, in the gradation processed image display area provided on the display screen, an image represented by the gradation processed image data obtained by performing gradation processing on the gradation processing target image data while including the hint information, based on a trained model that has been trained in advance.

[0019] In addition, the information processing device of the present invention includes a light source position information input tool display unit that displays on the display screen a light source position information input tool for specifying a light source position for an image represented by the gradation processing target image data displayed in the gradation processing target image display area and accepts input of light source position information, and the gradation processed image display unit displays, in the gradation processed image display area provided on the display screen, an image represented by the gradation processed image data obtained by performing gradation processing on the gradation processing target image data while including the light source position information, based on a trained model that has been trained in advance. [Effects of the Invention]

[0020] According to the present invention, training data is used for gradation processing of image data to be subjected to gradation processing. Based on a trained model that has been trained in advance, it becomes possible to automatically perform gradation processing on image data to be subjected to gradation processing. Also, it becomes possible to appropriately perform gradation processing by providing hint information regarding the position where gradation processing is to be performed on the acquired image data to be subjected to gradation processing and light source position information for specifying the light source position. By proceeding with learning in a manner that includes specification of hint information and light source position information during the learning process of the trained model used for gradation processing, it becomes possible to perform gradation processing with specification of hint information and light source position information on image data to be subjected to gradation processing.

[0021] Furthermore, according to the present invention, an information processing device is configured to display a gradation processing target image display area and a gradation-processed image display area on a display screen visually recognized by the user, so that the user can compare and observe the original gradation processing target image data with the gradation-processed image data, thereby enabling a direct comparison of the atmosphere of the image that changes before and after gradation processing.Furthermore, hint information and light source position information for specifying the portion to be gradated in the gradation processing target image indicated by the gradation processing target image data displayed in the gradation processing target image display area can be input, and gradation processing can be performed with the hint information and light source position information attached.Therefore, the user can freely input hints and light source position information for the portion of the gradation processing target image where he or she wants to perform gradation processing and perform gradation processing. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram showing the configuration of an automatic gradation processing device 10 according to the present invention. [Figure 2] FIG. 1 is a block diagram showing a hardware configuration required to realize an automatic gradation processing device 10. [Figure 3] 1 is a block diagram showing an example of a system configuration of an automatic gradation processing device 10. FIG. [Figure 4] FIG. 1 is an explanatory diagram illustrating the concept of a neural network. [Figure 5] 10A and 10B are explanatory diagrams showing how light source position information is given to image data to be subjected to gradation processing. [Figure 6] FIG. 2 is a flowchart showing the flow of gradation processing in the automatic gradation processing device 10 of the present embodiment. [Figure 7] FIG. 2 is an explanatory diagram showing an example of a display screen displayed by the graphical user interface program of the present example. [Figure 8] FIG. 11 is a flowchart showing a process flow when an automatic gradation processing tool is provided based on a GUI according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] [First embodiment] An example of an automatic gradation processing device according to a first embodiment of an information processing device of the present invention will be described below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of an automatic gradation processing device 10 according to the present invention. As shown in Fig. 1, the automatic gradation processing device 10 includes at least a gradation processing target image data acquisition unit 11, a hint information acquisition unit 12, a light source position information acquisition unit 13, a gradation processing unit 14, and a storage unit 15.

[0024] The automatic gradation processing device 10 may be a device designed as a dedicated machine, but is also realizable by a general-purpose computer. Fig. 2 is a block diagram showing the hardware configuration required to realize the automatic gradation processing device 10. As shown in Fig. 2, the automatic gradation processing device 10 includes a CPU (Central Processing Unit) 51, a GPU (Graphics Processing Unit: image processing device) 52, memory 53, and storage 54 such as a hard disk drive or SSD (solid state drive), which would normally be included in a general-purpose computer. It also includes an input device 55 such as a mouse or keyboard, an output device 56 such as a display or printer, and a communication device 57 for connecting to a communication network, all of which are connected via a bus 58. It is assumed that

[0025] 3 is a block diagram showing an example of a system configuration for the automatic gradation processing device 10. In FIG. 3, a server device 60 and a plurality of terminal devices 701 to 70n are configured to be mutually connectable via a communication network 80. For example, the server device 60 in FIG. 3 may function as the automatic gradation processing device 10, and any of the plurality of terminal devices 701 to 70n may connect to and use the server device 60 functioning as the automatic gradation processing device 10 via the communication network. In this case, the terminal device 70 may be configured to install a graphical user interface program for using the automatic gradation processing device 10, or the graphical user interface program on the server may be used via a browser, or the terminal device 70 may be an information processing device having a function for displaying various displays as a graphical user interface.

[0026] Furthermore, all of the components of the automatic gradation processing device 10 described below do not necessarily need to be included in the same device. Instead, some of the components may be included in other devices. For example, some of the components may be included in a server device 60 and one of multiple terminal devices 701-70n connectable via a communication network, and the automatic gradation processing device 10 may communicate with and use the components included in the other devices. The server device 60 is not limited to a single device, and multiple server devices may be used. Furthermore, the trained models described below may be stored in the device serving as the automatic gradation processing device 10, or may be distributed among other devices such as the server device 60 and multiple terminal devices 701-70n, and connected to a device having the trained model to be used via a communication network each time. In other words, as long as a trained model stored in some kind of trained model storage means can be used, it does not matter whether the trained model storage means is included in the automatic gradation processing device 10 itself or in another device.

[0027] The gradation processing target image data acquisition unit 11 has a function of acquiring gradation processing target image data. Here, gradation processing in this example refers to various processes for changing gradation information, such as shadow color processing, which applies shading to an image without shading; diagonal line drawing processing, which draws diagonal lines to express shading; tone processing, which applies tones to image data; highlight processing, which changes the gradation of image data; tone conversion processing, which changes tones in an image with tones to a grayscale representation; special printing processing, which generates data for special printing such as embossing or gold leafing; and other processes for changing gradation information. The gradation processing target image data conceptually includes both image data to which gradation information is to be added, such as binary black-and-white line drawing data that does not contain gradation information, and image data that already contains gradation information and whose gradation information is to be further modified, such as colored images, shaded images, toned images, and photographs. The gradation processing target image data acquisition unit 11 acquires the gradation processing target image data, for example, when a user selects the image data to be gradated.

[0028] The hint information acquisition unit 12 has a function of receiving and acquiring hint information from the user, including at least one specification of a portion of the gradation processing target image data where gradation processing is to be performed. An example of a method for the user to specify hint information may be a procedure of receiving a specification of a type of gradation processing such as shadow color processing, diagonal line processing, tone processing, highlight processing, etc. (A processing content specification acceptance unit may be provided as one of the components of the automatic gradation processing device 10) and a procedure of specifying a portion of the gradation processing target image data displayed on the display screen where the specified gradation processing is to be performed. Furthermore, not only when specifying one type of gradation processing, but also when specifying two or more types of gradation processing simultaneously, the portion of the gradation processing target image data may be specified. Note that when gradation processing is performed without obtaining hint information, the obtaining of hint information by the hint information acquisition unit 12 can be omitted. The obtained hint information may be used in the gradation processing described later. The hint information acquisition unit 12 is used for gradation processing in the gradation processing unit 14. Note that the hint information acquisition unit 12 is not an essential component, and even if hint information is not provided, gradation processing is possible using a trained model trained using training data that does not include hint information.

[0029] The light source position information acquisition unit 13 has a function of acquiring light source position information when providing light source position information when performing gradation processing on image data to be gradated. Gradation processing can be considered an act of depicting the influence of a virtual light source on image data to be gradated, and the position at which gradation processing is performed varies depending on the position of the light source. Therefore, light source position information may be provided to the image data to be gradated. Various methods are conceivable for providing light source position information to image data to be gradated. For example, the user may specify the light source position, or the light source position in the image data to be gradated may be automatically determined based on a trained model for light source position determination that has previously been trained to determine the light source position in image data to be gradated. The obtained light source position information is used for gradation processing in the gradation processing unit 14, which will be described later. Note that the light source position information acquisition unit 13 is not a required component. Even if light source position information is not provided, gradation processing is possible using a trained model trained using training data that does not include light source position information. Furthermore, if light source position information that contradicts the hint information acquired by the hint information acquisition unit 12 is specified, there is a risk that gradation processing that reflects both information cannot be executed. Therefore, a configuration may be adopted in which either the hint information or the light source position information is given priority.

[0030] FIG. 5 is an explanatory diagram showing how light source position information is assigned to image data to be subjected to gradation processing. FIG. 5(a) is an explanatory diagram showing a case where light source position information is assigned two-dimensionally to image data to be subjected to gradation processing, and FIG. 5(b) is an explanatory diagram showing a case where light source position information is assigned three-dimensionally to image data to be subjected to gradation processing. When light source position information is assigned two-dimensionally as in FIG. 5(a), light source position information is assigned from any direction on the same plane as the image data to be subjected to gradation processing. When light source position information is assigned three-dimensionally as in FIG. 5(b), light source position information is assigned by specifying any position on the surface of a sphere centered on the image data to be subjected to gradation processing. In either case of FIG. 5(a) or (b), the location of the light source position information may be specified based on a predetermined division. For example, when light source position information is assigned two-dimensionally, eight directions as in FIG. 5(a) may be assigned, or twice as many as 16 directions, or even more detailed designation may be possible. When providing light source position information three-dimensionally, it may be divided into 8 directions (xy plane) x 8 directions (xz plane) for a total of 64 directions, or into 16 directions (xy plane) x 16 directions (xz plane) for a total of 256 directions, or it may be possible to specify more precisely. In this way, by accepting specification from a predetermined division of direction, it becomes possible to perform gradation processing using examples learned based on training data tagged with light source position information for directions in the same division in a trained model.

[0031] The gradation processing unit 14 has a function of outputting at least one or more mask channels for performing gradation processing on the image data to be gradated, based on a trained model that has been trained in advance using training data to generate gradation processing results as mask channels to be superimposed on the image data to be gradated. In order for the gradation processing unit 14 to perform gradation processing, at least one type of specification regarding the image data to be gradated and the gradation processing content must be given. Also, hint information regarding the position to perform gradation processing acquired by the hint information acquisition unit 12 and light source position information acquired by the light source position information acquisition unit 13 may be given. The gradation processing unit 14, having received this information, performs gradation processing using a trained model that has been trained in advance regarding gradation processing. Note that for some gradation processing, such as shadow color processing, it is also possible to perform image processing directly on the image data to be gradated, without outputting it as a mask channel. Processing such as tone processing, which expresses a specific texture within a predetermined range, can be performed by specifying which type of tone to apply where. The mask channel output from the trained model may be output as a mask channel for performing actual tone pasting or the like in the automatic gradation processing device 10 based on the information of the mask channel output from the trained model, or a tone may be pasted in a layer different from the image data to be gradated using layers, which will be described later. Furthermore, these processes may be performed in a server device serving as the automatic gradation processing device 10, or a terminal device serving as the automatic gradation processing device 10 may acquire the mask channel output from the trained model on the server device and perform actual tone pasting or the like in the terminal device.

[0032] Here, a learning method for the trained model will be described. The trained model is a model that learns to input image data to be subjected to gradation processing and output at least one or more mask channels for performing gradation processing, and is, for example, a model configured with a neural network including a convolutional neural network (CNN). FIG. 4 is an explanatory diagram showing the concept of a neural network. A model for learning can be configured, for example, with an encoder that extracts abstract features from input image data to be subjected to gradation processing, and a decoder that outputs a mask channel for gradation processing based on the extracted features. This is merely an example, and various neural network configurations are applicable. Learning is performed on a neural network configured in this way.

[0033] Training data is used during learning. The training data may be, for example, a set of image data to be gradated and correct image data obtained by gradation processing of the image data to be gradated. The image data to be gradated set in the training data may include various types of image data, such as line drawing data, colored images, shaded images, toned images, and photographs. Correct image data to which gradation processing has been further applied is also prepared to form the training data set. The learning process involves inputting the image data to be gradated into a model to be trained, and outputting the results of the gradation processing as a mask channel. A loss function is calculated using the correct image data and the gradation-processed image data, in which the mask channel resulting from the gradation processing has been superimposed on the image data to be gradated, and the loss function is updated to reduce the neural network parameters. This parameter update is performed based on multiple training data to obtain a trained model.

[0034] Furthermore, during learning, hint information on the position where gradation processing is to be performed may be provided together with the image data to be gradated, thereby allowing the model to learn about gradation processing with hint information. For example, the hint information may be a position randomly picked from the correct image data or gradation information generated from the correct image data, and input as hint information to the model together with the image data to be gradated, and the model is trained. At this time, it is preferable to perform learning by randomly switching the number of pieces of hint information provided from 0 to a finite predetermined number, thereby obtaining a trained model that can perform gradation processing in the same way whether 0 hint information or a large number of hint information is provided.

[0035] Furthermore, during learning, light source position information may be provided along with the image data to be gradated, allowing the model to learn about gradation processing with light source position information. Light source position information from the correct image data is extracted, and the light source position information is input into the model along with the image data to be gradated, and the model is trained. The light source position information is similar to tag information, and is specified based on a predetermined classification. By performing sufficient learning that includes specifying light source position information from various directions, it becomes possible to perform gradation processing regardless of the direction specified. Note that light source position information may be generated mechanically from 3D data or 2D data. Also, light source position information does not necessarily have to specify an exact position, and may be rough information such as backlighting.

[0036] Furthermore, while the above explanation of learning has been presented as a technique common to all types of gradation processing, there are also elements unique to each type of gradation processing. Shadow color processing and highlight processing achieve shadow and highlight effects by differentiating the gradation information (brightness information) from the surrounding area in the final gradation-processed image data of the image data to be gradation-processed. In contrast, tone processing involves determining the area to be processed and which of multiple types of tones to use, and then applying the tones. That is, in the case of tone processing, a set of valid classes corresponding to the number of types of tones is established, and processing is performed using a neural network as a classification problem for the enabled classes for the gradation-processed image data, thereby processing each area of ​​the gradation-processed image data with the tone corresponding to the most likely class. Furthermore, to simultaneously process multiple types of tones, a mask channel is prepared for each class, and multiple tone processes using multiple mask channels are ultimately superimposed on the gradation-processed image data to obtain the gradation-processed image data. It goes without saying that not only tone processing but also other gradation processing including shadow color processing, highlight processing, diagonal line processing, etc. may be performed by setting a set of multiple effective classes according to the number of types of processing methods and performing processing using a neural network as a classification problem of the effective classes, thereby allowing the processing methods to be used appropriately.

[0037] Furthermore, the trained model is generated by performing training for each type of gradation processing, such as shadow color processing, highlight processing, diagonal line processing, and tone processing, to obtain a trained model. When performing gradation processing, when at least one type of gradation processing content is specified, the trained model for that gradation processing content is used. Note that an automatic gradation processing device may be specialized for one type of gradation processing, in which case only a trained model for that one type of gradation processing is generated and stored.

[0038] The storage unit 15 has a function of storing data required for various processes performed in the automatic gradation processing device 10, which includes the gradation processing target image data acquisition unit 11, the hint information acquisition unit 12, the light source position information acquisition unit 13, and the gradation processing unit 14, as well as data obtained as a result of the processes. In addition, a trained model for each type of gradation processing obtained by performing learning in advance may be stored in this storage unit 15.

[0039] Next, the flow of gradation processing in the automatic gradation processing device 10 of this example will be described. FIG. 6 is a flowchart showing the flow of gradation processing in the automatic gradation processing device 10 of this example. Gradation processing in the automatic gradation processing device 10 of this example begins by first acquiring image data to be subjected to gradation processing (step S101). For example, acquisition is performed by a user selecting image data to be subjected to gradation processing. The content of gradation processing to be performed on the image data to be subjected to gradation processing may be determined before acquiring the image data to be subjected to gradation processing, or may be determined after acquiring the image data to be subjected to gradation processing.

[0040] Following acquisition of the image data to be subjected to gradation processing, hint information regarding gradation processing for the image data to be subjected to gradation processing is acquired (step S102). The hint information is information including a designation of which position of the image data to be subjected to gradation processing with the determined gradation processing content. Not only can one type of gradation processing be designated, but two or more types of gradation processing may also be designated at the same time, in which case the hint information includes a designation of the type of gradation processing and a designation of the location to be subjected to gradation processing. Note that gradation processing can be performed even if hint information is not input.

[0041] Next, light source position information for the image data to be subjected to gradation processing is acquired (step S103). The light source position information may be specified by the user, or may be acquired by automatically determining the light source position in the image data to be subjected to gradation processing based on a trained model for determining the light source position. Note that gradation processing can be performed even if light source position information is not input. be.

[0042] Then, gradation processing is performed on the image data to be gradated (step S104). The gradation processing is performed by inputting the image data to be gradated, hint information, and light source position information to a trained model corresponding to the determined gradation processing content. Then, the automatic gradation processing device 10 outputs at least one mask channel for performing gradation processing as a result of the gradation processing in the trained model (step S105), and ends the processing.

[0043] As described above, the automatic gradation processing device 10 according to the first embodiment makes it possible to appropriately perform gradation processing after providing hint information regarding the position where gradation processing is to be performed on the acquired image data to be subjected to gradation processing and light source position information for specifying the light source position. By proceeding with learning in a manner that includes specification of hint information and light source position information during the learning process of the trained model used for gradation processing, it becomes possible to perform gradation processing with specification of hint information and light source position information on the image data to be subjected to gradation processing.

[0044] [Second embodiment] An information processing device according to a second embodiment will be described below with reference to the drawings. While the first embodiment has been described as an automatic gradation processing device 10, the second embodiment will describe an information processing device for providing a graphical user interface used when using the automatic gradation processing device 10 according to the first embodiment. The information processing device according to the second embodiment will be described as providing a graphical user interface as an automatic gradation processing tool. For example, a server device functioning as the automatic gradation processing device 10 according to the first embodiment may be provided, and the automatic gradation processing tool may be provided to a user who accesses the server device from a terminal device via a communication network. In such a case, the automatic gradation processing tool may be provided not only to the terminal device via a software package, but also by loading and running a graphical user interface (GUI) stored on the server using a browser or the like that displays it on the display of the terminal device. The automatic gradation processing tool refers to a tool used when a user uses the automatic gradation processing device 10 according to the first embodiment. The automatic gradation processing tool may be provided in various ways, such as as an independent program, provided as a web browser, or included as part of a software package such as image editing software.

[0045] In the following explanation, we will use as an example a case where at least one or more trained models are stored in a server device functioning as the automatic gradation processing device 10, and the server device is accessed from a terminal device via a communication network to use the automatic gradation processing tool.However, even if all of these are stored in a terminal device, a similar GUI can be used, so it goes without saying that both are subject to this example.

[0046] 7A and 7B are explanatory diagrams showing an example of a display screen displayed by a GUI as an automatic gradation processing tool of this example, where (a) is the display screen when inputting image data to be gradated, and (b) is a display screen showing an example of a hint information input tool, a gradation processing target image display area for displaying image data to be gradated, and a gradation processed image display area for displaying gradation processed image data in which a mask channel resulting from gradation processing is superimposed on the image data to be gradated. When the automatic gradation processing tool is provided from a server device to a terminal device, first, as shown in FIG. 7A, a gradation processing target image data input form, which is a form area for a user to input image data to be gradated, is displayed on the display of the terminal device via, for example, a web browser. In FIG. 7A, this gradation processing target image data input form allows the user to input image data to be gradated by specifying a file path. However, the present invention is not limited to this, and may be implemented by, for example, selecting line drawing data by drag and drop. Note that in this example, the display screen refers to a screen displayed to a user when a GUI is provided as an automatic gradation processing tool by a graphical user interface program, a web browser, or the like, and includes both a display screen generated by a server device and a display screen generated by a terminal device.

[0047] When the image data to be gradated is designated, an image represented by the image data to be gradated is displayed on the display screen in the image display area for image gradation, as shown in FIG. 7(b). Also, as shown in FIG. 7(b), a hint information input tool is displayed on the display screen for specifying the portion of the image data to be gradated, displayed in the image display area for image gradation. In the example shown in FIG. 7(b), the hint information input tools include "Undo one operation," "Advance one operation," "Select pen to input hint information," "Delete input hint information (eraser)," "Select type of gradation processing," and "Input light source position information," which can also be called a light source position information input tool. However, the hint information input tools are not limited to these. Also, FIG. 7(b) shows the display screen after the type of gradation processing to be performed has been determined. However, a "Switch type of gradation processing" item may be provided as a hint information input tool to input hint information while switching between multiple types of gradation processing. To use the hint information input tool, for example, in the case of tone processing, hint information regarding the position where tone processing is to be performed is given by operating the mouse to specify the position by actually adding dots with the pointer, drawing lines, filling in areas, etc. Then, by clicking the execute button displayed on the same screen by operating the mouse, etc., the gradation processing is performed with the hint information included, and the gradation-processed image data reflecting the hint information is displayed in the gradation-processed image display area.

[0048] Furthermore, among the hint information input tools shown in Fig. 7(b), the way to use "Input light source position information" is as follows: when "Input light source position information" is selected, an animation for specifying the light source position as shown in Fig. 5(a) or (b) is superimposed on the display area for the image to be gradation processed and the user's specification of the light source position is accepted, or a transition is made to a separate screen for specifying the light source position as shown in Fig. 5(a) or (b) and the specification of the light source position is accepted. When the execute button on the display screen is clicked with the light source position information provided, gradation processing is executed with the light source position information included, and the gradation-processed image data reflecting the light source position information is displayed in the gradation-processed image display area.

[0049] FIG. 8 is a flowchart showing the process flow when an automatic gradation processing tool is provided based on a GUI according to the second embodiment. As shown in FIG. 8, the process flow for providing the automatic gradation processing tool begins when the server device displays an input form for image data to be gradated on the display screen of the terminal device and accepts input of image data to be gradated (step S201). When the user inputs the image data to be gradated, the image data to be gradated is transmitted to the server device. The server device, which has acquired the image data to be gradated, displays the image data to be gradated in a display area for image data to be gradated on the display screen (step S202). Next, input of hint information is accepted using a hint information input tool for the image data to be gradated displayed on the display screen (step S203). Furthermore, in step S203, designation of light source position information is also accepted. It is assumed that the type of gradation processing has been determined before the hint information is accepted. It is then determined whether or not an execute button for gradation processing has been pressed (step S204). If the execute button for gradation processing has not been pressed (S204-N), the process returns to step S203 to accept further input of hint information. If the execute button for gradation processing has been pressed (S204-Y), gradation processing that reflects the hint information is executed (step S205). The execution of the process is the same as the flow of the gradation process in the first embodiment described using the flowchart in Fig. 6, and the same processes as steps S101 to S105 in Fig. 5 are executed in this step S205. Finally, the gradation-processed image data obtained by the gradation process is displayed in the gradation-processed image display area on the display screen (step S206), and the process ends.

[0050] As described above, a GUI is provided from the server device to the display screen of the terminal device, and an automatic gradation processing tool is provided to the user via the GUI. The GUI functions to provide a gradation processing target image display area and a gradation-processed image display area on the same display screen. This allows the user to observe the original gradation processing target image data and the gradation-processed image data side by side, allowing for direct comparison of the atmosphere of the image that changes before and after gradation processing. Furthermore, hint information can be input to specify the area where gradation processing should be performed on the gradation processing target image indicated by the gradation processing target image data displayed in the gradation processing target image display area. Gradation processing can be performed with the hint information attached. This allows the user to freely provide hints for the area where gradation processing is desired on the gradation processing target image and perform gradation processing. Note that adding this hint information does not necessarily determine that gradation processing will be performed on that location. Since the trained model is caused to perform gradation processing with the hint information included, it can be said that gradation processing is not necessarily performed on the specified location. Since the automatic gradation processing program learns by incorporating hint information in the learning process of the trained model used, it can be said that how the specified hint information is adopted depends on the tendency of the training data and hint information used for learning. This is a function that is completely different from the gradation processing in conventional image editing software, etc., which executes processing on specified locations, and can be said to be a distinctive feature of the present invention.

[0051] [Third embodiment] In the second embodiment, the automatic gradation processing device 10 has been described as an information processing device for providing a graphical user interface used when using the automatic gradation processing device 10, but the automatic gradation processing device 10 and automatic gradation processing program according to the present invention can also be incorporated into drawing creation software, image editing software, and web services (hereinafter also referred to as editing software) that provide services equivalent to these software from a server device to a terminal device. In this case, the editing software itself may have all of the configuration of the automatic gradation processing device 10, including the trained model, or the editing software in the terminal device may connect to and use the configuration of the automatic gradation processing device 10, including the trained model, provided on the server device via a communication network.

[0052] Editing software, including drawing software and image editing software, may have a function for creating image data to be subjected to gradation processing or a function for pasting image data to be subjected to gradation processing. It is common for the creation and pasting of image data to be subjected to gradation processing to be managed using multiple layers, with the image data to be subjected to gradation processing being created or pasted on a specific layer. Editing software also has a function for creating a closed area within a layer and performing specific processing only within that area, such as a fill function or a hatching function. By incorporating the automatic gradation processing program according to the present invention into editing software having such a configuration and functions, it is possible to provide editing software with even more functions.

[0053] For example, it is conceivable that gradation processing can be performed by an automatic gradation processing program on gradation processing target image data created based on the functions of editing software or pasted gradation processing target image data. By adopting such a configuration, it is possible to perform gradation processing on gradation processing target image data created by editing software or gradation processing target image data imported into editing software. Furthermore, it is possible to further edit the gradation-processed image data obtained based on the various functions of the editing software.

[0054] Furthermore, if the editing software supports management of multiple layers, it is possible to output the mask channel resulting from gradation processing based on the automatic gradation processing program to a specific layer. In other words, since the mask channel can be output to a layer different from the layer on which the original image data to be gradated is placed, the application of gradation processing can be determined by switching the layer display on or off. Furthermore, when the results of gradation processing are output as multiple mask channels, they can all be output separately to different layers. Furthermore, when different texture information is included for each applied tone, such as in tone processing, after the data is output to a layer in the editing software, it is possible to edit the tone by changing the texture tone to a tone type different from the tone type specified when output as a mask channel.

[0055] It is also possible to use the editing software's function for managing multiple layers to enable gradation processing using an automatic gradation processing program only on specific layers. Using this function, it is possible to write image data for each part of an image to be gradated on a different layer, perform gradation processing on each part, and ultimately obtain edited image data by overlaying all layers. For example, a layer for image data for people to be gradated and a layer for image data for the background to be gradated could be created, and each layer could be gradated before being overlaid to obtain a single image.

[0056] It is also possible to create a closed area within a layer and use a function to perform specific processing only within that area, allowing gradation processing by an automatic gradation processing program to be performed only within that closed area within the layer.If an image drawn within the closed area is considered to be a single piece of image data to be gradation processed, this can be applied without any particular technical difficulty.This function makes it possible to perform gradation processing only on the area selected by the user.

[0057] As described above, by applying the automatic gradation processing program of the present invention to existing editing software, it becomes possible to further edit the gradation-processed image represented by the gradation-processed image data that has been subjected to the automatic gradation processing based on the editing software, and it becomes possible to perform automatic gradation processing on a layer-by-layer basis or on a closed area-by-area basis within a layer, thereby making it possible to provide editing software with an automatic gradation processing function that is more convenient for users. Such editing software with an automatic gradation processing function can potentially contribute to improving the overall efficiency of work in computer graphics and animation production sites by performing some gradation processing tasks based on the automatic gradation processing program.

[0058] In the first to third embodiments, the generation of the trained model has been described as being performed by learning each type of gradation processing, such as shadow color processing, highlight processing, diagonal line processing, and tone processing, to obtain a trained model. However, it is not necessary to prepare one trained model for each type of gradation processing; instead, multiple trained models (preferably multiple trained models with different tendencies in the training data used for learning) may be prepared for each type of gradation processing, allowing the user to select which trained model to use for gradation processing. Furthermore, in the first to third embodiments, the generation of the trained model has been described as being performed by learning each type of gradation processing, such as shadow color processing, highlight processing, diagonal line processing, and tone processing, to obtain a trained model. However, it is also possible to train one model for all types of gradation processing, to obtain one trained model that can handle any gradation processing.

[0059] In the first to third embodiments, the automatic gradation processing device 10 has been described as including the hint information acquisition unit 12 and the light source position information acquisition unit 13, but this is not limited thereto, and the automatic gradation processing device 10 may have only one of the hint information acquisition unit 12 and the light source position information acquisition unit 13, or may have neither of the components. Even if neither of the components is included, if learning is performed so that gradation processing can be performed appropriately without providing hints when learning the trained model, it is possible to perform gradation processing appropriately.

[0060] In the first to third embodiments, the gradation processing in the trained model is described as outputting at least one mask channel for performing the gradation processing as a result, but the present invention is not limited to this, and the gradation processing result may be directly combined with the gradation processing target image data to obtain the gradation processed image data. In this case, the trained model is trained so as to directly combine the gradation processing result with the gradation processing target image data. [Explanation of symbols]

[0061] 10 Automatic gradation processing device 11 Gradation processing target image data acquisition unit 12 Hint information acquisition section 13 Light source position information acquisition section 14 Gradation processing section 15 Storage section 51 CPU 52 GPU 53 Memory 54 Storage 55 Input Device 56 Output Device 57 Communication equipment 58 Bus 60 Server equipment 70, 701 to 70N terminal equipment 80 Communication Network

Claims

1. a first acquisition means for acquiring image data of a person; a second acquisition means for acquiring information corresponding to at least one of a position and a direction of a virtual light source applied to the image data of the person; A processing means for acquiring data used for performing gradation processing on the acquired image data of the person according to at least one of the position and direction of the virtual light source from a trained machine learning model based on the acquired image data of the person and the acquired information; and the processing means inputs information of the image data of the person to be acquired and the acquired information into the trained machine learning model, and causes the trained machine learning model to output the data to be used; the gradation processing includes processing for changing gradation information of the image data of the person at a position corresponding to the data to be used; The acquired image data of the person is data of an image selected by a user and not learned by the machine learning model. Information processing device.

2. a first acquisition means for acquiring image data of a person; a second acquisition means for acquiring information corresponding to at least one of a position and a direction of a virtual light source applied to the image data of the person; A processing means for acquiring data in which gradation processing has been performed on the acquired image data of the person according to at least one of the position and direction of the virtual light source from a trained machine learning model based on the acquired image data of the person and the acquired information; and the processing means inputs information of the image data of the person to be acquired and the acquired information into the trained machine learning model, and outputs the data on which the gradation processing has been performed; The acquired image data of the person is data of an image selected by a user and not learned by the machine learning model. Information processing device.

3. The information processing apparatus according to claim 1 , wherein the acquired image data of the person is image data that is not affected by the virtual light source.

4. The information processing apparatus according to claim 1 , wherein the gradation processing is performed by rendering, on the acquired image data of the person, an effect of shading caused by the virtual light source according to a position or a direction of the virtual light source.

5. The information processing apparatus according to claim 1 , wherein the virtual light source is a light source different from a real light source for the acquired image data of the person.

6. The information processing device according to claim 1 , wherein the trained machine learning model is a trained machine learning model that processes shading in areas according to the content of image data that is the target of gradation processing.

7. The information processing device according to claim 1 , wherein the processing means executes the gradation processing by using the machine learning model without using a plurality of image data of the person.

8. The information processing device according to claim 1 , wherein the machine learning model includes a neural network that performs a convolution operation.

9. 9. The information processing device according to claim 1, wherein the gradation processing is a process of applying shading to the image data of the person in accordance with at least the position or direction of the light source, or a process of changing part of the shading in accordance with at least the position or direction of the light source, or a process of changing gradation information in accordance with at least the position or direction of the light source.

10. An information processing device described in any one of claims 1 to 8, wherein the gradation processing is a process of changing gradation information for image data of the person in order to express shadows or highlights that at least correspond to the position or direction of the light source.

11. The information processing device according to claim 1 , wherein the position and direction of the virtual light source are specified by a user.

12. a management means for managing a plurality of layers including at least a first layer and a second layer; 12. An information processing device according to claim 1 or any one of claims 3 to 11 dependent on claim 1, wherein the processing means outputs image data of the person to the first layer, outputs the data to be used to the second layer, and performs the modification process by combining the first and second layers.

13. a switching means for switching whether or not the data to be used is applied to the image data of the person; The information processing apparatus according to claim 1 , wherein the changing process is executed when the data to be used is applied to the image data of the person by the switching means.

14. An information processing program for causing a computer to function as each of the means according to any one of claims 1 to 13.

15. a first acquisition step of acquiring image data of a person; a second acquisition step of acquiring information corresponding to at least one of a position and a direction of a virtual light source applied to the image data of the person; a processing step of acquiring data used for performing gradation processing on the acquired image data of the person according to at least one of the position and direction of the virtual light source from a trained machine learning model based on the acquired image data of the person and the acquired information; and the processing step inputs information of the image data of the person to be acquired and the acquired information into the trained machine learning model, and outputs the data to be used; the gradation processing includes processing for changing gradation information of the image data of the person at a position corresponding to the data to be used; The acquired image data of the person is data of an image selected by a user and not learned by the machine learning model. Information processing methods.

16. a first acquisition step of acquiring image data of a person; a second acquisition step of acquiring information corresponding to at least one of a position and a direction of a virtual light source applied to the image data of the person; a processing step of acquiring data in which gradation processing has been performed on the acquired image data of the person according to at least one of the position and direction of the virtual light source from a trained machine learning model based on the acquired image data of the person and the acquired information; and the processing step inputs information of the image data of the person to be acquired and the acquired information into the trained machine learning model, and outputs the data on which the gradation processing has been performed; The acquired image data of the person is data of an image selected by a user and not learned by the machine learning model. Information processing methods.

17. The information processing method according to claim 15 or 16, wherein the acquired image data of the person is image data that is not affected by the virtual light source.

18. 18. The information processing method according to claim 15, wherein the gradation processing depicts, in the acquired image data of the person, an effect of shading caused by the virtual light source according to a position or a direction of the virtual light source.

19. 19. The information processing method according to claim 15, wherein the virtual light source is a light source different from a real light source for the acquired image data of the person.

20. 20. The information processing method according to claim 15, wherein the trained machine learning model is a trained machine learning model that processes shading in areas according to the content of image data that is the target of gradation processing.

21. The information processing method according to claim 15 , wherein the processing step uses the machine learning model to perform the gradation processing without using a plurality of image data of the person.

22. The information processing method according to claim 15 , wherein the machine learning model includes a neural network that performs a convolution operation.

23. 23. An information processing method according to any one of claims 15 to 22, wherein the gradation processing is a process of applying shading to the image data of the person in accordance with at least the position or direction of the light source, or a process of changing part of the shading in accordance with at least the position or direction of the light source, or a process of changing gradation information in accordance with at least the position or direction of the light source.

24. An information processing method described in any of claims 15 to 22, wherein the gradation processing is a process of changing gradation information for image data of the person in order to express shadows or highlights that at least correspond to the position or direction of the light source.

25. 25. The information processing method according to claim 15, wherein the position and direction of the virtual light source are specified by a user.

26. A plurality of layers including at least a first and a second layer are managed; 26. An information processing method according to claim 15 or any one of claims 17 to 25 dependent on claim 15, wherein the processing step outputs image data of the person to the first layer, outputs the data to be used to the second layer, and performs the modifying process by combining the first and second layers.

27. a switching step of switching whether or not the data to be used is applied to the image data of the person; The information processing method according to claim 15 or 26, wherein the changing process is executed when the data to be used is applied to the image data of the person in the switching step.

Images