Learning device and inference device

Abstract

To provide a learning device capable of preparing higher quality data, and certainly suppressing reduction of learning efficiency of a machine learning model, even when performing learning by performing augmentation of teacher data to become a source, in data processing by machine learning.SOLUTION: A learning device comprises: an input unit 110 which receives input of teacher data to become a source; data processing unit 113 which creates the teacher data after processing processed on the basis of the teacher data to become the source; a learning process execution unit 101 for executing processing which learns a setting value in a machine learning model 111 of a learning object, on the basis of the teacher data including at least the teacher data after processing, by using the machine learning model 111 of the learning object; and a storage unit 12 which stores the setting value. By this configuration, the learning device is capable of preparing higher quality data, and suppressing reduction of learning efficiency of the machine learning model 111, even when performing the learning by performing augmentation of the teacher data to become the source, in data processing by machine learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a learning device and an inference device that use a machine learning model, and more particularly to a learning device and an inference device that use a machine learning model for image processing. [Background technology]

[0002] Conventionally, a data processing method using machine learning has been known in which a data set serving as training data is provided to a computer program, and the program parameters are learned to generate a trained model capable of performing data processing on any data.

[0003] For example, in the process of "input image (teacher data) → learning program → output image (teacher data)" using machine learning, the parameters of the learning program are calculated so that the error between the input image and the output image is minimized, and a "trained model" is generated. Then, by using this "trained model", it is possible to generate an output image by inferring the output image from the input image, which is "input image (any data: for example, a low-resolution image) → trained model → output image (inference data: for example, a high-resolution image)".

[0004] In recent years, machine learning using neural networks has been applied to many fields. Especially in the fields of image and voice recognition, deep learning using neural networks in a multi-layer structure has demonstrated high recognition accuracy. In multi-layered deep learning, image processing is performed using a convolutional neural network (CNN), which uses multiple convolutional layers and pooling layers to extract input features.

[0005] Examples of image processing using neural networks include a super-resolution device that increases the resolution of signals (see, for example, Patent Document 1) and a diagnosis support device that facilitates grasping differences in diseased areas and provides highly accurate diagnosis support (see, for example, Patent Document 2). Also disclosed are a program, an image processing method, and an image processing device that avoid degradation due to free deformation of a digital image (see, for example, Patent Document 3). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-27557 A [Patent Document 2] JP 2018-38789 A [Patent Document 3] Patent No. 6570164 Summary of the Invention [Problem to be solved by the invention]

[0007] In machine learning, the quality and quantity of training data directly affect the performance of the machine learning model. Since it is very laborious to prepare a large amount of high-quality training data set, data augmentation is commonly used to obtain high performance with a small amount of training data.

[0008] For example, in order to efficiently learn from a small amount of training image data, editing such as enlargement / reduction, rotation, rhombic / trapezoidal transformation, shift, color shift, sharpness / unsharpness, etc. is applied to the training data to augment the data volume (data augmentation). Of these, enlargement / reduction, rotation, rhombic / trapezoidal transformation, and shift cause blurring and aliasing when small image data is transformed as is, so when data augmentation is performed, new augmented data with a certain degree of blurring and noise is always generated. For this reason, in conventional data augmentation, fine high-frequency data contained in images with good image quality is cut off, so learning using augmented data with such degraded image quality causes the problem that the performance of the machine learning model deteriorates.

[0009] Furthermore, the above Patent Documents 1 to 3 do not disclose anything about preparing a large amount of high-quality learning data from a small amount of learning data.

[0010] The present invention has been made in view of the above problems, and aims to provide a learning device that can reliably prevent a decrease in the learning efficiency of a machine learning model by preparing higher quality data even when learning is performed by padding the original teacher data in data processing by machine learning. It also aims to provide an inference device using this machine learning model. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, the present invention provides a learning device that learns setting values ​​in a machine learning model based on teacher data, comprising an input unit that accepts input of original teacher data, a data processing unit that creates processed teacher data processed based on the original teacher data, a machine learning model to be learned, a learning process execution unit that uses the machine learning model to execute a process of learning setting values ​​in the machine learning model to be learned based on teacher data including at least the processed teacher data, and a memory unit that stores the setting values, wherein the data processing unit identifies a target resolution of the processed teacher data, obtains a provisional reference image that is a predetermined multiple of an image having the target resolution based on the original teacher data, accepts editing including rotation, movement or deformation of the provisional reference image, samples the provisional reference image after editing at a sampling rate higher than the predetermined multiple, and converts the sampled image to the size of an image having the target resolution, which is used as the processed teacher data.

[0012] In this learning device, it is preferable that the learning device further has an outpainting model that automatically generates the periphery of the original teacher data in advance to create larger teacher data, and the data processing unit converts the teacher data created by the outpainting model into the processed teacher data.

[0013] In this learning device, it is preferable that the data processing unit determines an oversampling rate for the provisional reference image after editing at a rate higher than the specified rate in accordance with the editing, performs oversampling at the determined rate, and converts the image obtained by oversampling into the processed teacher data.

[0014] In this learning device, it is preferable that the learning device further has a super-resolution model that creates a super-resolution image of the specified resolution based on the original teacher data when the resolution of the original teacher data is less than the specified multiple of the target resolution, and that the data processing unit uses the super-resolution image created by the super-resolution model as the provisional reference image.

[0015] In order to achieve the above-mentioned object, the present invention is a learning device characterized in that at least a part of a machine learning model in which the setting value is learned in a learning process execution unit of the learning device is used as a loss function.

[0016] In addition, in order to achieve the above-mentioned object, the present invention provides an inference device that executes a predetermined inference process on target data using a machine learning model, comprising: an input unit that accepts input of the target data; a machine learning model to which the target data is input from the input unit; and an inference process execution unit that executes a predetermined inference process on the target data using the machine learning model that executes the inference process, wherein the machine learning model is a machine learning model in which the setting value has been learned in the learning process execution unit of the learning device.

[0017] The present invention also provides a computer program for causing a computer to operate as the above-described learning device or inference device.

[0018] In addition, in order to achieve the above-mentioned object, the present invention provides a learning method for learning setting values ​​in a machine learning model based on teacher data, comprising: an input step for accepting input of original teacher data; a data processing step for creating processed teacher data processed based on the original teacher data; a machine learning model to be learned; a learning process execution step for executing a process to learn setting values ​​in the machine learning model to be learned based on teacher data including at least the processed teacher data using the machine learning model; and a storage step for storing the setting values. In the data processing step, a target resolution of the processed teacher data is identified, a provisional reference image having a size that is a predetermined multiple of an image having the target resolution is obtained based on the original teacher data, editing including rotation, movement or deformation of the provisional reference image is accepted, the provisional reference image after editing is sampled at a sampling rate that is higher than the predetermined multiple, and the sampled image is converted to the size of an image having the target resolution, which is used as the processed teacher data.

[0019] In addition, in order to achieve the above-mentioned object, the present invention provides an inference method for executing a predetermined inference process on target data using a machine learning model, comprising: an input step for accepting input of the target data; a machine learning model to which the target data is input from the input step; and an inference process execution step for executing a predetermined inference process on the target data using the machine learning model that executes the inference process, wherein the machine learning model is a machine learning model that has learned the setting value in the learning process execution step. Effect of the Invention

[0020] The learning device according to the present invention includes an input unit that accepts input of original teacher data, a data processing unit that creates processed teacher data processed based on the original teacher data, a learning process execution unit that executes a process of learning setting values ​​in the machine learning model to be learned based on teacher data including at least the processed teacher data using the machine learning model to be learned, and a storage unit that stores the setting values. The data processing unit (1) specifies a target resolution of the processed teacher data, (2) acquires a provisional reference image having a size that is a predetermined multiple of an image having the target resolution based on the original teacher data, (3) accepts editing including rotation, movement, or deformation of the provisional reference image, (4) samples the provisional reference image after editing at a sampling rate that is higher than the predetermined multiple, and (5) converts the sampled image to the size of an image having the target resolution to obtain processed teacher data. With this configuration, the learning device according to the present invention can prepare higher quality data and reliably suppress a decrease in the learning efficiency of the machine learning model even when learning is performed by padding the original teacher data in data processing by machine learning. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a functional block diagram of the image processing device. [Diagram 3] 13 is a diagram showing an example of creation of processed teacher data during a learning operation of the image processing device. FIG. [Figure 4] 11A and 11B are diagrams for explaining data padding during a learning operation of the image processing device according to the first embodiment. [Diagram 5] 4 is an image diagram of a data set given during a learning operation in the image processing device according to the first embodiment; FIG. [Figure 6] 13A is a diagram for explaining a loss function used during a learning operation in a modified example of the image processing device of the above embodiment, and FIG. 13B is a diagram showing an example of a synthetic image generated in the modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] (Embodiment) An image processing device according to an embodiment of the present invention will be described with reference to Figures 1 to 5. In this embodiment, the image processing device functions as at least one of a learning device that learns setting values ​​in a machine learning model based on predetermined learning image data, and an inference device that executes a predetermined inference process on target image data using a machine learning model.

[0023] First, each processing unit of the image processing device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the image processing device 1 includes a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and a reading unit 16. Note that, although the image processing device 1 and the operation in the image processing device 1 will be described below as a single server computer, the image processing device 1 may be configured to distribute processing among multiple computers.

[0024] The control unit 10 uses a processor such as a CPU and a memory to control the components of the device to realize various functions. The image processing unit 11 uses a processor such as a GPU or a dedicated circuit and a memory to execute image processing in response to control instructions from the control unit 10. The control unit 10 and the image processing unit 11 may be configured as a single piece of hardware (SoC: System on a Chip) that integrates processors such as a CPU and a GPU, memory, and further a storage unit 12 and a communication unit 13.

[0025] The storage unit 12 uses a hard disk or a flash memory. The storage unit 12 stores an image processing program 1P, a machine learning library 1L that functions as a machine learning model (e.g., CNN), a super-resolution model 1M, and an outpainting model 1N. The storage unit 12 also stores definition data that defines the machine learning model, parameters including setting values ​​in the trained machine learning model, and the like. The image processing unit 11 outputs a digital image with a resolution higher than the original resolution based on the super-resolution model 1M stored in the storage unit 12. The outpainting model 1N and the super-resolution model 1M can be generated by a machine learning model.

[0026] The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet, etc. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

[0027] The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, etc. The display unit 14 is capable of displaying an image by processing in the image processing unit 11 according to an instruction from the control unit 10.

[0028] The operation unit 15 includes a user interface such as a keyboard or a mouse. A physical button provided on the housing may be used. The operation unit 15 may also use a software button or the like displayed on the display unit 14. The operation unit 15 notifies the control unit 10 of operation information by the user. Specifically, when editing the original teacher data, the operation unit 15 accepts settings of various transformation methods for the teacher data from the user, such as enlargement / reduction, rotation, shifting in the left / right direction, and changes in brightness / saturation and contrast.

[0029] The reading unit 16 can use, for example, a disk drive to read the image processing program 2P, the machine learning library 2L, the super-resolution model 2M, and the outpainting model 2N stored in the recording medium 2 using an optical disk or the like. The image processing program 1P, the machine learning library 1L, the super-resolution model 1M, and the outpainting model 1N stored in the storage unit 12 may be the image processing program 2P, the machine learning library 2L, the super-resolution model 2M, and the outpainting model 2N read by the reading unit 16 from the recording medium 2 and copied by the control unit 10 to the storage unit 12.

[0030] Next, the image processing function of the image processing device 1 will be described with reference to Fig. 2. The control unit 10 of the image processing device 1 includes a learning process execution unit 101 and an inference process execution unit 102. The learning process execution unit 101 functions as a machine learning model (machine learning engine) based on the machine learning library 1L, definition data, and parameter information stored in the storage unit 12. That is, the learning process execution unit 101 executes a process of learning setting values ​​(parameters, etc.) in the machine learning model to be learned based on learning image data or learning text data using the machine learning model to be learned. This learning of setting values ​​is a process of minimizing the difference between input data and answer data, for example, by updating parameters using a mini-batch gradient descent method. In addition, the learning process execution unit 101 also functions as an image processing execution unit that edits image data, which is input data, based on an operation using the operation unit 15.

[0031] The inference processing execution unit 102 performs image processing based on the image processing program 1P stored in the storage unit 12. That is, the inference processing execution unit 102 executes a predetermined inference processing on the input target data (target image data and inference target text data) using a machine learning model. The inference processing execution unit 102 also functions as an image processing execution unit that inputs image data, which is input data, to the input unit 110 based on a user's operation using the operation unit 15.

[0032] The image processing unit 11 includes an input unit 110, a machine learning model 111, a generator 111a and a discriminator 111b as the machine learning model 111, an output unit 112, a processing unit 113, and a cutout unit 114. For example, a machine learning model such as CNN learns using given teacher data, and definition data and parameter information functioning as the generator 111a and definition data and parameter information functioning as the discriminator 111b are stored in the storage unit 12 to create a model.

[0033] Typical examples of learning for machine learning models include classification tasks (input teacher image data → machine learning model → output label ⇔ teacher label), image conversion (input teacher image data → machine learning model → output image data ⇔ output teacher image data), and image generation (input teacher text data → machine learning model → output image data ⇔ output teacher image data). Note that → here indicates the flow of data, and ⇔ indicates a comparison during learning.

[0034] The input unit 110 accepts input of teacher (learning) image data, classification labels, text data, or target image data and text data to be inferred (see 12a in FIG. 2). These data are recorded and stored in the storage unit 12.

[0035] The data processing unit 113 performs processing for data augmentation of the teacher data. Specifically, the data processing unit 113 performs the following processing: (1) specifies a target resolution of the teacher data after processing (after augmentation), (2) acquires a provisional reference image having a size that is a predetermined multiple of the image having the target resolution based on the original teacher data input to the input unit 110, (3) accepts editing including rotation, movement, or deformation of the provisional reference image, (4) samples the provisional reference image after editing at a sampling rate that is higher than the predetermined multiple, and (5) converts the sampled image to the size of the image having the target resolution, and uses the resultant image as the processed teacher data.

[0036] 3, (1) a target resolution (here, same size) of processed teacher data 304 for original teacher data 301 is specified. Next, (2) a provisional reference image 302 is prepared at a predetermined multiple of the target resolution (here, twice, i.e., 224*224 pixels) for the original teacher data 301 (here, 112*112 pixels). Note that, to obtain the provisional reference image 302, for example, image data stored in advance in the storage unit 12 or a super-resolution image with a higher resolution than the original teacher data is generated by the image processing unit 11 based on the super-resolution model 1M and temporarily stored in the storage unit 12 or the like.

[0037] Next, (3) editing (here, rotation) of the provisional reference image 302 is accepted, and (4) the edited provisional reference image 303 is sampled at a sampling rate setting (here, 3x) higher than the predetermined factor (here, 2x).Then, (5) the sampled image is reduced to the target resolution (here, the image that extends beyond the area due to rotation is cut out and reduced to 1 / 3) and converted to the size of an image having the target resolution, which is used as processed teacher data 304.

[0038] In this way, by preparing an image (provisional reference image) that is larger (for example, twice the size) than the target image (image with the target resolution), rotating and moving this image at a high sampling rate, and then reducing it to the target size, the image will not be degraded.

[0039] In the description of this embodiment, the target resolution is the magnification of enlargement from the original teacher data 301 to the processed teacher data 304, and the control unit 10 may initially set the same resolution as the unprocessed digital image (original) as the target resolution (same size). Also, if the editing includes enlargement / reduction of data, the resolution of the image data after enlargement / reduction may be set as the target resolution (1 or more, or 1 or less). Furthermore, if the editing includes rotation / transformation, the sampling rate may be set according to the angle of rotation.

[0040] The target resolution does not necessarily have to be exactly the same as the resolution after processing, and may be set to a value close to the resolution. In that case, the resolution will be degraded by the amount that is insufficient. For example, if the processed training data is roughly close to the size of the original training data, the target resolution is 1x, the provisional reference image is 2x, and the oversampling is 3x. For example, if the processed training data is to be slightly larger than the original training data, the target resolution is 1.5x, the provisional reference image is 3x, and the oversampling is 5x. This value may be set for each image or for each pixel.

[0041] The data processing unit 113 may determine the oversampling rate for the provisional reference image after editing to be higher than a predetermined rate in accordance with the editing, perform oversampling at the determined rate, and convert the image obtained by oversampling into processed teacher data. Furthermore, the settings for these data processing operations may be recorded and stored in the storage unit 12.

[0042] In the example in Figure 3, a twice-sized image is prepared and then reduced to 1 / 2, but when rotation is performed as part of the editing process, the sampling rate differs between the vertical and horizontal directions and the diagonal direction, so the number of samples decreases relative to the sampling rate in the vertical and horizontal directions, while the number of samples exceeds the sampling rate in the diagonal direction. This causes aliasing noise, especially in the diagonal direction. To suppress this noise, rotate the image with an increased sampling rate and then reduce it to the target size, which will prevent this noise from occurring. For example, an image twice the target size is enlarged to 1.5 times its size while rotating it, and then reduced to 1 / 3 (3x oversampling).

[0043] In addition, in the case of image deformation, each pixel becomes larger or smaller, so adjustments may be made so that each pixel has a provisional reference image that is larger than the target resolution and has a higher sampling rate. It is also possible to set a provisional reference image with a sufficiently high resolution and a sufficiently high sampling rate uniformly so that all pixels satisfy the conditions, without considering each pixel individually. For example, a provisional reference image with a high resolution that can accommodate the enlarged portion and a high sampling rate that can accommodate the reduced or rotated portion may be set.

[0044] Regarding editing, the control unit 10 of the image processing device 1 performs editing on the image data based on the image processing program 1P stored in the storage unit 12. In particular, the control unit 10 accepts user operations via the operation unit 15 based on the image processing program 1P, and realizes editing on the image data. Note that editing here refers to performing processing such as enlargement / reduction, rotation, rhombus / trapezoid transformation, shift, color shift, sharpness / unsharpness, etc. on the provisional reference image via the operation unit 15 in order to pad the learning data.

[0045] Next, data padding will be described with reference to Figures 4 and 5. As shown in Figure 4, the data processing unit 113 performs various processes on the original teacher data 401 to generate processed teacher data 402 in which the apparent number of teacher data used during learning is padded. In this case, by using the data processing unit 113 according to this embodiment, deterioration of the processed teacher data 402 is effectively suppressed, and the quantity and quality of the padded teacher data are improved, resulting in a dramatic improvement in the performance of the learning model.

[0046] Also, as shown in Fig. 5 (II), for example, in the past, the original teacher data was directly edited to pad the image, and even if the image remained the same size, the input teacher image data 503 and output teacher image data 504 after padding deteriorated when rotated or moved (shifted) by a fraction of a pixel. This is because the image has parts such as boundaries that change suddenly and do not show waveform properties, and when waveform processing such as image deformation is performed, the signal of those parts is lost. As a result, even if the teacher data is padded, the performance of the learning model deteriorates because the input teacher image data 503 and output teacher image data 504 themselves have deteriorated.

[0047] On the other hand, as shown in Figure 5 (I), by using the data processing unit 113 of this embodiment, the quality of the input side teacher image data 501 and the output side teacher image data 502 after padding is maintained, and high performance of the machine learning model can be achieved.

[0048] On the other hand, when the image processing device 1 functions as an inference device, it is an inference device that executes a predetermined inference process on target data using a machine learning model, and is equipped with an input unit 110 that accepts input of the target data, a machine learning model 111 to which the target data is input from the input unit 110, and an inference process execution unit 102 that executes a predetermined inference process on the target data using the machine learning model 111 that executes the inference process, and the machine learning model 111 is a machine learning model 111 whose setting values ​​have been learned in the learning process execution unit 101 of the above-mentioned learning device.

[0049] When a trained model is used, the machine learning model 111 performs data optimization processing (e.g., increasing resolution, classifying, removing noise, and generating images) based on the parameters that have already been trained. When the machine learning model 111 during inference is a CNN, it may include multiple stages of convolution layers and pooling layers defined by definition data, and a fully connected layer, extracting the feature amount of image data and performing image processing processing based on the extracted feature amount.

[0050] Specifically, during inference by the machine learning model 111, image data to be inferred and text data to be inferred can be input to the machine learning model 111, and post-inference image data can be obtained as an output from the output unit 112. The image data here is image data expressed using YCbCr or RGB. Furthermore, the output is not limited to image data, and in the case of classification, a specified class is output. The output unit 112 outputs the image data and the classification class to the storage unit 12. Note that the output data may be rendered as an image in the image processing unit 11 and output to the display unit 14.

[0051] Next, outpainting of image data in the learning device according to the present embodiment will be described. Conventionally, padding of image data basically involves increasing the amount of image data of the same size, so when enlargement, reduction, or rotation is applied, the image is cut or holes are filled in the cropping unit 114 to return it to its original size. In conventional image padding operations, the unnaturalness of the image is reduced and degradation of image quality is suppressed by performing calculations using the pixel values ​​of surrounding pixels, such as filling holes created by rotation or reduction of the image after transformation with nearby colors.

[0052] On the other hand, the image processing unit 11 according to the present embodiment automatically generates the periphery of the original teacher data in advance based on the outpainting model 1N stored in the storage unit 12 to create larger teacher data, and the data processing unit 113 converts the teacher data created by the outpainting model 1N into processed teacher data. That is, in the conventional method, holes generated in the processed image data are filled with nearby data or filled with black, in which case an unnatural image is generated. However, the image processing device 1 according to the present embodiment uses the outpainting model 1N to automatically generate and enlarge naturally continuous images around the teacher image data in advance, thereby preventing holes from being generated. Then, by using this in combination with the processing method of padding the teacher data by the data processing unit 113 described above, it becomes possible to obtain processed teacher data of higher quality.

[0053] As described above, when the image processing device 1 functions as a learning device that learns the setting values ​​in the machine learning model 111 based on the teacher data, the learning device includes an input unit 110 that accepts input of the original teacher data, a data processing unit 113 that creates processed teacher data processed based on the original teacher data, a learning process execution unit 101 for executing a process of learning the setting values ​​in the machine learning model 111 to be learned based on the teacher data including at least the processed teacher data using the machine learning model 111 to be learned, and a storage unit 12 that stores the setting values. The data processing unit 113 (1) specifies the target resolution of the processed teacher data, (2) acquires a provisional reference image having a size that is a predetermined multiple of the image having the target resolution based on the original teacher data, (3) accepts editing including rotation, movement, or deformation of the provisional reference image, (4) samples the provisional reference image after editing at a sampling rate that is higher than the predetermined multiple, and (5) converts the sampled image to the size of the image having the target resolution to obtain the processed teacher data. With this configuration, in a learning device, even when the original training data is augmented during data processing using machine learning, higher quality data can be prepared, thereby reliably preventing a decrease in the learning efficiency of the machine learning model.

[0054] That is, when the image processing device 1 functions as a learning device, the data processing unit 113 performs deformation and the like using a sufficiently large provisional reference image such as super-resolution when padding the original teacher data, and obtains the processed teacher data after using oversampling at a large magnification. This makes it possible to maintain a suitable image quality in the processed teacher data after deformation, and also improves the performance of the machine learning model learned with that data. In addition, by using this machine learning model, when the image processing device 1 functions as an inference device, it is possible to generate and classify more highly accurate post-inference image data.

[0055] (Modification) A modified example of the image processing device 1 according to the embodiment of the present invention will be described with reference to Fig. 6. In this modified example, a machine learning model 111 trained by a learning process execution unit 101 is used as a loss function, and the loss function is, for example, a content loss and a style loss. For example, as shown in Fig. 6(a), style transfer is known, which is a type of CNN that takes two inputs, a content image 601 that ensures a structure and a style image 602 that ensures a style, and outputs a synthetic image that has both the structure of the former and the style of the latter.

[0056] In this style transfer, first, a content image 601 and a style image 602 are input to the left and right trained VGG (Visual Geometry Group) networks, respectively, and a vector representing the resemblance of each image is extracted from the feature map along the way to define a loss function 603. Then, optimization is performed by minimizing the loss. For the trained VGG, which is a type of machine learning model used here, if the content image 601 and the style image 602 are learned using undegraded images generated by the data processing unit 113 in the above embodiment, it is expected that a composite image 604 combining content and style as exemplified in FIG. 6(b) that captures the characteristics of high-frequency components (high resolution) can be generated at the time optimization is completed.

[0057] Of the hardware configuration of the image processing device 1 according to this embodiment, the communication unit 13, the display unit 14, and the reading unit 16 are not essential. The communication unit 13 may not be used after it has been used once, for example, when the image processing program 1P and the machine learning library 1L stored in the storage unit 12 are obtained from an external server device. Similarly, the reading unit 16 may not be used after the image processing program 1P and the machine learning library 1L are read out and obtained from a storage medium. The communication unit 13 and the reading unit 16 may be the same device using serial communication such as USB.

[0058] The image processing device 1 may be configured as a Web server to provide the function of the above-mentioned machine learning model 111 to a Web client device including a display unit and a communication unit. In this case, the communication unit 13 is used to receive requests from the Web client device and transmit processing results.

[0059] The error used during learning may be an appropriate function such as squared error, absolute error, or cross entropy error depending on the input and output data and the learning purpose. For example, if the output is classification, cross entropy error is used. Flexible operation can be applied, such as using other criteria instead of using an error function. This error function itself may be evaluated using an external machine learning model.

[0060] The present invention is not limited to the configuration of the above embodiment, and various modifications are possible within the scope of the invention. In order to achieve the object of the present invention, the present invention can be realized as an image processing method (learning method and inference method) having characteristic constituent means included in an image processing device (learning device and inference device) as steps, or as a program including those characteristic steps. The program can be stored in a ROM or the like, or can be distributed via a recording medium such as a USB memory or a communication network.

[0061] The present invention can also be realized as a computer system that transmits input data to an image processing device or a computer program and receives and uses output data from the image processing device or the computer program. This system is a processing system that uses data obtained from a machine learning model that has been trained by the above-mentioned processing, and can provide various services. The device used in this system is an image processing device equipped with a display unit and a communication unit, or an information processing device that can transmit and receive information to and from a computer, such as a so-called PC, a smartphone, a mobile terminal, or a game device. [Explanation of symbols]

[0062] 1 Image processing device (learning device and inference device) 10 Control section 12 Memory unit (learning result memory unit) 15 Control section 101 Learning processing execution unit 102 Inference processing execution unit 110 Input section 111 Machine Learning Models 111a generator 111b Discriminator 112 Output section 113 Data Processing Department 114 Cutout part 301,401 Original training data 302 Provisional Reference Image 303 Edited provisional reference image 304,402 Processed training data

Claims

1. A learning device that learns a setting value in a machine learning model based on teacher data, An input unit that accepts input of original teacher data; A data processing unit that creates processed teacher data based on the original teacher data; a learning process execution unit for executing a process of learning a setting value in the machine learning model to be learned based on teacher data including at least the processed teacher data using the machine learning model; A storage unit that stores the set value, In the data processing unit, Specifying a target resolution of the processed teacher data; Obtaining a provisional reference image having a size that is a predetermined multiple of an image having the target resolution based on the original teacher data; Accepting edits to the temporary reference image, including rotation, translation, or deformation; sampling the provisional reference image after editing at a sampling rate higher than the predetermined rate; A learning device characterized in that the sampled image is converted to the size of an image having the target resolution, and the processed teacher data is used.

2. The learning device further comprises: An outpainting model is provided to automatically generate the periphery of the original teacher data in advance to create larger teacher data; 2. The learning device according to claim 1, wherein the data processing unit converts the teacher data created by the outpainting model into the processed teacher data.

3. The data processing unit includes: determining an oversampling rate for the provisional reference image after editing to be a rate higher than the predetermined rate in accordance with the editing; Oversampling is performed at the determined rate, 2. The learning device according to claim 1, wherein the image obtained by oversampling is converted into the processed teacher data.

4. The learning device further comprises: a super-resolution model that creates a super-resolution image having a resolution of the predetermined resolution based on the original teacher data when the resolution of the original teacher data is less than the predetermined resolution of the target resolution; 2. The learning device according to claim 1, wherein the data processing unit uses the super-resolution image created by the super-resolution model as the provisional reference image.

5. 5. A learning device, comprising: a learning process execution unit of the learning device according to claim 1, wherein at least a part of a machine learning model that has been trained using the setting value is used as a loss function.

6. An inference device that executes a predetermined inference process on target data using a machine learning model, an input unit that accepts input of the target data; A machine learning model to which target data is input from the input unit; An inference processing execution unit that executes a predetermined inference processing on target data using a machine learning model that executes the inference processing, The inference device, characterized in that the machine learning model is a machine learning model in which the setting value is learned in a learning process execution unit of the learning device described in any one of claims 1 to 4.

7. A computer program causing a computer to operate as the learning device according to any one of claims 1 to 4.

8. A computer program for causing a computer to operate as the learning device according to claim 5.

9. A computer program for causing a computer to operate as the inference device according to claim 6.

10. A learning method for learning a setting value in a machine learning model based on teacher data, comprising: An input step for accepting input of original training data; A data processing step of creating processed teacher data that is processed based on the original teacher data; a learning process execution step for executing a process of learning a setting value in the machine learning model to be learned based on teacher data including at least the processed teacher data by using the machine learning model; A storage step of storing the set value, In the data processing step, Specifying a target resolution of the processed teacher data; Obtaining a provisional reference image having a size that is a predetermined multiple of an image having the target resolution based on the original teacher data; Accepting edits to the temporary reference image, including rotation, translation, or deformation; sampling the provisional reference image after editing at a sampling rate higher than the predetermined rate; A learning method characterized in that the sampled image is converted to the size of an image having the target resolution, and the processed teacher data is used.

11. An inference method for executing a predetermined inference process on target data using a machine learning model, comprising: an input step of accepting an input of the target data; A machine learning model to which target data is input from the input step; An inference processing execution step of executing a predetermined inference processing on the target data using a machine learning model that executes the inference processing, The inference method, wherein the machine learning model is a machine learning model that has been trained with the setting value in the learning process execution step described in claim 10.

Images