Information processing apparatus, information processing method, and program

Abstract

To generate images with high image quality.SOLUTION: An information processing apparatus divides an input image into a plurality of subdivided images and selects an appropriate image generator for each subdivided image from among a plurality of image generators based on the subdivided images of the input image. The information processing apparatus generates a plurality of generated images from the input image or the subdivided images of the input image using the plurality of selected image generators and combines two or more of the generated images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to information processing technology for generating images.

Background Art

[0002] There is information processing technology for generating high-quality images by inputting low-quality images including high noise, blur, etc. into an image generator such as a machine learning model prepared in advance. Depending on differences in components, learning data, etc., there may be differences in the characteristics, hue, etc. of the generated images in the image generator. Therefore, even if the same image is input to a plurality of image generators with different components, learning data, etc., the image quality of the finally output images will be different for each image generator. Also, for example, when it is desired to generate an image with a quality of the user's personal preference, that is, an image with a quality evaluated as good by the user's subjective evaluation, it is conceivable that the user checks the performance of the image generator in advance and uses an image generator that generates an image evaluated as good by subjective evaluation. However, in many cases, a single image generator cannot generate an image with the quality of the user's preference, and similarly, in many cases, it cannot generate a good high-quality image from a low-quality image. That is, depending on the image generator used, there are spatial characteristics and hues in the images that the generator is good or bad at, and therefore, an image with insufficient image quality may be generated.

[0003] Patent Document 1 discloses a technique for generating an image in which noise, etc. is appropriately removed by processing and synthesizing a single image with a plurality of image generators. Patent Document 1 gives examples of images captured by a magnetic resonance imaging device, an ultrasonic diagnostic device, etc. as input images, and in those images, the noise levels added vary depending on the spatial regions within the image. Therefore, when noise removal is performed with a single image generator, images with different image qualities are generated depending on the region. The technique of Patent Document 1 enables high image quality throughout the image by using a plurality of image generators that are good at generating images in which noise, etc. with different noise levels has been removed, and synthesizing the plurality of images generated thereby according to a predetermined region pattern.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology disclosed in Patent Document 1 is premised on the fact that the noise levels added by the spatial regions in the image are different. Therefore, if the noise level corresponding to each region is not grasped, a high-quality image cannot be generated. On the other hand, in the image acquired by shooting, it is indefinite which region in the image has what kind of features and hues, and it is also indefinite what kind of brightness and what kind of noise level each region in the image has. Therefore, even if the technology disclosed in Patent Document 1 is applied to the captured image, there is a high possibility that a high-quality image as desired cannot be generated. In addition, the image quality of the images generated by individual image generators is not necessarily the preferred image quality desired by the user.

[0006] Therefore, an object of the present invention is to enable the generation of an image with a desired image quality.

Means for Solving the Problems

[0007] The information processing apparatus of the present invention includes a dividing means for dividing an input image into a plurality of divided images, a selecting means for selecting, from among a plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image, an image generating means for generating a plurality of generated images from the input image or the divided images of the input image using the plurality of image generators selected by the selecting means, and a synthesizing means for synthesizing two or more of the generated images.

Effects of the Invention

[0008] According to the present invention, an image with a desired image quality can be generated.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each of the embodiments described hereinafter does not limit the present invention, and not all of the plurality of features described in this embodiment are essential for the solution means of the present invention, and those plurality of features may be arbitrarily combined. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Also, a configuration may be formed by appropriately combining a part of each of the embodiments described later. In the following embodiments, the same or similar configurations and processing steps are given the same reference numerals, and duplicate explanations are omitted.

[0011] <First Embodiment> In this embodiment, information processing is described in which a high-quality image can be generated by inputting a low-quality image into an image generator such as a machine learning model. As the low-quality image, for example, an image including high noise or blur is assumed. In this embodiment, information processing for generating a high-quality image from which noise has been removed from a high-noise image will be described as an example.

[0012] FIG. 1 is a diagram showing an application example of the information processing apparatus according to this embodiment, and shows a configuration example of an information processing system having an image processing apparatus 130, a dataset creation apparatus 110, and a data storage unit 101. The information processing apparatus according to this embodiment may include only the image processing apparatus 130, or may include the dataset creation apparatus 110 in addition to the image processing apparatus 130, and may further include the data storage unit 101.

[0013] The data storage unit 101 stores one or more sets of image sets that are a set of a high-quality learning reference image with low noise (or no noise) and a low-quality learning image that is an image of the same scene as the learning reference image and has high noise. It is assumed that the set of the learning reference image and the low-quality learning image is prepared and stored in advance. Note that the data storage unit 101 may be provided outside, such as a server (not shown), or may exist in a local environment as shown in FIG. 1.

[0014] The image processing device 130 includes an image generation unit 120, and the image generation unit 120 is composed of a plurality of image generators 1 to N. The image generators 1 to N included in the image generation unit 120 are each composed of a machine learning model such as a neural network. It is assumed that the machine learning model is a model that has been pre-trained to generate a high-quality image with noise removed from a high-noise image. The machine learning model may use a CNN model (Convolutional Neural Network) having a convolutional layer, or a Transformer model having an attention mechanism. Also, it is assumed that the machine learning model is learned by changing the learning parameters and the breakdown of the learning data used for learning so as to obtain different image generation results. Note that the image generators 1 to N are not limited to those using a machine learning model, and for example, may be those that generate an image with noise removed from a high-noise image by image filter processing or the like. As the image filter, for example, a plurality of Gaussian filters with different parameters of Gaussian σ may be used.

[0015] In the case of this embodiment, it is assumed that the machine learning models of the image generators 1 to N are models in which pre-training for generating a high-quality image with noise removed from a high-noise image is individually performed for each. Therefore, even when the same image is input to the image generators 1 to N, there is a possibility that images with different image qualities are generated by those image generators 1 to N. In the following description, the images respectively generated by the image generators 1 to N of the image generation unit 120 are referred to as "generated images".

[0016] The image set acquisition unit 111 of the dataset creation device 110 acquires one or more image sets, which are sets of a learning reference image and a low-quality learning image, from the data storage unit 101. The generated image acquisition unit 112 then inputs the low-quality learning images included in the image set acquired by the image set acquisition unit 111 to the image generators 1 to N included in the image generation unit 120, respectively. As a result, in these image generators 1 to N, images with noise removed from the high-noise low-quality learning images are generated respectively. However, as described above, since the image generators 1 to N use machine learning models that are individually learned, there is a possibility that images with different image qualities are generated respectively from the same low-quality learning image in the image generators 1 to N. In this way, the generated image acquisition unit 112 acquires a plurality of generated images generated by the image generators 1 to N respectively.

[0017] The set image division unit 113 divides the input image into a plurality of regions. In the case of this embodiment, the set image division unit 113 divides the reference learning image and the low-quality learning image acquired by the image set acquisition unit 111, and the plurality of generated images acquired by the generated image acquisition unit 112 from the image generators 1 to N, into a plurality of regions respectively. As an image division method, a linear division method such as a rectangle may be used, or a non-linear division method based on the contour of an object in the image may be used. In the case of a division method based on the contour of an object, for example, by using an object detector or a semantic segmentation method, an object is detected from the image and divided based on the contour of the detected object. However, in the reference learning image, the low-quality learning image, and the generated image of the same scene respectively, the division method is the same, and the positions of the divided images divided into regions also correspond to each other.

[0018] The display unit 114 is composed of, for example, a liquid crystal display, and displays the learning reference image and the low-quality learning image for which region division has been performed by the set image division unit 113, and a plurality of generated images so that they can be compared by the user. As a display method that enables comparison between the learning reference image and the low-quality learning image and the generated images, any of various existing display methods such as arranging these images side by side or switching and displaying them in order may be used. In the case of this embodiment, since the image display on the display unit 114 is performed to evaluate the image quality of the generated images, the display of the low-quality learning image may be omitted.

[0019] The evaluation unit 115 acquires and stores, as an image quality evaluation result, for example, the result of comparative evaluation of the image quality based on the learning reference image, the low-quality learning image, and a plurality of generated images displayed on the display unit 114. In the case of this embodiment, the evaluation unit 115 acquires an image quality evaluation result based on subjective evaluation in which the user observes the images displayed on the display unit 114 and compares the image quality for each divided image, or a quantitative image quality evaluation result calculated by a computer.

[0020] For example, in the case of subjective evaluation by the user, the user performs a subjective comparative evaluation such as comparing the image quality for each divided image of the learning reference image and a plurality of generated images. Therefore, in this case, the evaluation unit 115 acquires the image quality comparison result based on the user's subjective comparative evaluation.

[0021] For example, when quantitative image quality evaluation calculated by a computer is performed, the evaluation unit 115 acquires an image quality evaluation result by an objective evaluation method that correlates with subjective evaluation. In the case of an objective evaluation method that correlates with subjective evaluation, the evaluation unit 115 calculates the result of objective evaluation based on an image after performing a plurality of image processes with weights correlated with subjective evaluation on both the learning reference image and the generated image, for example. As the weighting correlated with subjective evaluation, a method of increasing the weight for an image process with a high degree of agreement with the result of subjective evaluation can be used. When such an objective evaluation method is used, the user does not have to perform an operation of comparing corresponding divided images of the learning reference image and the generated image by subjective evaluation, and the work load of the user is reduced.

[0022] Based on the image quality evaluation result acquired by the evaluation unit 115, the dataset storage unit 116 stores a reference dataset in which a divided image of the low-quality learning image and an image generator that generated the generated image with the highest image quality evaluation result in the divided image are associated. In the following description, the divided image of the low-quality learning image included in the reference dataset will be referred to as a "reference divided image".

[0023] The image processing apparatus 130 receives the input image 102 from the outside and inputs it to the image division unit 131. In the case of the present embodiment, it is assumed that the input image 102 is a high-noise image. The image division unit 131 divides the input image 102 into a plurality of regions by the same division process as that performed by the set image division unit 113 described above. The image division unit 131 of the first embodiment sends the divided image obtained by dividing the input image 102 into a plurality to the selection unit 133. In the following description, the divided image obtained by dividing the input image 102 will be referred to as an "input divided image".

[0024] The dataset acquisition unit 132 acquires the reference dataset created by the dataset creation device 110 and stored in the dataset storage unit 116. Then, the dataset acquisition unit 132 sends the reference dataset to the selection unit 133. As described above, the reference dataset is information associating a reference divided image, which is a divided image of the low-quality learning image, with the image generator that generated the generated image having the highest image quality evaluation result in the reference divided image.

[0025] Based on the input image 102 divided into a plurality of input divided images by the image division unit 131 and the reference dataset, the selection unit 133 selects an image generator in the image generation unit 120 for each input divided image. The selection unit 133 selects, for each input divided image, the image generator that generated the generated image having the highest image quality evaluation result at the time of creating the reference dataset, based on the input divided image, the reference divided image in the reference dataset, and the image quality evaluation result.

[0026] In the case of the first embodiment, the image generation unit 120 generates a generated image from the input divided image using the image generator selected by the selection unit 133 so as to correspond to the input divided image, thereby obtaining generated images corresponding to the respective input divided images.

[0027] The composition unit 134 composes two or more of the plurality of generated images respectively generated by the image generators 1 to N of the image generation unit 120. As described above, in the selection unit 133, the image generator that generated the generated image having the highest image quality evaluation result in the reference dataset is selected. Therefore, the composition unit 134 outputs a high-quality composite image obtained by composing two or more generated images generated by the image generator having the highest image quality evaluation result.

[0028] <Hardware Configuration> FIG. 2 is a diagram showing a hardware configuration example of a computer, for example, capable of realizing the image processing apparatus 130 of the present embodiment. The dataset creation apparatus 110 can also be realized with a similar hardware configuration. In addition, both the image processing apparatus 130 and the dataset creation apparatus 110 may be realized by one computer shown in FIG. 2. Further, the hardware configuration of FIG. 2 is also applicable to the apparatus according to the second embodiment described later.

[0029] In FIG. 2, the processor 201 is, for example, a CPU and controls the operation of the entire computer. The memory 202 is, for example, a RAM and temporarily stores the information processing program, image data, reference dataset, etc. according to the present embodiment. The storage medium 203 is a computer-readable storage medium, such as an HDD, SSD, CD-ROM, etc. The storage medium 203 stores various programs including the information processing program according to the present embodiment, image data, reference dataset, data of the learning reference image and the learning low-quality image stored in the data storage unit 101, etc. in a long term. The information processing program of the present embodiment stored in the storage medium 203 is read into the memory 202, and the processor 201 executes the program, whereby each functional unit shown in FIG. 1 is realized.

[0030] The input interface 204 is an interface for acquiring information from an external device. A mouse, keyboard, etc. for the user to input instructions, etc. are connected to the input interface 204. Further, the output interface 205 is an interface for outputting information to an external device. A liquid crystal display or an organic EL display of the display unit 114, etc. are connected to the output interface 205, and for example, display data of an image used for subjective evaluation by the user is output. The bus 206 connects the above-described units and enables data exchange.

[0031] FIG. 3 is a flowchart showing the overall flow of processing in the information processing system according to the present embodiment shown in FIG. 1. First, as the process of step S301, the dataset creation device 110 generates a reference dataset using the image set stored in the data storage unit 101. Next, as the process of step S302, the image processing device 130 generates a high-quality composite image with noise removed from the input image 102 with high noise based on the reference dataset created by the dataset creation device 110.

[0032] FIG. 4 is a flowchart showing the detailed flow of the reference dataset creation process performed by the dataset creation device 110 in step S301 of FIG. 3. First, as the process of step S401, the image set acquisition unit 111 acquires, from the data storage unit 101, a pair of a learning reference image and a learning low-quality image that are images of the same scene.

[0033] Next, as the process of step S402, the generated image acquisition unit 112 inputs the learning low-quality image to each of the image generators 1 to N in the image generation unit 120, and acquires the generated images respectively generated by those image generators 1 to N. Note that in the image generators 1 to N, since machine learning models that have been individually pre-trained as described above are used, the image qualities of the plurality of generated images generated by those image generators 1 to N may be different. Note that in this embodiment, an example in which the learning low-quality image is input to all of the image generators 1 to N is given, but the present invention is not limited to this, and the learning low-quality image may be input to at least two (that is, two or more) image generators respectively.

[0034] Next, as the process of step S403, the set image division unit 113 divides the plurality of generated images respectively generated by the image generators 1 to N from the learning low-quality image, and the learning reference image and the learning low-quality image acquired by the image set acquisition unit 111 into a plurality of regions respectively.

[0035] Next, as the process of step S404, the display unit 114 displays the plurality of generated images and the learning reference image so that the user can compare them. At this time, the user compares the plurality of generated images and the learning reference image for each divided image at the corresponding position, and selects the image generator that generated the generated image that the user feels has the best image quality. Note that the display of the image by the display unit 114 may be performed only when the user performs subjective evaluation of the image quality, and when the above-described objective evaluation method is used, the display unit 114 may not display the image. Then, the evaluation unit 115 acquires the image quality evaluation results obtained for each corresponding divided image of the plurality of generated images and the learning reference image.

[0036] Next, as the process of step S405, the dataset storage unit 116 stores, as a reference dataset, a set of the divided image of the low-quality learning image for which the image quality evaluation was performed in step S404 and the information indicating the image generator selected corresponding to the divided image. Next, as the process of step S406, the dataset creation device 110 determines whether or not the image quality evaluation process in step S404 and the storage process of the reference dataset in step S405 for all the divided images and the generated images have been completed. If there is a divided image for which the process has not been completed, the dataset creation device 110 performs the processes from step S404 to step S405 for the divided image for which the process has not been completed. On the other hand, if it is determined that the process has been completed, the process proceeds to step S407. At this time, even if the image quality evaluation process in step S404 and the storage process of the reference dataset in step S405 for all the divided images and the generated images have not been completed, if the images of other divided images are similar and the same evaluation continues (for example, when the set of the learning reference image and the low-quality learning image acquired from the data storage unit 101 is an image showing a uniform scene such as sky or sea), in order to save the evaluation effort, the process may move to step S407 based on the user's judgment.

[0037] When proceeding to step S407, the dataset creation device 110 checks whether the processing from step S401 to step S406 has been completed for all image sets used for creating the reference dataset. All image sets used for creating the reference dataset refer to the set of all learning reference images and learning low-quality images existing in the data storage unit 101, or one or more sets of the number of learning reference images and learning low-quality images preset by the user. And when there is a set of learning reference images and learning low-quality images for which the image quality evaluation has not been completed, the dataset creation device 110 changes the set of learning reference images and learning low-quality images, and repeats the processing from step S401 to step S406.

[0038] FIG. 5 is a flowchart showing the detailed flow of the image generation process performed by the image processing device 130 in step S302 of FIG. 3. Further, FIG. 6(a) is a diagram used to explain the process performed by the image processing device 130. First, as the processing of step S501, the image processing device 130 acquires the input image 102 which is a high-noise image.

[0039] Next, as the processing of step S502, the image division unit 131 divides the input image 102 into a plurality of regions. The image 601 shown in FIG. 6(a) shows an example in which the input image 102 is divided into nine input divided images divided vertically into three and horizontally into three by the image division unit 131.

[0040] Next, as the processing of step S503, the dataset acquisition unit 132 acquires a reference dataset from the dataset storage unit 116 of the dataset creation device 110. Further, in step S503, the selection unit 133 selects one image generator to be used for each of the input divided images based on the input divided images divided by the image division unit 131 in step S502 and the reference dataset.

[0041] In the case of this embodiment, the selection unit 133 designates one of the plurality of input divided images divided by the image division unit 131 as the target divided image, and searches for a reference divided image similar to the target divided image from among the reference divided images of the low-quality training images in the reference data set.

[0042] At this time, the selection unit 133 compares the feature amount of the target divided image with the feature amounts of all the reference divided images in the reference data set, and identifies the reference divided image with the highest similarity to the feature amount of the target divided image among all the reference divided images. Then, the selection unit 133 selects, as the image generator for the target divided image, the image generator corresponding to the identified reference divided image from among the image generators associated with each reference divided image in the reference data set.

[0043] Note that, as the feature amounts calculated from the input divided image and the reference divided image, values such as the value of the spatial frequency characteristic, the average luminance value, and the contrast in those divided images can be used. For the determination of the similarity of the feature amounts, for example, indexes such as the cosine similarity and the Euclidean distance are used. As an example, consider the case where the similarity is determined using the cosine similarity. The cosine similarity is defined by the following formula (1).

[0044]

Equation

[0045] For example, the feature amount of the target divided region is represented by an n-dimensional feature vector x = (x1, x2, ···, x n ), and similarly, the feature amount of the reference divided image is represented by an n-dimensional feature vector y = (y1, y2, ···, y n) Let it be represented by. The cosine similarity is calculated by substituting the values of x and y into the above formula (1). At least one value among pixel values, brightness, and again luminance values is stored in the feature vector. The cosine similarity becomes a larger value as the similarity between the feature vectors in two divided images at corresponding positions is higher (the maximum value is 1). That is, the larger the value of the cosine similarity, the higher the similarity between those two corresponding divided images. That is, the selection unit 133 identifies the reference divided image searched as having the cosine similarity closest to 1 with respect to the target divided image.

[0046] Then, in the reference dataset, the selection unit 133 selects the image generator associated with the identified reference divided image as the image generator for the target divided image, that is, the appropriate (optimal) image generator for removing the noise of the target divided image. In the example of FIG. 6(a), an example is shown in which one of the image generators 1 to 3 is selected for each of the nine input divided images shown in the image 602. That is, in FIG. 6(a), for each input divided image of the image 602, "1" is entered in the input divided image for which the image generator 1 is selected, "2" is entered in the input divided image for which the image generator 2 is selected, and "3" is entered in the input divided image for which the image generator 3 is selected.

[0047] Next, as the process of step S504, the image generation unit 120 performs a process of generating an image in which noise is removed from the input divided image by the image generator selected by the selection unit 133. FIG. 6(a) shows an example in which the generated images from which noise has been removed by the image generator are arranged corresponding to the positions of the respective input divided images of the original input image 102. That is, in FIG. 6(a), the arrangement 610 indicates the position where the generated image 611 generated by the image generator 1 is arranged, the arrangement 620 indicates the position of the generated image 612 generated by the image generator 2, and the arrangement 630 indicates the position of the generated image 613 generated by the image generator 3.

[0048] Next, as the process of step S505, the image generation unit 120 determines whether the image generation process (generation of a noise-removed image) has been completed for all the input divided images obtained by dividing the input image 102 by the image division unit 131. If it is determined that the image generation process has not been completed for all the input divided images, the process of the image processing apparatus 130 returns to step S503. Then, in step S503, the selection unit 133 designates an input divided image for which the image generation process has not been performed as the target divided image, and selects one image generator to be used by the image generation unit 120 for the designated target divided image. Further, in the next step S504, the image generation unit 120 performs the same image generation process as described above. In this way, the processes of steps S503 and S504 are repeated until the image generation unit 120 determines in step S505 that the image generation process has been completed for all the input divided images.

[0049] On the other hand, if it is determined in step S505 that the image generation process has been completed for all the input divided images, the process of the image processing apparatus 130 proceeds to step S506, and in step S506, the composition unit 134 performs a composition process of the generated images. In the example of FIG. 6(a), an example of a composite image 107 obtained by combining the generated images by the image generation unit 120 is shown. As shown in FIG. 6(a), the composite image 107 is an image obtained by combining a generated image 611 generated by the image generator 1, a generated image 612 generated by the image generator 2, and a generated image 613 generated by the image generator 3. That is, when the composition unit 134 composes the generated images generated for each input divided image, the composition unit 134 arranges and composes the generated images so that the positions of the generated images correspond to the positions of the respective input divided images of the original input image 102.

[0050] Here, when composing the generated images for each input divided image, adjacent generated images are joined together. FIG. 7(a) is a diagram schematically showing two adjacent generated images 701 and 702. However, if the two adjacent generated images 701 and 702 are simply joined together, there is a possibility that image discontinuity, that is, an unnatural joint, may occur at the boundary between the generated images 701 and 702.

[0051] In this embodiment, as a method for reducing the discontinuity of the image during the synthesis of the generated image, that is, making the unnatural joints less noticeable, for example, the following methods can be used. For example, when the image segmentation unit 131 divides the input image 102, it divides the input image so that the boundary portions of the respective input divided images overlap each other with a predetermined width. Then, when the synthesis unit 134 synthesizes the generated images generated for each input divided image, as shown in FIG. 7(b), weights are assigned to the overlapping region 704 with a predetermined width between adjacent generated images, and then they are added together. Thereby, the discontinuity of the image at the boundary portion between adjacent generated images can be reduced (or eliminated).

[0052] Also, in the case of this embodiment, since the synthesized image is an image obtained by synthesizing the generated images generated by different image generators in the image generation unit 120, the sense of unity may be lost in the synthesized image. In this embodiment, as a method for enabling the generation of a synthesized image with a sense of unity, for example, any one of the following several methods, or a combination of two or more of them can be used. For example, the synthesis unit 134 synthesizes the adjacent generated images while aligning their average luminances. Specifically, the synthesis unit 134 adds or subtracts a bias to the luminance of each pixel in one generated image so that the average luminance in one generated image matches the average luminance in the other generated image. Alternatively, the synthesis unit 134 may adjust the luminance for both adjacent generated images. Also, for example, the synthesis unit 134 may adjust the luminance so that the average luminance of the overlapping region 704 with a predetermined width between the adjacent generated images 701 and 702 described in FIG. 7(b) matches. In addition, for example, the image segmentation unit 131 obtains the average luminance of the input image 102, and the synthesis unit 134 may adjust the average luminance of the synthesized image to match the average luminance of the input image 102. In this embodiment, a synthesized image with a sense of unity can be obtained by any one of these methods, or a combination of two or more of them.

[0053] As described above, in the information processing system of the present embodiment, the dataset creation device 110 creates and holds a reference dataset in which an appropriate (optimal) image generator is selected for each divided image through subjective evaluation or objective evaluation. The image processing device 130 generates a high-quality image from which noise has been removed from the input image 102 with high noise based on the reference dataset. In the case of the present embodiment, the image processing device 130 selects an appropriate image generator based on the reference dataset for each input divided image obtained by dividing the input image 102. Then, the image processing device 130 generates a high-quality composite image by synthesizing the generated images generated from the input divided images using the appropriate image generators selected for each divided image. According to the present embodiment, it is possible to provide a higher-quality image than an image obtained by synthesizing the generated images generated using each image generator alone. In particular, when the reference dataset is created in the dataset creation device 110 based on, for example, the result of subjective evaluation by a user, the image processing device 130 can generate a high-quality image according to the personal preference of the user from the input image with high noise. Further, according to the present embodiment, since a noise-removed image is generated for each divided image having a small image size obtained by dividing the input image 102, the amount of calculation can be reduced compared to the case where a noise-removed image is directly generated from the undivided input image 102. Therefore, even an information processing device that cannot perform a large amount of calculation at once can finally generate a high-quality image from which noise has been removed.

[0054] <Second Embodiment> Next, the second embodiment will be described. Note that since the configuration of the second embodiment is substantially the same as the configuration of the first embodiment shown in FIGS. 1 and 2, the illustration and detailed description thereof are omitted. Also in the second embodiment, an example of generating a high-quality composite image from which noise has been removed from the input image 102 with high noise will be described. In the second embodiment, mainly, the image generation process in the plurality of image generators 1 to N of the image generation unit 120 and the image synthesis process using the generated images generated by these image generators 1 to N are different from those in the first embodiment. Hereinafter, in the second embodiment, the differences from the first embodiment will be mainly described.

[0055] In the case of the second embodiment, the selection unit 133 selects the image generators 1 to N in the image generation unit 120 so as to generate an image obtained by removing noise for all the divided images of the input image 102. That is, in the case of the second embodiment, for each of the image generators 1 to N of the image generators, an image generation process of removing noise is performed for all the input divided images of the input image 102. Also in the case of the second embodiment, by searching based on the same image feature amount as in the first embodiment, for each input divided image, the image generator having the highest image quality evaluation result in the reference data set is selected. Then, the information indicating the image generator selected for each input divided image is sent to the synthesis unit 134 via the image generation unit 120.

[0056] The synthesis unit 134 of the second embodiment extracts and synthesizes the divided images corresponding to the image generators selected for each input divided image from the generated images in which each of the image generators 1 to N has removed noise from all the input divided images. As a result, also in the second embodiment, a high-quality synthesized image obtained by synthesizing two or more generated images by the image generator having the highest image quality evaluation result for each input divided image is output from the synthesis unit 134.

[0057] FIG. 8 is a flowchart showing a detailed flow of the image generation process performed by the image processing apparatus 130 in step S302 of FIG. 3 in the second embodiment. In the flowchart of FIG. 8, steps S801 and S802 are the same processes as steps S501 and S502 in the flowchart of FIG. 5, and thus the descriptions thereof are omitted. Also, FIG. 6(b) is a diagram used for explaining the process performed by the image processing apparatus 130 according to the second embodiment.

[0058] In the case of the second embodiment, in step S803, the selection unit 133 selects, for example, the image generators 1 to N in the image generation unit 120 in order so as to perform image generation processing for noise removal on all the input divided images of the input image 102. Then, in the case of the second embodiment, after the processing of step S803, the processing of the image processing apparatus 130 proceeds to step 804.

[0059] When the process proceeds to step S804, the image generation unit 120 determines whether or not the image generation processing for all the input divided images has been completed in each of the image generators 1 to N. If it is determined that the image generation processing for all the input divided images has not been completed in the image generators 1 to N, the processing of the image processing apparatus 130 returns to step S803, and in step S803, the selection unit 133 selects an image generator in which the image generation processing has not been performed. Thus, in the case of the second embodiment, the processing of step S803 is repeated until the image generation unit 120 determines in step S804 that all the image generators 1 to N have each completed the image generation processing for all the input divided images.

[0060] On the other hand, if it is determined in step S804 that the image generation processing has been completed by all the image generators 1 to N, the processing of the image processing apparatus 130 proceeds to step S805. When the process proceeds to step S805, the synthesis unit 134 performs a process of extracting and synthesizing, from the generated images generated by the image generators 1 to N, the generated images generated from the input divided images by the image generator having the highest image quality evaluation result in the reference data set.

[0061] In the example of Fig. 6(b), an example is shown in which an image selected for each input divided image from among the generated images generated by the image generation unit 120 is combined to generate a combined image 107. As shown in Fig. 6(a), in the image generation unit 120, in each image generation unit, generated images are generated from all the input divided images. Also in the example of Fig. 6(b), the input image 102 is divided into 9 input divided images in the same manner as in the example of Fig. 6(a). Note that the numbers entered in each input divided image of the image 602 in Fig. 6(b) are numbers corresponding to the image generator with the highest image quality evaluation result among the image generators 1 to N in the reference data set. Also in the example of Fig. 6(b), it is assumed that the image generators 1 to 3 have the highest image quality evaluation results for the 9 input divided images shown in the image 602.

[0062] In the case of the second embodiment, in the image generation unit 120, image generation processing for all the input divided images of the image 602 is performed in each of the image generators 1 to N. In the example of Fig. 6(b), it is assumed that the image 641 is a generated image generated by the image generator 1, the image 642 is a generated image generated by the image generator 2, and the image 643 is a generated image generated by the image generator 3.

[0063] In the case of the second embodiment, in the combining unit 134, from among the images 641, 642, and 643, the generated images generated from the input divided images by the image generator with the highest image quality evaluation result in the reference data set are extracted respectively. Then, the combining unit 134 generates the combined image 107 by arranging and combining the generated images generated from the input divided images by the image generator with the highest image quality evaluation result in the reference data set at positions corresponding to the input divided images respectively. Note that also in the case of the second embodiment, by the same method as in the case of the first embodiment, the discontinuity of the images at the boundary portions of the adjacent generated images may be reduced, and a combined image with a sense of unity may be generated.

[0064] As described above, in the image processing apparatus 130 of the second embodiment as well, it is possible to generate a high-quality composite image from which noise has been removed from the input image 102 with high noise. Also, in the second embodiment as well, similar to the case of the first embodiment, it is possible to provide a higher-quality image than an image obtained by synthesizing the generated images generated using each image generator alone. Also, in the second embodiment as well, when the reference dataset is created based on the results of subjective evaluation by the user, the image processing apparatus 130 can generate a high-quality image according to the personal preference of the user.

[0065] In addition, in the first and second embodiments described above, an example of generating an image from which noise has been removed from a high-noise image has been given, but the image generation process is not limited to noise removal. For example, the image generation process may be a super-resolution process for generating a high-resolution image from a low-resolution input image. Also, for example, the image generation process may be a style conversion for the input image (for example, a process of converting a color image into a monochrome image). That is, the plurality of image generators 1 to N are not limited to those using the machine learning model that performs the noise removal process described above, and may use a machine learning model that performs a super-resolution process or a style conversion process. In addition, the plurality of image generators 1 to N are not limited to those using a machine learning model, and may, for example, generate an image by an image filter process.

[0066] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. Also, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

[0067] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) Division means for dividing an input image into a plurality of divided images, Selection means for selecting, from among a plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image, Image generation means for generating a plurality of generated images from the input image or the divided images of the input image using the plurality of image generators selected by the selection means, Composition means for composing two or more of the generated images, An information processing apparatus characterized by having the above. (Configuration 2) The image generation means generates the generated image from the divided image using the image generator selected for the divided image by the selection means, The composition means arranges and composes the plurality of generated images generated for each divided image by the image generation means at positions corresponding to each divided image of the input image, respectively. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 3) The image generation means generates a plurality of generated images from the input image using the plurality of image generators selected for each divided image by the selection means, The composition means extracts an image of a region generated by the appropriate image generator selected for each divided image by the selection means from among the plurality of generated images generated from the input image, and arranges and composes the image at positions corresponding to each divided image of the input image, respectively. The information processing apparatus according to Configuration 1, characterized in that. (Configuration 4) The selection means performs the selection of an appropriate image generator for the divided image of the input image based on an evaluation result by a user's subjectivity performed in advance on a plurality of generated images generated by a plurality of image generators from a plurality of images. The information processing apparatus according to any one of Configurations 1 to 3, characterized in that. (Configuration 5) The selection means performs the selection of the appropriate image generator for the divided image of the input image based on an evaluation result correlated with the subjective evaluation of a plurality of generated images generated by a plurality of image generators from a plurality of images in advance, in any one of Configurations 1 to 3. The information processing apparatus according to the described configuration. (Configuration 6) Based on the plurality of images and the images after performing a plurality of image processes with weights correlated with the subjective evaluation on the plurality of generated images generated by the plurality of image generators from the plurality of images, the evaluation means for calculating an evaluation result correlated with the subjective evaluation is provided. The information processing apparatus according to Configuration 5, characterized in that it has. (Configuration 7) The selection means searches for a divided image similar to the divided image of interest of the input image from the divided images of the plurality of images, and selects the image generator corresponding to the similar divided image as the appropriate image generator for the divided image of interest. The information processing apparatus according to any one of Configurations 4 to 6, characterized in that it does so. (Configuration 8) The selection means discriminates the similarity between the plurality of generated images generated by the plurality of image generators from the plurality of images and the divided image of interest using at least one feature amount among the spatial frequency characteristics of the image, the average luminance value, and the contrast, and performs the search based on the similarity. The information processing apparatus according to Configuration 7, characterized in that it does so. (Configuration 9) The selection means converts the feature amount into a feature vector to obtain the similarity. The information processing apparatus according to Configuration 8, characterized in that it does so. (Configuration 10) The composition means composes the generated images respectively corresponding to adjacent divided images so as to overlap them with a predetermined width. The information processing apparatus according to any one of Configurations 1 to 9, characterized in that it does so. (Configuration 11) The composition means adjusts and composes the average luminance of the generated images respectively corresponding to the divided images so as to match them. The information processing apparatus according to any one of Configurations 1 to 10, characterized in that it does so. (Configuration 12) The information processing apparatus according to any one of Configurations 1 to 11, wherein the dividing means divides the input image linearly or divides the input image for each object in the input image. (Configuration 13) The information processing apparatus according to any one of Configurations 1 to 12, wherein the plurality of image generators generate the generated image using a learning model that performs noise removal processing, a learning model that performs super-resolution processing, a learning model that performs style conversion processing, or image filter processing. (Method 1) A dividing step of dividing an input image into a plurality of divided images; A selection step of selecting, from among the plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image; An image generation step of generating a plurality of generated images from the input image or the divided image of the input image using the plurality of image generators selected in the selection step; A synthesis step of synthesizing two or more of the generated images; An information processing method characterized by including the above steps. (Program 1) A program for causing a computer to function as the information processing apparatus according to any one of Configurations 1 to 13.

Explanation of Reference Numerals

[0068] 110: Dataset creation device, 111: Image set acquisition unit, 112: Generated image acquisition unit, 113: Set image division unit, 114: Display unit, 115: Evaluation unit, 116: Dataset storage unit, 120: Image generation unit, 130: Image processing device, 131: Image division unit, 133: Selection unit, 132: Dataset acquisition unit, 133: Selection unit, 134: Synthesis unit

Claims

1. A dividing means for dividing an input image into a plurality of divided images; A selecting means for selecting, from among a plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image; An image generating means for generating a plurality of generated images from the input image or the divided images of the input image using the plurality of image generators selected by the selecting means; A synthesizing means for synthesizing two or more of the generated images; An information processing apparatus characterized by comprising the above.

2. The image generating means generates the generated image from the divided image using the image generator selected for the divided image by the selecting means, The synthesizing means arranges and synthesizes the plurality of generated images generated for each divided image by the image generating means at positions corresponding to each divided image of the input image. The information processing apparatus according to Claim 1, characterized in that.

3. The image generating means generates a plurality of generated images from the input image using the plurality of image generators selected for each divided image by the selecting means, The synthesizing means extracts an image of a region generated by the appropriate image generator selected for each divided image by the selecting means from among the plurality of generated images generated from the input image, and arranges and synthesizes the image at a position corresponding to each divided image of the input image. The information processing apparatus according to Claim 1, characterized in that.

4. The selecting means performs the selection of an appropriate image generator for the divided image of the input image based on an evaluation result based on the user's subjectivity performed on a plurality of generated images generated by a plurality of image generators from a plurality of images in advance. The information processing apparatus according to Claim 1, characterized in that.

5. The selecting means performs the selection of an appropriate image generator for the divided image of the input image based on an evaluation result correlated with the subjective evaluation of a plurality of generated images generated by a plurality of image generators from a plurality of images in advance. The information processing apparatus according to Claim 1, characterized in that.

6. An evaluation means for calculating an evaluation result correlated with the subjective evaluation based on an image after performing a plurality of image processes with weights correlated with the subjective evaluation on the plurality of images and the plurality of generated images generated by the plurality of image generators from the plurality of images. The information processing apparatus according to Claim 5, characterized in that it comprises.

7. The selection means searches for a divided image similar to the divided image of interest of the input image from the divided images of the plurality of images, and selects, as an image generator appropriate for the divided image of interest, the image generator corresponding to the similar divided image. The information processing apparatus according to any one of claims 4 to 6, characterized in that.

8. The selection means discriminates the degree of similarity between a plurality of generated images generated by a plurality of image generators from the plurality of images and the divided image of interest, using at least one feature amount of the spatial frequency characteristics, average luminance value, and contrast of the image, and performs the search based on the degree of similarity. The information processing apparatus according to claim 7, characterized in that.

9. The selection means converts the feature amount into a feature vector to obtain the degree of similarity. The information processing apparatus according to claim 8, characterized in that.

10. The synthesizing means synthesizes the generated images respectively corresponding to adjacent divided images so as to overlap them with a predetermined width. The information processing apparatus according to claim 1, characterized in that.

11. The synthesizing means adjusts and synthesizes the average luminance of the generated images respectively corresponding to the divided images so as to match. The information processing apparatus according to claim 1, characterized in that.

12. The dividing means linearly divides the input image or divides the input image for each object therein. The information processing apparatus according to claim 1, characterized in that.

13. The plurality of image generators generate the generated images using a learning model that performs noise removal processing, a learning model that performs super-resolution processing, a learning model that performs style conversion processing, or image filter processing. The information processing apparatus according to claim 1, characterized in that.

14. A dividing step of dividing an input image into a plurality of divided images; A selection step of selecting, from among a plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image; An image generation step of generating a plurality of generated images from the input image or the divided images of the input image using the plurality of image generators selected in the selection step; A synthesizing step of synthesizing two or more of the generated images; An information processing method, characterized by comprising.

15. A computer, A dividing means for dividing an input image into a plurality of divided images; Selection means for selecting, from among a plurality of image generators, an image generator appropriate for the divided image based on the divided image of the input image; Image generation means for generating a plurality of generated images from the input image or the divided image of the input image using the plurality of image generators selected by the selection means; Combining means for combining two or more of the generated images; A program that causes an information processing apparatus to function as described above.

Images