Information processing apparatus, information processing method, and storage medium

Abstract

The invention discloses an information processing apparatus, an information processing method, and a storage medium. The information processing apparatus divides an input image into a plurality of divided images, and selects an appropriate image generator for each divided image from a plurality of image generators based on the divided images of the input image. The information processing apparatus generates a plurality of generated images from an input image or a segmented image of the input image using a plurality of selected image generators, and synthesizes two or more of the generated images.

Description

Technical Field

[0001] The present disclosure relates to an information processing technology for generating images. Background Art

[0002] There is an information processing technology for generating an image with high image quality by inputting an image with low image quality including high noise, blurring, etc. into an image generator such as a pre-created machine learning model. The features, hues, etc. of the images generated by each image generator may vary depending on differences in components, learning data, etc. Therefore, even when the same image is input into multiple image generators including different components, learning data, etc., the image generators may ultimately output images with different image qualities. For example, if a user wishes to generate an image with a desired image quality based on the user's personal preferences, or generate an image with good image quality based on the user's subjective evaluation, the user can pre-check the performance of each image generator and can use the image generator for generating an image with good image quality based on the user's subjective evaluation. However, in many cases, using only one image generator may not be sufficient to generate an image with a desired image quality based on the user's personal preferences, and may not be sufficient to generate an image with good image quality from an image with low image quality. In other words, some image generators are good at generating images based on the spatial features and hues within the image, while other image generators are not good at generating images based on the spatial features and hues in the image. Therefore, in some cases, an image with insufficient image quality may be generated.

[0003] Japanese Patent Application Laid-Open No. 2022-6869 discusses a technology in which an image is processed by multiple image generators and the processed images are synthesized to generate an image by appropriately removing noise, etc. In the technology discussed in Japanese Patent Application Laid-Open No. 2022-6869, images taken by a magnetic resonance imaging device, an ultrasonic diagnostic device, etc. are described as examples of input images. These images have different noise levels depending on the spatial region within the image. Therefore, if noise is removed by one image generator, an image with image quality varying by region may be generated. The technology discussed in Japanese Patent Application Laid-Open No. 2022-6869 can improve the image quality of the entire image by synthesizing multiple images generated by multiple image generators according to a predetermined region pattern, where these image generators are good at generating images by removing different levels of noise, etc.

[0004] However, the image quality of the image generated by any one of the multiple image generators discussed in Japanese Patent Application Laid-Open No. 2022-6869 may not match the desired image quality. Summary of the Invention

[0005] The present invention aims to generate an image with a desired image quality.

[0006] According to one aspect of the present invention, an information processing apparatus includes: a segmentation unit configured to segment an input image into a plurality of segmented images; a selection unit configured to select, for each of the segmented images, an appropriate image generator from a plurality of image generators based on the segmented images of the input image; an image generation unit configured to generate a plurality of generated images from the input image (e.g., from the segmented images of the input image) using the plurality of image generators selected by the selection unit; and a synthesis unit configured to synthesize two or more of the generated images.

[0007] According to a second aspect of the present invention, an information processing includes: segmenting an input image into a plurality of segmented images; selecting, for each of the segmented images, an appropriate image generator from a plurality of image generators based on the segmented images of the input image; generating a plurality of generated images from the input image or the segmented images of the input image using the plurality of image generators selected by the selection; and synthesizing two or more of the generated images.

[0008] According to a third aspect of the present invention, a non-transitory computer-readable storage medium stores a computer-executable program for causing a computer to execute the method described in the second aspect.

[0009] Known image processing techniques do not consider that the noise level may vary according to the spatial region within the image. Therefore, it is impossible to generate an image with high image quality unless the noise level corresponding to each region is identified. For an image obtained by imaging, the image features (e.g., hue, brightness, and / or noise level) in each region of the image are uncertain. Therefore, it is very likely that an image with the desired image quality cannot be generated (i.e., even when known image processing techniques are applied to the captured image). According to an embodiment of the present disclosure, the input image is segmented into image parts that can be processed separately, which means that the quality of the synthesized generated images can achieve the desired image quality.

[0010] Optional features will now be listed. These can be used alone or in combination with any aspect of the present disclosure.

[0011] The image generation unit may be configured to generate at least one or each generated image from each (respective) input segmented image using the image generator corresponding to the input segmented image selected by the selection unit, thereby obtaining generated images corresponding to the respective input segmented images.

[0012] In an embodiment, the image generation unit may be configured to generate a generated image (e.g., at least one or each generated image) from a segmented image (e.g., each segmented image) using an image generator (e.g., respective image generators) selected by the selection unit for the segmented image. For example, a first generated image may be generated by a first image generator selected for a first segmented image, and a second generated image may be generated by a second image generator selected for a second segmented image.

[0013] The synthesis unit may be configured to synthesize a plurality of generated images generated for the segmented image (e.g., by the image generation unit) by arranging the generated images at positions corresponding to the segmented images of the input image.

[0014] The image generation unit may be configured to generate a plurality of generated images from an input image using a plurality of image generators selected (e.g., by the selection unit) for the segmented image, and wherein the synthesis unit may be set to extract a region image generated by an appropriate image generator (e.g., each selected by the selection unit for the segmented image) from among the plurality of generated images generated based on the input image. Further, the synthesis unit may be configured to synthesize the region images by arranging the region images at positions corresponding to the segmented images (e.g., each segmented image) of the input image.

[0015] The selection unit may be configured to select an appropriate image generator for at least one or each of the segmented images of the input image based on the result of a user's subjective evaluation of a plurality of generated images initially generated by a plurality of image generators from a plurality of images. Thus, the overall quality of the generated images is improved by synthesizing two or more images generated by the image generator having the highest image quality evaluation result.

[0016] The selection unit may be configured to select an appropriate image generator for at least one or each of the segmented images of the input image based on an evaluation result related to a user's subjective evaluation of a plurality of generated images initially generated by a plurality of image generators from a plurality of images. By processing the images based on a plurality of (previous) user evaluations, this saves the user's work of currently (and / or subsequently) generating images.

[0017] The information processing apparatus may include an evaluation unit configured to calculate an evaluation result related to the user's subjective evaluation. The user's subjective evaluation may be based on an image obtained after performing a plurality of image processes (e.g., weights related to the user's subjective evaluation are applied thereto) on a plurality of images and a plurality of generated images generated by a plurality of image generators from the plurality of images. By performing such an objective evaluation on the input image, this eliminates the need for the user to separately compare the segmented image (e.g., corresponding to the generated image) with a learning reference image based on the user's subjective evaluation. Thus, the user's workload can be reduced.

[0018] In this way, it is possible to reduce the image discontinuity at the boundary portion between adjacent generated images, making the image have a unified feeling.

[0019] The selection unit may be configured to search for a segmented image similar to the segmented image of interest in the input image among the segmented images of multiple images, and select the image generator corresponding to the similar segmented image as the appropriate image generator for the segmented image of interest.

[0020] The selection unit may be configured to use at least one feature quantity such as the spatial frequency characteristic, average luminance value, and contrast value of the image to determine the similarity between the segmented image of interest and the multiple generated images generated from multiple images by multiple image generators, and perform a search based on the similarity.

[0021] The selection unit may be configured to obtain the similarity by converting the feature quantity into a feature vector.

[0022] The synthesis unit may be configured to synthesize the generated images corresponding to adjacent segmented images such that the generated images overlap each other with a predetermined width. Therefore, by making the unnatural seams less obvious, the image discontinuity can be further reduced.

[0023] The synthesis unit may be configured to synthesize the generated images by adjusting the average luminance value such that the generated images corresponding to adjacent segmented images have the same average luminance value.

[0024] The segmentation unit may be configured to segment (e.g., linearly) the input image or at least a part of the input image. For example, the segmentation unit may be configured to segment (e.g., linearly) the input image for each object in the input image.

[0025] The multiple image generators may be configured to generate at least one or each of the generated images using at least one of a learning model for noise removal processing, a learning model for super-resolution processing, a learning model for style conversion processing, and image filtering processing.

[0026] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram showing a functional configuration of an information processing system according to a first exemplary embodiment.

[0028] Figure 2 is a block diagram showing an example of a hardware configuration of a computer.

[0029] Figure 3 is a schematic flowchart showing information processing according to a first exemplary embodiment.

[0030] Figure 4 is a flowchart showing the creation process of a reference data set according to a first exemplary embodiment.

[0031] Figure 5 is a flowchart showing the image generation process according to a first exemplary embodiment.

[0032] Figure 6A and Figure 6B is a diagram for explaining the composite image creation process according to an exemplary embodiment.

[0033] Figure 7A and Figure 7B is a diagram for explaining the segmented image composition process.

[0034] Figure 8 is a flowchart showing the image generation process according to a second exemplary embodiment. Detailed Description

[0035] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. The exemplary embodiments described below do not limit the present disclosure, and all the multiple features described in the exemplary embodiments are not necessarily essential for the solution means of the present invention, and multiple features can be arbitrarily combined. The configuration of the exemplary embodiments can be appropriately modified or changed according to the specifications and various conditions (usage conditions, usage environments, etc.) of the device to which the present disclosure is applied. In addition, a part of each of the embodiments described below can be appropriately combined. In the following exemplary embodiments, the same or similar configurations and processing steps are denoted by the same reference numerals, and redundant descriptions thereof will be omitted.

[0036] The first exemplary embodiment describes an example of information processing for generating an image with high image quality by inputting an image with low image quality into an image generator such as a machine learning model. Examples of images with low image quality can include images containing high noise, blurring, etc. The first exemplary embodiment describes an example of information processing for generating an image with high image quality by removing noise from a high-noise image.

[0037] Figure 1 Shows an application example of the information processing apparatus according to the first exemplary embodiment. Figure 1 is a block diagram showing a configuration example of an information processing system including an image processing apparatus 130, a data set creation apparatus 110, and a data storage unit 101. The information processing apparatus according to the first exemplary embodiment may include only the image processing apparatus 130, or may further include the data set creation apparatus 110 in addition to the image processing apparatus 30, or may further include the data storage unit 101.

[0038] The data storage unit 101 stores one or more image sets including a reference image for learning with low noise (or no noise) and high image quality, and a low-quality image for learning that is a high-noise image and has the same scene as the reference image for learning. It is assumed that one or more image sets of the reference image for learning and the low-quality image for learning are prepared and stored in advance. The data storage unit 101 can be provided in an external device such as a server (not shown), and can exist in a local environment as shown in Figure 1 the local environment shown.

[0039] The image processing device 130 includes an image generation unit 120, and the image generation unit 120 is composed of a plurality of image generators 1 to N. Each of the image generators 1 to N included in the image generation unit 120 is composed of a machine learning model such as a neural network. It is assumed that a model pre-trained to generate an image with high image quality by removing noise from a high-noise image is used as the machine learning model. As the machine learning model, a convolutional neural network (CNN) model including a convolutional layer or a transformer model including an attention mechanism can be used. It is also assumed that the machine learning model is trained to obtain different image generation results by changing learning parameters or details of learning data used for learning. The image generators 1 to N are not limited to those using machine learning models, but can be, for example, image generators for generating an image by removing noise from a high-noise image using image filtering processing or the like. For example, as the image filter, a plurality of Gaussian filters with different Gaussian σ parameters can be used.

[0040] In the first exemplary embodiment, it is assumed that a model pre-trained to generate an image with high image quality by removing noise from a high-noise image is used as the machine learning model for each of the image generators 1 to N. Therefore, even when the same image is input to the image generators 1 to N, the image generators 1 to N can generate images with different image qualities. The image generated by each of the image generators 1 to N in the image generation unit 120 is hereinafter referred to as a "generated image".

[0041] The image set acquisition unit 111 of the data set creation device 110 acquires one or more image sets including a reference image for learning and a low-quality image for learning from the data storage unit 101.

[0042] After that, the generated image acquisition unit 112 inputs the low image quality images for learning included in the image set acquired by the image set acquisition unit 111 into each of the image generators 1 to N included in the image generation unit 120. Accordingly, the image generators 1 to N generate images obtained by removing noise from the low image quality images for learning, which are high-noise images. However, as described above, the image generators 1 to N using separately trained machine learning models can generate images with different image qualities from the same low image quality images for learning. Therefore, the generated image acquisition unit 112 acquires a plurality of generated images generated by the image generators 1 to N.

[0043] The set image segmentation unit 113 divides the input image into a plurality of regions. In the first exemplary embodiment, the set image segmentation unit 113 divides the reference image for learning, the low image quality images for learning, and the plurality of generated images acquired by the generated image acquisition unit 112 from the image generators 1 to N into a plurality of regions. As an image segmentation method, a linear segmentation method based on a rectangle or the like, or a non-linear segmentation method based on the contour of each object in the image can be used. In the case of using a segmentation method based on the contour of each object, for example, an object detector or a semantic segmentation method can be used to detect each object from the image, and the image can be segmented based on the contour of the detected object. However, it is assumed that the same segmentation method is used for the reference image for learning, the low image quality images for learning, and the generated images of the same scene, and the regions of these images are segmented at corresponding positions.

[0044] The display unit 114 is composed of a liquid crystal display or the like, and displays the reference image for learning, the low image quality images for learning, and the plurality of generated images divided into a plurality of regions by the set image segmentation unit 113 so that the user can compare these images. As a display method that enables the user to compare the reference image for learning and the low image quality images for learning with the generated images, various known display methods can be used, such as a display method of arranging these images and a display method of sequentially switching the images to be displayed. In the first exemplary embodiment, image display on the display unit 114 is performed to evaluate the image quality of each generated image. Therefore, the display of the low image quality images for learning can be omitted.

[0045] The evaluation unit 115 obtains, as an image quality evaluation result, a comparison and evaluation result of the image quality of, for example, a learning reference image, a learning low image quality image, and a plurality of generated images displayed on the display unit 114, and stores the obtained image quality evaluation result. In the first exemplary embodiment, the evaluation unit 115 obtains an image quality evaluation result based on a subjective evaluation by the user in which the user observes the images displayed on the display unit 114 and compares the image quality of the segmented images, or a quantitative image quality evaluation result calculated by a calculator.

[0046] For example, in the case of performing a subjective evaluation by the user, the user performs a subjective comparison and evaluation process by, for example, comparing the image quality of the segmented images of the plurality of generated images with the learning reference image. Therefore, in this case, the evaluation unit 115 obtains an image quality comparison result based on the subjective comparison and evaluation by the user.

[0047] For example, in the case of obtaining a quantitative image quality evaluation result calculated by a calculator, the evaluation unit 115 obtains an image quality evaluation result based on an objective evaluation method related to the subjective evaluation by the user. In the case of using an objective evaluation method related to the subjective evaluation by the user, the evaluation unit 115 calculates an objective evaluation result based on, for example, an image obtained after performing a plurality of image processes to which weights related to the subjective evaluation by the user are applied to both the learning reference image and the generated image. As a method of applying weights related to the subjective evaluation by the user, for example, a method of increasing the weight of the image process having the highest degree of match with the subjective evaluation result can be used. The use of the objective evaluation method as described above eliminates the need for the user to perform an operation of comparing the segmented image corresponding to the generated image with the learning reference image based on the subjective evaluation by the user, thereby reducing the workload of the user.

[0048] The data set storage unit 116 stores a reference data set in which the segmented image of the learning low image quality image is associated with an image generator having the highest image quality evaluation result among the segmented images generated based on the image quality evaluation result obtained by the evaluation unit 115. Hereinafter, each segmented image of the learning low image quality image included in the reference data set will be referred to as a "reference segmented image".

[0049] The image processing device 130 receives an input image 102 from an external device and inputs the input image 102 to the image segmentation unit 131. In the first exemplary embodiment, it is assumed that the input image 102 is a high-noise image.

[0050] The image segmentation unit 131 divides the input image 102 into a plurality of regions by a segmentation process similar to the segmentation process used by the above-described set image segmentation unit 113. The image segmentation unit 131 according to the first exemplary embodiment transmits the segmentation image obtained by dividing the input image 102 to the selection unit 133. Each segmentation image obtained by dividing the input image 102 is hereinafter referred to as an "input segmentation image".

[0051] The data set acquisition unit 132 acquires the reference data set created by the data set creation device 110 and stored in the data set storage unit 116. The data set acquisition unit 132 transmits the reference data set to the selection unit 133. As described above, the reference data set is information in which a reference segmentation image that is a segmentation image of a learning low-image-quality image is associated with an image generator for generating a generated image having the highest image quality evaluation result among the reference segmentation images.

[0052] The selection unit 133 selects an image generator in the image generation unit 120 for each input segmentation image based on the reference data set and the input image 102 divided into a plurality of input segmentation images by the image segmentation unit 131. The selection unit 133 selects an image generator that generates a generated image having the highest image quality evaluation result during the creation of the reference data set for each input segmentation image based on the input segmentation image, the reference segmentation image in the reference data set, and the image quality evaluation result.

[0053] In the first exemplary embodiment, the image generation unit 120 uses the image generator corresponding to the input segmentation image selected by the selection unit 133 to generate the generated image from the input segmentation image, thereby obtaining the generated images corresponding to the respective input segmentation images.

[0054] The synthesis unit 134 synthesizes two or more of the plurality of generated images generated by the image generators 1 to N in the image generation unit 120. As described above, the selection unit 133 selects an image generator that generates a generated image having the highest image quality evaluation result in the reference data set. Therefore, the synthesis unit 134 outputs a synthesized image with high image quality obtained by synthesizing two or more generated images generated by the image generator having the highest image quality evaluation result.

[0055] <Hardware Configuration>

[0056] Figure 2 is a block diagram showing an example of the hardware configuration of a computer that can implement, for example, the image processing apparatus 130 according to the first exemplary embodiment. The data set creation device 110 can also be implemented by the same as Figure 2The hardware configuration shown is implemented by a similar hardware configuration. In addition, the image processing device 130 and the data set creation device 110 can be implemented by Figure 2 the single computer shown. Figure 2 The hardware configuration shown in

[0057] In Figure 2 the configuration example shown, the processor 201 is, for example, a central processing unit (CPU) and controls the overall operation of the computer. The memory 202 is, for example, a random access memory (RAM) and temporarily stores an information processing program, image data, reference data sets, etc. according to the present exemplary embodiment. The storage medium 203 is a computer-readable storage medium and is, for example, a hard disk drive (HDD), a solid state drive (SSD), or an optical disc (CD) read-only memory (ROM). The storage medium 203 stores various programs for a long time, including the information processing program according to the present exemplary embodiment, data such as image data, reference data sets, and learning reference images and learning low-image-quality images stored in the data storage unit 101. The information processing program according to the present exemplary embodiment stored in the storage medium 203 is loaded into the memory 202, and the processor 201 executes the information processing program, thereby implementing Figure 1 the respective functional units shown.

[0058] The input interface 204 is an interface for obtaining information from an external device. The input interface 204 is connected to a mouse, a keyboard, etc. for user input of instructions, etc. The output interface 205 is an interface for outputting information to an external device. The output interface 205 is connected to a liquid crystal display, an organic electroluminescence (EL) display, etc. for the display unit 114. For example, the output interface 205 outputs image display data for subjective user evaluation, etc. The bus 206 connects the above units so that these units can exchange data.

[0059] Figure 3 is a flowchart showing the overall processing flow in the Figure 1 information processing system according to the first exemplary embodiment shown.

[0060] First, as the processing of step S301, the data set creation device 110 creates a reference data set using the image set stored in the data storage unit 101.

[0061] Next, as the processing of step S302, the image processing device 130 generates a synthetic image with high image quality by removing noise from the high-noise input image 102 based on the reference data set created by the data set creation device 110.

[0062] Figure 4 is shown inFigure 3 The flowchart of the detailed process of the reference data set creation process executed by the data set creation device 110 in step S301 shown.

[0063] First, as the process of step S401, the image set acquisition unit 111 acquires a set of learning reference images and learning low image quality images of the same scene from the data storage unit 101.

[0064] Next, as the process of step S402, the generated image acquisition unit 112 inputs the learning low image quality images into each of the image generators 1 to N in the image generation unit 120, and acquires the generated images generated by the image generators 1 to N. Since the image generators 1 to N use the machine learning models pre-trained separately as described above, the image generators 1 to N may generate images with different image qualities. Although the first exemplary embodiment shows an example of inputting the learning low image quality images into each of the image generators 1 to N, the first exemplary embodiment is not limited to this example. The learning low image quality images may be input into at least two (i.e., two or more) image generators.

[0065] Next, as the process of step S403, the set image segmentation unit 113 segments the multiple generated images generated by the image generators 1 to N based on the learning low image quality images, and the learning reference images and learning low image quality images acquired by the image set acquisition unit 111 into multiple regions.

[0066] Next, as the process of step S404, the display unit 114 displays the multiple generated images and the learning reference images so that the user can compare these images. In this case, the user compares the multiple generated images with the learning reference images for each segmented image at the corresponding position, and selects the image generator used to generate the generated image with the highest image quality. The display unit 114 may display images only when performing the subjective image quality evaluation of the user. If the above objective evaluation method is used, the image display of the display unit 114 may be omitted. In addition, the evaluation unit 115 obtains the results of the image quality evaluation performed on each segmented image corresponding to the multiple generated images and the learning reference images.

[0067] Next, as the process of step S405, the data set storage unit 116 stores the segmented images of the learning low image quality images for which the image quality evaluation is performed in step S404 and the set of information indicating the selected image generators corresponding to the segmented images as a reference data set.

[0068] Next, as the process of step S406, the data set creation device 110 determines whether the image quality evaluation process of step S404 and the reference data set storage process of step S405 for all the divided images and all the generated images have been completed. If there is a divided image for which the process has not been performed yet, the data set creation device 110 performs the processes of steps S404 and S405 on the divided image for which the process has not been performed yet. On the other hand, if the process is completed (Yes in step S406), the process proceeds to step S407. Even when the image quality evaluation process of step S404 and the reference data set storage process of step S405 have not been completed for all the divided images and all the generated images, if other divided images are similar to each other and similar evaluation results are continuously obtained (for example, the set of the learning reference images and the learning low image quality images obtained from the data storage unit 101 represents an image of a uniform scene such as the sky or the sea), the process can proceed to step S407 based on the determination of the user to save the evaluation work of the user.

[0069] In step S407, the data set creation device 110 checks whether the processes of steps S401 to S406 have been completed for all the image sets used to create the reference data set. All the image sets used to create the reference data set represent the set of all the learning reference images and the learning low image quality images stored in the data storage unit 101, or one or more sets of a preset number of learning reference images and learning low image quality images set by the user. If there is a set of learning reference images and learning low image quality images for which the image quality evaluation process has not been performed yet, the data set creation device 110 changes the set of learning reference images and learning low image quality images and repeats the processes of steps S401 to S406.

[0070] Figure 5 Yes shows the Figure 3 flowchart showing the detailed process of the image generation process executed by the image processing device 130 in step S302 shown. Figure 6A For explaining the process to be executed by the image processing device 130.

[0071] First, as the process of step S501, the image processing device 130 obtains the input image 102 as a high-noise image.

[0072] Next, as the process of step S502, the image segmentation unit 131 divides the input image 102 into a plurality of regions.

[0073] Figure 6AThe image 601 shown is an example of an image obtained by the image segmentation unit 131 dividing the input image 102 into nine input segmented images (including three vertically segmented images and three horizontally segmented images).

[0074] Next, as the process of step S503, the data set acquisition unit 132 acquires a reference data set from the data set storage unit 116 of the data set creation device 110. Further, in step S503, the selection unit 133 selects an image generator to be used for each input segmented image based on the input segmented images segmented by the image segmentation unit 131 in step S502 and the reference data set.

[0075] In the present exemplary embodiment, the selection unit 133 sets one of the plurality of input segmented images segmented by the image segmentation unit 131 as the segmented image of interest, and searches for a reference segmented image similar to the segmented image of interest from the reference segmented images of the low image quality images for learning in the reference data set.

[0076] In this case, the selection unit 133 compares the feature amount of the segmented image of interest with the feature amounts of all the reference segmented images in the reference data set, and identifies the reference segmented image having the highest similarity of feature amount to the segmented image of interest among all the reference segmented images. Further, the selection unit 133 selects, from the image generators associated with the reference segmented images in the reference data set, the image generator corresponding to the identified reference segmented image as the image generator corresponding to the segmented image of interest.

[0077] As the feature amount calculated for each of the input segmented image and the reference segmented image, for example, the spatial frequency characteristic value, the average luminance value, or the contrast value in the segmented image can be used. To determine the similarity between the feature amounts, for example, indexes such as cosine similarity or Euclidean distance can be used. For example, assume a case where cosine similarity is used to determine the similarity. Cosine similarity is defined by the following expression (1).

[0078]

[0079] For example, assume that the feature amount of the segmented region of interest is represented by the n-dimensional feature vector x = (x1, x2,..., x n ), and the feature amount of the reference segmented image is represented by the n-dimensional feature vector y = (y1, y2,..., y n) is represented. The cosine similarity is calculated by substituting the values of "x" and "y" into the above expression (1). As the feature vector, at least one of the pixel value, lightness value, saturation value, and brightness value is stored. The cosine similarity takes a value that increases as the similarity between the feature vectors in the two divided images at the corresponding positions increases (the maximum value is "1"). In other words, as the cosine similarity value increases, the similarity between the two corresponding divided images also increases. That is, the selection unit 133 identifies the searched reference divided image in which the cosine similarity between the divided image of interest and the reference divided image is closest to "1".

[0080] In addition, the selection unit 133 selects the image generator associated with the reference divided image identified in the reference data set as the image generator corresponding to the divided image of interest, that is, the appropriate (optimal) image generator for performing noise removal processing on the divided image of interest.

[0081] Figure 6A An example of selecting any one of image generators 1 to 3 for each of the nine input divided images shown in image 602 is shown. Specifically, in Figure 6A the input divided images of image 602 in the example shown, "1" is set in the input divided image selected by image generator 1, "2" is set in the input divided image selected by image generator 2, and "3" is set in the input divided image selected by image generator 3.

[0082] Next, as the processing of step S504, the image generation unit 120 performs processing for generating an image in which the noise has been removed by the image generator selected by the selection unit 133 using the input divided image.

[0083] Figure 6A An example is shown in which the generated image with noise removed by the image generator is located at a position corresponding to each input divided image of the original input image 102. Specifically, in Figure 6A the example shown, layout 610 indicates the position of the generated image 611 generated by image generator 1. Layout 620 indicates the position of the generated image 612 generated by image generator 2, and layout 630 indicates the position of the generated image 613 generated by image generator 3.

[0084] Next, as the processing in step S505, the image generation unit 120 determines whether the image generation process (the process of generating an image with noise removed) has been completed for all the input segmented images obtained by segmenting the input image 102 by the image segmentation unit 131. If it is determined that the image generation process has not been completed for all the input segmented images ("No" in step S505), the processing of the image processing apparatus 130 returns to step S503. In step S503, the selection unit 133 sets the input segmented image for which the image generation process has not been performed as the segmented image of interest, and selects an image generator for the segmented image of interest of the image generation unit 120. Next, in step 504, the image generation unit 120 performs the image generation process in the same manner as described above. Therefore, the processing of steps S503 and S504 is repeatedly executed until the image generation unit 120 determines in step S505 that the image generation process for all the input segmented images has been completed.

[0085] On the other hand, if the image generation process has been completed for all the input segmented images ("Yes" in step S505), the processing of the image processing apparatus 130 proceeds to step S506. In step S506, the synthesis unit 134 performs the generated image synthesis process.

[0086] Figure 6A An example of a synthesized image 107 obtained by synthesizing the generated images generated by the synthesized image generation unit 120 is shown. As Figure 6A shown, the synthesized image 107 is an image obtained by synthesizing the generated image 611 generated by the image generator 1, the generated image 612 generated by the image generator 2, and the generated image 613 generated by the image generator 3. In other words, in the case of synthesizing the generated images generated for each input segmented image, the synthesis unit 134 synthesizes the generated images by arranging the generated images so that the positions of the generated images correspond to the positions of the input segmented images of the original input image 102.

[0087] In the case of synthesizing the generated images of each input segmented image, adjacent generated images are synthesized. Figure 7A Two adjacent generated images 701 and 702 are schematically shown. However, if the two adjacent generated images 701 and 702 are simply synthesized together, image discontinuity or unnatural seams may appear at the boundary portion between the generated images 701 and 702.

[0088] In the present exemplary embodiment, as a method for reducing image discontinuity or making unnatural seams less noticeable during image synthesis processing, for example, the following methods can be used. For example, in the case of segmenting the input image 102, the image segmentation unit 131 segments the input image 102 in such a way that boundary portions of the input segmented images overlap each other by a predetermined width. Further, in the case of synthesizing the generated images generated for each input segmented image, the synthesis unit 134 synthesizes the generated images after applying weights to an overlapping region 704 of a predetermined width between adjacent generated images 701 and 702, as Figure 7B shown. This makes it possible to reduce (eliminate) image discontinuity at the boundary portion between adjacent generated images.

[0089] In the present exemplary embodiment, the synthesized image is an image obtained by synthesizing generated images generated by different image generators in the image generation unit 120. Therefore, a sense of unity may be lost in the synthesized image.

[0090] In the present exemplary embodiment, as a method for generating a synthesized image with a sense of unity, for example, any one of the following methods, or a combination of two or more of the following methods can be used. For example, the synthesis unit 134 synthesizes adjacent generated images in such a way that the adjacent generated images have the same average luminance value. Specifically, the synthesis unit 134 adds a deviation to or subtracts a deviation from the luminance value in each pixel within one of the generated images so that the average luminance value in one generated image matches the average luminance value in the other generated image. Alternatively, the synthesis unit 134 may perform luminance adjustment processing on both of two adjacent generated images. Further, for example, the synthesis unit 134 may adjust the luminance value in such a way that the average luminance value in an overlapping region 704 of a predetermined width in one of the adjacent generated images 701 and 702 matches the average luminance value in an overlapping region 704 of a predetermined width in the other of the adjacent generated images 701 and 702 described above with reference to Figure 7B the description. Further, for example, the image segmentation unit 131 may obtain the average luminance value of the input image 102, and the synthesis unit 134 may adjust the average luminance value in the synthesized image to match the average luminance value in the input image 102. In the present exemplary embodiment, a synthesized image with a sense of unity can be obtained by any one of the above methods, or by a combination of two or more of the above methods.

[0091] As described above, in the information processing system according to the first exemplary embodiment, the data set creation device 110 creates a reference data set using an appropriate (optimal) image generator selected for each segmented image through subjective or objective evaluation by the user, and holds the created reference data set. The image processing device 130 generates an image with high image quality by removing noise from the high-noise input image 102 based on the reference data set. In the first exemplary embodiment, the image processing device 130 selects an appropriate image generator based on the reference data set set for each input segmented image obtained by segmenting the input image 102. In addition, the image processing device 130 synthesizes the generated images generated from the input segmented images using the appropriate image generator selected for each input segmented image, thereby generating a synthesized image with high image quality. According to the first exemplary embodiment, an image with higher image quality than an image obtained by separately synthesizing the generated images generated using each image generator can be provided. In particular, if the data set creation device 110 creates a reference data set based on, for example, the subjective evaluation result of the user, the image processing device 130 can generate an image with high image quality from the high-noise input image based on the personal preference of the user. According to the first exemplary embodiment, for each small segmented image obtained by segmenting the input image 102, an image obtained by removing noise is generated, which results in a reduction in the amount of calculation compared to the case of directly generating a noise-removed image from the unsegmented input image 102. Therefore, even in an information processing device not configured to perform a large amount of calculations at once, a noise-removed image with high image quality can be finally generated.

[0092] Next, a second exemplary embodiment will be described. The configuration according to the second exemplary embodiment is substantially the same as the configuration according to the first exemplary embodiment shown in Figure 1 and Figure 2 Therefore, the illustration and detailed description of the configuration according to the second exemplary embodiment are omitted. The second exemplary embodiment also shows an example of generating a synthesized image with high image quality by removing noise from the high-noise input image 102. The second exemplary embodiment is mainly different from the first exemplary embodiment in terms of the image generation process among the plurality of image generators 1 to N in the image generation unit 120 and the image synthesis process of the generated images generated using the image generators 1 to N. The differences between the second exemplary embodiment and the first exemplary embodiment will be mainly described below.

[0093] In the second exemplary embodiment, the selection unit 133 selects image generators 1 to N in the image generation unit 120, so that an image obtained by removing noise from all the segmented images of the input image 102 can be generated. Specifically, in the second exemplary embodiment, an image generation process for generating an image by removing noise from all the input segmented images of the input image 102 is performed for each of the image generators 1 to N. In addition, in the second exemplary embodiment, as in the first exemplary embodiment, an image generator having the highest image quality evaluation result in the reference data set is selected for each input segmented image by searching based on the feature amount of the image. Then, information on the image generators selected for each input segmented image is sent to the synthesis unit 134 through the image generation unit 120.

[0094] The synthesis unit 134 according to the second exemplary embodiment extracts segmented images corresponding to the image generators selected for each input segmented image from the generated images generated by removing noise from all the input segmented images by each of the image generators 1 to N, and synthesizes the extracted segmented images. Thus, also in the second exemplary embodiment, the synthesis unit 134 outputs a synthesized image with high image quality obtained by synthesizing two or more of the generated images generated by the image generators having the highest image quality evaluation result for each input segmented image.

[0095] Figure 8 is a flowchart showing the detailed process of the image generation process executed by the image processing apparatus 130 according to the second exemplary embodiment in Figure 3 step S302 shown. In the Figure 8 flowchart shown, the processes of steps S801 and S802 are respectively similar to the processes of steps S501 and S502 in the Figure 5 flowchart shown, and thus their descriptions are omitted. Figure 6B is used to explain the process to be executed by the image processing apparatus 130 according to the second exemplary embodiment.

[0096] According to the second exemplary embodiment, in step S803, the selection unit 133 sequentially selects, for example, image generators 1 to N in the image generation unit 120, so as to perform an image generation process for generating an image by removing noise from all the input segmented images of the input image 102. In the second exemplary embodiment, after the process of step S803, the process of the image processing apparatus 130 proceeds to step S804.

[0097] In step S804, the image generation unit 120 determines whether the image generation process has been completed for all the input divided images in each of image generators 1 to N. If it is determined that the image generation process has not been completed for all the input divided images in each of image generators 1 to N (No in step S804), the process of the image processing apparatus 130 returns to step S803. In step S803, the selection unit 133 selects an image generator for which the image generation process has not been performed. Thus, in the second exemplary embodiment, the process of step S803 is repeatedly executed until the image generation unit 120 determines in step S804 that the image generation process has been completed for all the input divided images in each of image generators 1 to N.

[0098] On the other hand, if it is determined that the image generation process in each of image generators 1 to N has been completed (Yes in step S804), the process of the image processing apparatus 130 proceeds to step S805.

[0099] In step S805, the synthesis unit 134 performs a process of extracting, from the generated images generated by each of image generators 1 to N, the generated image generated from the input divided image by the image generator having the highest image quality evaluation result in the reference data set, and synthesizing the extracted generated images.

[0100] Figure 6B An example of generating a composite image 107 by synthesizing the generated images selected for each input divided image from the generated images generated by the image generation unit 120 is shown. As Figure 6A shown, the image generation unit 120 generates the generated images from all the input divided images in each of image generators 1 to N. Further, in the Figure 6B example shown, as in the Figure 6A example shown, the input image 102 is divided into nine input divided images to obtain the image 602. Figure 6B The numbers set in each input divided image of the image 602 shown correspond to the numbers corresponding to the image generator having the highest image quality evaluation result among image generators 1 to N in the reference data set. Further, in the Figure 6B example shown, it is assumed that image generators 1 to 3 have the highest image quality evaluation results for the nine input divided images shown in the image 602.

[0101] In the second exemplary embodiment, the image generation unit 120 performs the image generation process on all the input divided images of the image 602 in each of image generators 1 to N. In Figure 6BIn the example shown, it is assumed that image 641 is a generated image generated by image generator 1, image 642 is a generated image generated by image generator 2, and image 643 is a generated image generated by image generator 3.

[0102] In the second exemplary embodiment, the synthesis unit 134 extracts, from image 641, image 642, and image 643, the generated images generated by the image generator with the highest image quality evaluation result in each reference data set from the input segmented images. Further, the synthesis unit 134 synthesizes the generated images generated by the image generator with the highest image quality evaluation result in the reference data set from the input segmented images by arranging the generated images at positions corresponding to the input segmented images, thereby generating a synthesized image 107. Further, in the second exemplary embodiment, the image discontinuity at the boundary portion between adjacent generated images can be reduced by a method similar to the method used in the first exemplary embodiment, thereby generating a synthesized image with a sense of unity.

[0103] As described above, the image processing apparatus 130 according to the second exemplary embodiment can also generate a synthesized image with high image quality by removing noise from the high-noise input image 102. Further, in the second exemplary embodiment, as in the first exemplary embodiment, an image with higher image quality than the image obtained by synthesizing the generated images generated by each image generator used alone can be provided. Further, in the second exemplary embodiment, if the reference data set is created based on the subjective evaluation result of the user, the image processing apparatus 130 can generate an image with high image quality based on the personal preference of the user.

[0104] Although the above-described first and second exemplary embodiments show examples of generating images by removing noise from high-noise images, the image generation process is not limited to noise removal processing. Examples of the image generation process may include super-resolution processing for generating a high-resolution image from a low-resolution input image. Other examples of the image generation process may include style conversion processing of the input image (for example, processing for converting a color image into a monochrome image). In other words, the plurality of image generators 1 to N are not limited to the image generators using the machine learning model for performing noise removal processing as described above, but may be image generators using the machine learning model for performing super-resolution processing or style conversion processing. The plurality of image generators 1 to N are not limited to the image generators using the machine learning model, but may be, for example, image generators for generating images by image filtering processing.

[0105] <Other Exemplary Embodiments>

[0106] The present disclosure can also be implemented by providing a program for implementing one or more functions of the above-described exemplary embodiments to a system or device via a network or a storage medium, and a processor or processors in a computer of the system or device reading and executing the program, or by a circuit (e.g., an application specific integrated circuit (ASIC)) for implementing one or more functions.

[0107] The above-described exemplary embodiments are merely examples of embodiments for implementing the present disclosure, and the technical scope of the present disclosure should not be construed in a limited manner by these exemplary embodiments. That is, the present disclosure can be implemented in various forms without departing from its technical idea or main features.

[0108] According to one aspect of the present disclosure, an image having a desired image quality can be generated.

[0109] Other embodiments

[0110] Embodiments of the present disclosure can also be implemented by the following method, that is, by providing software (program) for executing the functions of the above-described embodiments to a system or device via a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0111] Although the present disclosure has been described with reference to exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: A segmentation unit configured to segment the input image into a plurality of segmented images; a selection unit configured to select an appropriate image generator for each of the segmented images from a plurality of image generators based on the segmented images of the input image; an image generating unit configured to generate a plurality of generated images from the segmented images of the input image using the plurality of image generators selected by the selecting unit; as well as The synthesis unit is configured to synthesize two or more of the generated images.

2. The information processing device according to claim 1, in, an image generating unit generating a generated image from the segmented image using the image generator selected by the selecting unit for the segmented image, and The synthesis unit synthesizes a plurality of generated images generated for the segmented images by arranging the generated images at positions corresponding to the segmented images of the input image.

3. The information processing device according to claim 1, in, an image generating unit generating a plurality of generated images from the input image using a plurality of image generators selected for the segmented image, and Among them, the synthesis unit extracts the regional images generated by the appropriate image generator selected for each of the segmented images from multiple generated images generated based on the input image, and synthesizes the regional images by arranging the regional images at positions corresponding to the segmented images of the input image.

4. The information processing device according to claim 1, wherein: The selection unit selects an appropriate image generator for each of the segmented images of the input image based on a result of a user's subjective evaluation of a plurality of generated images preliminarily generated by the plurality of image generators from the plurality of images.

5. The information processing device according to claim 1, wherein: The selection unit selects an appropriate image generator for each of the segmented images of the input image based on an evaluation result related to a user's subjective evaluation of a plurality of generated images preliminarily generated by the plurality of image generators from the plurality of images.

6. The information processing device according to claim 5, further comprising: An evaluation unit is configured to calculate an evaluation result related to user subjective evaluation based on an image obtained after performing a plurality of image processes to which weights related to user subjective evaluation are applied on a plurality of images and a plurality of generated images generated from the plurality of images by a plurality of image generators.

7. The information processing device according to claim 4, wherein: The selection unit searches for a segmented image similar to the segmented image of interest in the input image among the segmented images of the plurality of images, and selects an image generator corresponding to the similar segmented image as an appropriate image generator for the segmented image of interest.

8. The information processing device according to claim 7, wherein: The selection unit determines similarity between the segmented image of interest and a plurality of generated images generated from a plurality of images by a plurality of image generators using at least one feature amount of spatial frequency characteristics, average brightness value, and contrast value of the image, and performs a search based on the similarity.

9. The information processing device according to claim 8, wherein: The selection unit obtains the similarity by converting the feature quantity into a feature vector.

10. The information processing device according to claim 1, wherein: The synthesis unit synthesizes generated images corresponding to adjacent divided images so that the generated images overlap each other with a predetermined width.

11. The information processing device according to claim 1, wherein: The synthesis unit synthesizes the generated images by adjusting the average brightness value so that the generated images corresponding to the adjacent segmented images have the same average brightness value.

12. The information processing device according to claim 1, wherein: The segmentation unit linearly segments the input image, or linearly segments the input image for each object in the input image.

13. The information processing device according to claim 1, wherein: The plurality of image generators generates generated images using at least one of a learning model for a noise removal process, a learning model for a super-resolution process, a learning model for a style transfer process, and an image filtering process.

14. An information processing method, comprising: Segment the input image into multiple segmented images; selecting, based on the segmented images of the input image, an appropriate image generator for each of the segmented images from a plurality of image generators; generating a plurality of generated images from the segmented images of the input image using the selected plurality of image generators; and Synthesize two or more of the generated images.

15. A non-transitory computer-readable storage medium storing a computer-executable program for causing a computer to execute the method according to claim 14.

Images