High-definition processing device and high-definition processing method

Abstract

To realize high-definition processing in which transparency and restoration accuracy are high.SOLUTION: A high-definition processing device reconfigures an input image on the basis of function approximation to perform high-definition. The high-definition processing device comprises: an area-specific image processing hyper parameter estimation unit 112 which estimates a hyper parameter when performing the function approximation for each pixel or area of an input image; and a signal processing unit 113 which performs the high-definition for the input image by using a machine learning model, on the basis of the hyper parameter estimated by the area-specific image processing hyper parameter estimation unit 112.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a high definition processing device and a high definition processing method. [Background technology]

[0002] Satellite imagery, capable of capturing wide-area, repeated images of the Earth's surface, has been widely used for surveillance and monitoring purposes for both civilian and national security. In 2014, Maxar Technologies' WorldView-3 (WV3) achieved a resolution of 0.31 meters, reaching the resolution limit for satellite imagery available to civilians set by the U.S. Department of Commerce. The company subsequently planned to launch the World View Legion satellite for capturing satellite images at the same resolution, and Airbus of France also launched Pleiades Neo 3 and Pleiades Neo 4, providing satellite imagery with a resolution of approximately 30 centimeters. Furthermore, falling satellite launch costs have led to the entry of venture companies, intensifying competition in the satellite imagery business. Furthermore, in 2020, Maxar Technologies began selling 15-centimeter satellite imagery that had been processed to a resolution of 30 centimeters (also known as super-resolution). This has led to a growing demand for higher resolution satellite imagery, including existing satellite images and previously captured images.

[0003] The mainstream method for increasing the resolution of images has been interpolation using signal processing, but since the Super-Resolution Convolutional Neural Network proposed a method for increasing the resolution using deep learning, many new methods have been proposed. These methods significantly reduce the error from the target image and are widely used for hobby purposes.

[0004] Background art in this technical field is the technology disclosed in Patent Document 1. Patent Document 1 states that "The learning device receives, as learning data, a pair of a reference image having higher image quality than a target image obtained by remote sensing of a target satellite and a degraded image, which is a reference image with lower image quality. The learning device then generates a trained model by learning the learning data. The degraded image is generated by applying degradation processing based on target satellite information to the reference image. The target satellite information indicates remote sensing information." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-150145 Summary of the Invention [Problem to be solved by the invention]

[0006] Existing high-resolution techniques using machine learning, including the technology disclosed in Patent Document 1, have the problem that the high-resolution process is opaque because the image is generated directly from a neural network during high-resolution processing. This type of opaque process is called black-box processing. While black-box processing is not a problem for hobbyists, it is difficult to use for important decision-making. On the other hand, high-resolution techniques using signal processing have the problem that the processing is highly transparent but the restoration accuracy is low.

[0007] The present invention has been made in view of the above circumstances, and aims to provide a high definition processing device and a high definition processing method that realize high definition processing with high transparency and restoration accuracy. [Means for solving the problem]

[0008] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is a high-definition processing device that reconstructs an input image based on function approximation to increase its definition, and includes a hyperparameter estimation unit that estimates hyperparameters for function approximation for each pixel or region of the input image, and a signal processing unit that uses a machine learning model to increase the definition of the input image based on the hyperparameters estimated by the hyperparameter estimation unit.

[0009] In addition, the high-resolution processing method of the present invention for solving the above problem is a high-resolution processing method that reconstructs an input image based on function approximation to increase its resolution, estimates hyperparameters for function approximation for each pixel or region of the input image, and increases the resolution of the input image using a machine learning model based on these estimated hyperparameters. [Effects of the Invention]

[0010] According to the present invention, high-definition processing using hyperparameters can be performed to achieve high transparency and restoration accuracy.

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the mode for carrying out the invention (hereinafter referred to as the embodiment). [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a functional block diagram showing an example of a system configuration of a high definition processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing a list of information recorded in an image DB. [Figure 3] FIG. 10 is a diagram showing a list of information recorded in a machine learning model DB. [Figure 4] 5 is a flowchart showing an example of processing of a high definition processing method executed in the high definition processing device according to the first embodiment. [Figure 5] FIG. 1 is a diagram illustrating hyperparameters of Bicubic. [Figure 6] FIG. 10 is a diagram illustrating the estimation of hyperparameters by region and the high-definition processing. [Figure 7] FIG. 10 is a diagram illustrating a linear sum of an unsharpened mask. [Figure 8] FIG. 10 is a block diagram showing an example of the system configuration of a high definition processing device according to a first modification of the first embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of the system configuration of a high definition processing device according to a second modification of the first embodiment. [Figure 10] FIG. 10 is a functional block diagram showing an example of the system configuration of a high definition processing apparatus according to a second embodiment of the present invention. [Figure 11] 10 is a flowchart showing an example of processing of a high definition processing method executed in the high definition processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant explanations will be omitted.

[0014] First Embodiment [System configuration example] 1 is a functional block diagram showing an example of the system configuration of a high definition processing device according to a first embodiment of the present invention. The high definition processing device 100 according to the first embodiment of the present invention has a system configuration including an image processing unit 110 and a database unit 120. Hereinafter, the database unit will be referred to as a DB unit.

[0015] The image processing unit 110 includes an image receiving unit 111, a region-specific image processing hyperparameter estimation unit 112, a signal processing unit 113, and an output unit 114. The database unit 120 includes an image DB 121 and a machine learning model DB 122.

[0016] (Configuration example of image processing unit) In the image processing unit 110, the image receiving unit 111 selects a desired image from the image DB 121 and receives it as an input image. The image receiving unit 111 may also be configured to obtain the input image using a sensor that obtains images, such as a camera. Furthermore, the image receiving unit 111 may use an image obtained by cutting out a partial area of ​​the obtained image as the input image.

[0017] The input image may be an R (red), G (green), and B (blue) image, a black-and-white image, or a spectral image. The input image also includes an image obtained by irradiating a radar and observing the reflected light, such as a SAR (synthetic aperture radar) image. The input image also includes a pseudo-image, such as a heat map, in which the locations of posts on a social networking site are converted into a two-dimensional histogram based on location information.

[0018] The region-specific image processing hyperparameter estimation unit 112 estimates region-specific image processing hyperparameters for each pixel or region of the input image. Specifically, the region-specific image processing hyperparameter estimation unit 112 selects a hyperparameter estimation model from the machine learning model DB 122 and estimates image processing hyperparameters for each region using the machine learning model. Hyperparameters are parameters that control the behavior of the machine learning algorithm.

[0019] The signal processing unit 113 performs signal processing based on the regional image processing hyperparameters estimated by the regional image processing hyperparameter estimation unit 112. Specifically, the signal processing unit 113 performs processing to generate a high-definition image based on the input image and the regional image processing hyperparameters.

[0020] The output unit 114 outputs the high definition image generated by the high definition processing in the signal processing unit 113 to the outside of the image processing unit 110.

[0021] (Database section configuration example) 2, the image DB 121 stores information such as the image to be subjected to high-definition processing, the storage location of the image, the file name of the image, the size of the image, the number of bands of the image, and the type of the image (input image type). Here, the type of image is information indicating the format and acquisition method of the image, such as an optical image or a SAR (synthetic aperture radar) image. In the case of satellite images, the ground resolution may also be stored as the type of image.

[0022] The machine learning model DB 122 stores the machine learning model, the storage location of the machine learning model, the file name of the machine learning model, the input image type of the machine learning model, and the image processing type, as shown in Fig. 3. Here, the type of machine learning model refers to the type of image that is assumed to be the input image.

[0023] [Example of high-definition processing method] Next, an example of processing of a high definition processing method executed in the high definition processing device according to the first embodiment having the above configuration will be described. Fig. 4 is a flowchart showing an example of processing of a high definition processing method executed in the high definition processing device according to the first embodiment.

[0024] First, the image receiving unit 111 selects a desired image from the image DB 121 and receives it as an input image (step S11). Note that the image receiving unit 111 may receive the input image using a sensor that acquires an image, such as a camera, or may receive an image obtained by cutting out a partial area of ​​the acquired image as the input image.

[0025] Next, the region-specific image processing hyperparameter estimation unit 112 selects a hyperparameter estimation model from the machine learning model DB 122 (step S12). However, the hyperparameter estimation model may be selected automatically using type information, rather than being selected by a user.

[0026] Next, the region-specific image processing hyperparameter estimation unit 112 estimates image processing hyperparameters for each region of the input image (step S13). Specifically, a hyperparameter estimation model is selected from the machine learning model DB 122, and image processing hyperparameters (region-specific image processing hyperparameters) are estimated for each region using the machine learning model. Details of the image processing hyperparameters will be described later.

[0027] Next, the signal processing unit 113 generates a high-definition image based on the input image and the regional image processing hyperparameters estimated in step S13 (step S14). By using the regional image processing hyperparameters estimated in step S13, it is possible to flexibly calculate the pixel value for each pixel.

[0028] Next, the output unit 114 outputs the high definition image generated by the high definition processing in step S14 to the outside of the image processing unit 110 (step S15).

[0029] [About hyperparameters in image processing] One example of a hyperparameter for image processing is the hyperparameter of Bicubic, which is an algorithm used for pixel interpolation in image processing.

[0030] In high-resolution processing using Bicubic, the result is expressed as a linear sum of 16 surrounding pixel values, as shown in Figure 5. Figure 5 is a diagram explaining the hyperparameters of Bicubic. In Figure 5, ● indicates the position of a pixel in the high-resolution image, and ○ indicates 16 surrounding pixels in the input image.

[0031] The coefficients of the linear sum are determined by the formula in Figure 5, and there is a hyperparameter a. Normally, a uniform value is used for the entire image, but in this embodiment, different values ​​are used for each region. In other words, when doubling the resolution of an image with a size of 10 x 10, the linear sum is calculated at a total of 400 locations (20 x 20), and the machine learning model estimates the hyperparameter a for each of these 400 locations. In this case, the parameter that differs for each pixel is called a region-specific hyperparameter.

[0032] In the process of step S13, these regional hyperparameters are estimated using a machine learning model. That is, as shown in FIG. 6, regional hyperparameters are estimated from the input image. This is just like an existing high-resolution algorithm, generating an image with higher resolution than the input image. FIG. 6 is a diagram for explaining the estimation of regional hyperparameters and the high-resolution processing.

[0033] [Actions and Effects of the First Embodiment] As described above, the high definition processing device (high definition processing method) according to the first embodiment does not directly generate an image using a machine learning model, but estimates hyperparameters for image processing for each region and generates a high definition image based on the input image and the hyperparameters for each region. This makes it possible to achieve high transparency and restoration accuracy through high definition processing using the hyperparameters.

[0034] Furthermore, high-definition processing using hyperparameters can also be applied to other image processing. For example, it is possible to apply an unsharp mask (sharpening filter) after performing bicubic processing. As shown in Figure 7, the unsharp mask is determined by a hyperparameter called σ, so it can be applied in the same way. Figure 7 is a diagram explaining the linear sum of the unsharp mask. Furthermore, in cases with multiple hyperparameters, it can be similarly applied if a machine learning model that outputs multiple bands is used.

[0035] [Modification of the first embodiment] Various modifications are possible for the image processing unit 110 in the high definition processing device 100 according to the first embodiment. Two modifications will be described below as Modification 1 and Modification 2.

[0036] (Variation 1) FIG. 8 is a block diagram showing an example of the system configuration of a high definition processing device according to Modification 1 of the first embodiment.

[0037] The image processing unit 110 in the high definition processing device according to the first modification has a configuration including an image receiving unit 111, a regional image processing hyperparameter estimation unit 112, a signal processing unit 113, and an output unit 114, as well as a hyperparameter output unit 115. The hyperparameter output unit 115 has a function of outputting the regional image processing hyperparameters estimated by the regional image processing hyperparameter estimation unit 112 to the outside of the image processing unit 110.

[0038] As described above, in the high definition processing device according to the first modification, the image processing unit 110 has a function of outputting the regional image processing hyperparameters estimated by the regional image processing hyperparameter estimation unit 112. This makes it possible to check the image processing process in the image processing unit 110.

[0039] (Variation 2) FIG. 9 is a block diagram showing an example of the system configuration of a high definition processing device according to the second modification of the first embodiment.

[0040] The image processing unit 110 in the high definition processing device according to the second modification has a configuration including an image receiving unit 111, a regional image processing hyperparameter estimation unit 112, a signal processing unit 113, and an output unit 114, as well as a hyperparameter correction unit 116. The hyperparameter correction unit 116 has a function of correcting the regional image processing hyperparameters estimated by the regional image processing hyperparameter estimation unit 112 in response to a correction operation by the user.

[0041] The regional image processing hyperparameters corrected by the hyperparameter correction unit 116 are supplied to the signal processing unit 113. The signal processing unit 113 again performs processing to generate a high-definition image based on the input image and the regional image processing hyperparameters after the correction.

[0042] As described above, in the high definition processing device according to the second modification, the image processing unit 110 has a function of correcting the regional image processing hyperparameters estimated by the regional image processing hyperparameter estimation unit 112. This allows the image to be processed for high definition using the regional image processing hyperparameters corrected again in response to a correction operation by the user.

[0043] Second Embodiment The second embodiment is an example in which the quality of a high definition image generated by the high definition processing device according to the first embodiment is evaluated, and the preprocessing method is adjusted based on the processing that improves the evaluation value.

[0044] [System configuration example] 10 is a functional block diagram showing an example of the system configuration of a high definition processing device according to the second embodiment of the present invention. The high definition processing device 200 according to the second embodiment of the present invention has a system configuration including an image processing unit 210 and a database unit 220.

[0045] (Configuration example of image processing unit) The image processing unit 210 has an image reception unit 211, a regional image processing hyperparameter estimation unit 212, a signal processing unit 213, an output unit 214, a learning image processing unit 215, a model learning unit 216, an image optimization unit 217, and an image evaluation unit 218. Here, the image reception unit 211, the regional image processing hyperparameter estimation unit 212, the signal processing unit 213, and the output unit 214 basically have the same configurations as the image reception unit 111, the regional image processing hyperparameter estimation unit 112, the signal processing unit 113, and the output unit 114 in the first embodiment, and therefore a description thereof will be omitted.

[0046] The training image processing unit 215 determines image processing parameters and processes the training images. The training image processing unit 215 may use the hyperparameter σ of a Gaussian filter as the image processing parameter, or may prepare image processing parameters using a processing method when reducing the resolution of a high-resolution image. Alternatively, multiple processes may be used, and the ratio between them may be used as the parameter. Either low-resolution images or high-resolution images may be processed.

[0047] The model learning unit 216 learns a machine learning model using the training images processed by the training image processing unit 215. Here, the machine learning model is a mechanism for deriving results from input data in machine learning. Also, machine learning is a computer learning process.

[0048] The image optimization unit 217 uses the machine learning model created by the model learning unit 216 to increase the resolution of the low-resolution image for evaluation, and generates a high-resolution image.

[0049] The image evaluation unit 218 evaluates the high definition image and stores the evaluation results (for example, scores).

[0050] The image optimization unit 217 repeatedly executes the processes of the learning image processing unit 215, the model learning unit 216, and the image evaluation unit 218 so as to improve the evaluation results by the image evaluation unit 218, and ultimately outputs a highly evaluated machine learning model as the learning result.

[0051] (Database section configuration example) The database unit 220 has an image DB 221 and a machine learning model DB 222. Here, the image DB 221 and the machine learning model DB 222 basically contain the same information as the image DB 121 and the machine learning model DB 122 in the first embodiment, and therefore a description thereof will be omitted.

[0052] [Example of high-definition processing method] Next, an example of processing of a high definition processing method executed in the high definition processing device according to the second embodiment having the above configuration will be described. Fig. 11 is a flowchart showing an example of processing of a high definition processing method executed in the high definition processing device according to the second embodiment.

[0053] First, when executing this process, low-resolution images and high-resolution images to be used for learning and low-resolution images for evaluation are specified (step S21). Here, each image may be generated by degrading a high-resolution image, as in the case of learning a normal high-resolution model.

[0054] Next, the learning image processing unit 215 determines image processing parameters for processing the learning image, and processes the learning image (step S22).

[0055] Next, the model learning unit 216 learns a machine learning model using the learning image processed in the process of step S22 (step S23).

[0056] Next, the image optimization unit 217 uses the machine learning model created in step S23 to increase the resolution of the low-resolution image for evaluation specified in the processing of step S21, and generates a high-resolution image (step S24).

[0057] Next, the image evaluation unit 218 evaluates the high-definition image generated by the processing in step S24, saves the evaluation result (e.g., score) (step S25), and then determines whether the above series of processes has been repeated a predetermined number of times (step S26).

[0058] If it is determined that the process has not been repeated the predetermined number of times (NO in S26), the image processing parameters are changed so that the evaluation score increases (step S27), and then the process returns to step S22 and repeats the above-mentioned series of processes the predetermined number of times.

[0059] Then, if it is determined that the process has been repeated the predetermined number of times (YES in S26), the above-mentioned series of processes are repeated the predetermined number of times, and the machine learning model with the highest evaluation score is output as the learning result (step S28), thereby completing the series of processes for high definition according to the second embodiment.

[0060] [Actions and Effects of the Second Embodiment] As described above, in the high definition processing device according to the second embodiment, when learning a model for estimating image processing hyperparameters, the created high definition image is evaluated, and the process of changing the image processing parameters so as to improve the evaluation result is repeated a predetermined number of times. The machine learning model with the highest evaluation score is then output as the learning result. This enables processing with higher restoration accuracy to be achieved through high definition processing using hyperparameters.

[0061] <<Variations>> The present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as set forth in the claims. For example, the above-described embodiments have been described in detail and specifically to clearly explain the present invention, and are not necessarily limited to those including all of the components described. Furthermore, it is also possible to add, replace, or delete other components to or from part of the configuration of each embodiment.

[0062] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely realized in hardware, for example, by designing them as integrated circuits, etc. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. [Explanation of symbols]

[0063] 100... High definition processing device according to first embodiment, 110, 210... Image processing unit, 111, 211... Image receiving unit, 112, 212... Region-specific image processing hyperparameter estimation unit, 113, 213... Signal processing unit, 114, 214... Output unit, 115... Hyperparameter output unit, 116... Hyperparameter correction unit, 120, 220... Database unit, 121, 221... Image DB, 122, 222... Machine learning model DB, 200... High definition processing device according to second embodiment, 215... Learning image processing unit, 216... Model learning unit, 217... Image optimization unit, 218... Image evaluation unit

Claims

1. A high definition processing device that reconstructs an input image based on function approximation to increase the definition of the image, a hyperparameter estimation unit that estimates hyperparameters when performing function approximation for each pixel or region of the input image; a signal processing unit that uses a machine learning model to increase the resolution of the input image based on the hyperparameters estimated by the hyperparameter estimation unit; A high definition processing device comprising:

2. a hyperparameter output unit that outputs the hyperparameters estimated by the hyperparameter estimation unit; The high definition processing device according to claim 1 .

3. The hyperparameter estimation unit further includes a hyperparameter correction unit that corrects the hyperparameters estimated by the hyperparameter estimation unit. The high definition processing device according to claim 1 .

4. The signal processing unit performs image processing using Bicubic hyperparameters. The high definition processing device according to claim 1 .

5. an image evaluation unit that evaluates the high-definition image created by the signal processing unit; and an image optimization unit that repeats a process of changing image processing parameters for processing training images a predetermined number of times so as to improve the evaluation result by the image evaluation unit, and outputs a machine learning model with a high evaluation result as a training result. The high definition processing device according to claim 1 .

6. A high definition processing method for reconstructing an input image based on function approximation to increase its definition, comprising: Estimating hyperparameters for function approximation for each pixel or region of the input image; Based on the estimated hyperparameters, the input image is refined using a machine learning model. High definition processing method.

Images