Image processing apparatus and method, and storage medium

Abstract

Provided is an image processing device that uses a first image group and improves the resolution of an image in a second image group having fewer high-frequency components than the first image group. The image processing device: selects, based on an image to be improved in resolution selected from the second image group, supervised data for learning from supervised data including images included in the first image group as one of a pair of images; learns a learning model using the selected supervised data; infers a high-frequency component of the image to be improved in resolution using the learning model; and generates an image improved in resolution based on the image to be improved in resolution and the inferred high-frequency component.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and method for resolving image groups using machine learning, as well as a storage medium. Background Technology

[0002] Resolution enhancement techniques using machine learning refer to techniques that generate high-resolution images by inferring high-frequency components that cannot be interpolated using linear interpolation of pixel values ​​during resolution conversion along with image magnification. Using resolution enhancement techniques, firstly, a learning model is generated using image group G and degraded images obtained by degrading individual images in image group G using arbitrary methods as supervised data. The learning model is generated by learning the differences between the pixel values ​​of the original image and the degraded image and updating the parameters that preserve the resolution enhancement processing. As a result of inputting an image H with insufficient high-frequency components into the learning model generated in the aforementioned manner, high-frequency components are obtained through inference by the learning model. High-resolution images can be generated by superimposing the inferred high-frequency components onto image H. In the case of applying resolution enhancement processing to moving images, high-resolution moving images can be generated by inputting all frames one frame at a time into the learning model.

[0003] Typically, when providing products or services utilizing learning models, the process of collecting supervised data and generating the learning model is performed by the developer, and the generated learning model is provided to the user. Therefore, the content of the motion image input by the user is unknown during the learning process. In view of this, in order to enable inference with uniform accuracy for various inference object motion images, the developer prepares a large number of images of various types with balanced image patterns as supervised data and repeatedly learns from these images. For example, Patent Document 1 describes a technique for performing resolution enhancement processing on motion images using a learning model that has learned from various types of images. However, due to the existence of various types of supervised data, very little supervised data has a high similarity to the inference object motion image Q specified by the user. When using such a learning model, the result of learning images with low similarity to the inference object motion image Q is reflected in the inference process. As a result, for example, the resolution is improved only in the sense of edge enhancement of the subject, and it is difficult to accurately infer high-frequency components such as fine patterns of the subject; the inference accuracy cannot be said to be high. As an example of a system for solving the aforementioned problems, Patent Document 2 describes a method in which the user side learns by using only images that are similar to the motion images of the inferred object in terms of shooting location and shooting conditions as supervisory data, thereby obtaining high-definition motion images compared to the case of learning various types of images.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-204167

[0007] Patent Document 2: Japanese Patent Application Publication No. 2019-129328 Summary of the Invention

[0008] The problem the invention aims to solve

[0009] According to Patent Document 2, learning is performed using supervised data that has the same camera location but different camera times. More specifically, videos previously captured in a portion S of a route bus are accumulated and learned, and the resulting learned model is used to perform inference on the real-time video of interval S. In this case, since only the capture results in interval S are used as supervised data, these capture results represent a group of images with a relatively high similarity to the inference object, and thus an improvement in inference accuracy can be expected. However, among the images captured in interval S, for example, the image at the beginning of interval S and the image at the end of interval S are different in terms of camera location; these two images show completely different subjects, and it cannot be said that these two images have a high similarity to each other. This is a factor that reduces the inference accuracy for the entire interval S. In addition, the past video used as supervised data and the real-time video used as the inference object can be videos of the same location but showing different subjects. For subjects that are not learned, no accurate inference is performed; this is also a factor that reduces the inference accuracy.

[0010] Furthermore, Non-Patent Document 2 describes a method that classifies past videos into multiple groups based on camera conditions such as weather, generates multiple learning models by independently learning from data from each group, and switches the learning model to be used based on the camera conditions of the real-time video. This technique can suppress the decrease in inference accuracy attributable to different camera conditions. However, even when conditions such as weather are the same, the slightest difference in values ​​such as illumination can cause the supervised data and the inferred object to have different frequency components, and it cannot be said that the decrease in inference accuracy can be sufficiently suppressed. Due to these factors, even using the technique of Patent Document 2, it cannot be said that the inference accuracy of high-frequency components is sufficient.

[0011] One aspect of the present invention provides a technique for resolving images with high precision.

[0012] Solution for solving the problem

[0013] An image processing apparatus according to one aspect of the present invention includes the following configuration. Specifically, the image processing apparatus uses a first image group to upscale an image of a second image group, the second image group comprising fewer high-frequency components compared to the first image group, and includes: a selection unit for selecting, based on a high-resolution object image selected from the second image group, supervisory data to be used in learning from supervisory data comprising images included in the first image group as one of image pairs; an inference unit for learning a learning model using the supervisory data selected by the selection unit, and inferring the high-frequency components of the high-resolution object image using the learning model; and a generation unit for generating a high-resolution image based on the high-resolution object image and the high-frequency components inferred by the inference unit.

[0014] The effects of the invention

[0015] According to the present invention, images can be rendered in high definition with high precision.

[0016] Other features and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings. Note that throughout the drawings, the same reference numerals denote the same or similar components. Attached Figure Description

[0017] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the invention and, together with the specification, serve to explain the principles of the invention.

[0018] Figure 1 This is a block diagram illustrating the structure of an image processing apparatus according to a first embodiment.

[0019] Figure 2 This is a diagram illustrating the functional structure of the image processing apparatus according to the first embodiment.

[0020] Figure 3 This is a diagram illustrating the functional structure of the image processing apparatus according to the first embodiment.

[0021] Figure 4 This is a diagram illustrating an example of the frame structure of an input moving image according to the first embodiment.

[0022] Figure 5 This is a flowchart of the process for obtaining supervisory data candidates according to the first embodiment.

[0023] Figure 6 This is a diagram illustrating an exemplary data structure of the candidate database according to the first embodiment.

[0024] Figure 7 This is a flowchart of the high-definition motion image generation process according to the first embodiment.

[0025] Figure 8 is a schematic diagram for illustrating the learning / inference process according to the first embodiment.

[0026] Figure 9 is a diagram showing an example of the frame structure of an input moving image according to the second embodiment.

[0027] Figure 10 is a flowchart of supervisory data candidate acquisition processing according to the second embodiment.

[0028] Figure 11 is a diagram showing an example of the frame structure of an input moving image according to the third embodiment.

[0029] Figure 12 is a flowchart of supervisory data candidate acquisition processing according to the third embodiment.

[0030] Figure 13 is a diagram showing an example of the frame structure of an input moving image according to the fifth embodiment.

[0031] Figure 14 is a diagram for illustrating the functional configuration of an image processing apparatus according to the fifth embodiment.

[0032] Figure 15 is a flowchart of high-definition moving image generation processing according to the fifth embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in these embodiments, but the invention is not limited to requiring all such features, and a plurality of such features may be appropriately combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted.

[0034] <First Embodiment>

[0035] >(Description of Overview of Image Processing Apparatus)

[0036] Two moving images (moving image A and moving image B) captured simultaneously by the same imaging apparatus are input to the image processing apparatus of the first embodiment. The relationship between the resolution XA / frame rate FA of moving image A and the resolution XB / frame rate FB of moving image B is "XA>XB and FA<FB". The image processing apparatus has the following function (high-definition moving image generation function): generating a learning model using frames of moving image A and moving image B, and generating, by inference using the generated learning model, moving image C having resolution XA / frame rate FB from moving image B.

[0037] (Description of the structure of an image processing device)

[0038] Figure 1 This is a block diagram illustrating an exemplary hardware structure of an image processing apparatus according to a first embodiment. The control unit 101 is a computing device such as a CPU, and performs various types of functions by expanding a program stored in ROM 102 into the working area of ​​RAM 103 and executing that program. The control unit 101 can be used as various functional blocks, such as those described below. Figure 2 The analysis unit 111 and the decoding motion image generation unit 112, which will be described below, and the following will be used Figure 3 The candidate acquisition unit 113 and the supervision data extraction unit 114 are described below. ROM 102 stores the control program executed by control unit 101. RAM 103 is used as working memory for control unit 101 to execute programs, temporary storage area for various types of data, etc.

[0039] Decoding unit 104 decodes motion pictures or image data compressed according to the encoding format defined by the Moving Picture Experts Group (hereinafter referred to as MPEG) into uncompressed data. Learning and inference unit 105 includes functional blocks for generating / updating a learning model using the input supervised data (hereinafter referred to as MPEG). Figure 3 The learning unit 151 is described below. Furthermore, the learning inference unit 105 includes a functional block (hereinafter referred to as the learning inference unit) for analyzing the input image and inferring high-frequency components using a learning model generated through learning, thereby generating a high-resolution image corresponding to the input image. Figure 3 The inference unit 152 described herein. In this embodiment, a CNN model used for resolution enhancement processing based on a convolutional neural network (hereinafter referred to as CNN) is used as the learning model. This includes magnifying the input image through linear interpolation, generating high-frequency components to be added to the magnified image, and combining them.

[0040] Storage unit 106 comprises a storage medium (such as a hard disk drive (HDD) and memory card) that is attachably and removably connected to image processing device 100, and a storage medium control device for controlling the storage medium. According to commands from control unit 101, the storage medium control device controls the initialization of the storage medium, data transfer between the storage medium and RAM 103 for data reading and writing purposes, etc. Bus 107 is an information communication path for connecting discrete functions. Control unit 101, ROM 102, RAM 103, decoding unit 104, learning and inference unit 105, and storage unit 106 are connected in a manner that allows them to communicate with each other.

[0041] Note that the hardware blocks and functional blocks executing within these hardware blocks described in this embodiment do not necessarily have to be configured as described above. For example, two or more of the blocks in control unit 101, decoding unit 104, and learning inference unit 105 can be implemented by a single hardware component. Furthermore, the function of one or more functional blocks can be executed via the cooperative operation of several hardware components. Additionally, each functional block can be implemented by a CPU executing a computer program expanded to memory, or by dedicated hardware. Furthermore, it is permissible to employ a configuration where a portion of the discrete functional blocks resides in a cloud server and the data of the processing results is transmitted via communication. For example, decoding unit 104 can be implemented by the same CPU as control unit 101, or by a different CPU. Alternatively, decoding unit 104 can be implemented by a GPU for operating upon receiving instructions from control unit 101. Alternatively, decoding unit 104 can be implemented by hardware processing executed by electronic circuitry configured for decoding processing. Furthermore, for example, learning inference unit 105 can be implemented by the same CPU as control unit 101, or by a different CPU. Alternatively, the learning and inference unit 105 may be implemented by a GPU for operating upon receiving instructions from the control unit 101. Alternatively, the learning and inference unit 105 may be implemented by hardware processing performed by electronic circuitry configured for learning and inference.

[0042] (The data stored in the storage medium, and the methods for decoding and unfolding that data)

[0043] Figure 2This diagram illustrates the functional blocks (analysis unit 111 and decoding motion image generation unit 112) in control unit 101 that perform the processing for expanding compressed motion image data. Motion image a and motion image b, which serve as input data for the high-definition motion image generation process, are stored in storage unit 106. Here, a motion image represents a collection of one or more image data that are sequentially linked in time. In this embodiment, motion image a and motion image b are simultaneously captured by a camera device including an image sensor and compressed using the MPEG method. Motion image a and motion image b may be generated by applying interval culling and downsampling processing to individual images captured by a single image sensor, or they may be generated by capturing the same subject using image sensors with different resolutions and frame rates. Hereinafter, it is assumed that motion image a and motion image b are two sets of images obtained by performing different types of image processing on an image captured by an image sensor included in a camera device. The motion image data of motion image a and motion image b are each compressed using the MPEG method, multiplexed together with the capture time information, and stored in MP4 format. Note that any format other than the one described above can be used, as long as the format enables the image data and the corresponding camera time information to be obtained from the storage unit 106.

[0044] The analysis unit 111 has the following functions: parsing the motion image data (in this example, an MP4 file) stored in the storage unit 106, and calculating the storage location of the included compressed image data and the time information registered as metadata within the file. According to the MP4 format, location information indicating the storage location of discrete frame data and camera time information within the file is stored in the Moov portion. The analysis unit 111 expands the Moov portion of motion image a from the storage unit 106 to RAM 103, parses the Moov portion, and generates a table Pa including the frame number of motion image a, the location information of the frame data, and the location information of the camera time. Similarly, the analysis unit 111 parses the Moov portion of motion image b and generates a table Pb including the frame number of motion image b, the location information of the frame data, and the location information of the camera time. Tables Pa and Pb are stored in RAM 103.

[0045] To use motion image a and motion image b in high-definition motion image generation processing, a process is needed to convert motion image a and motion image b into an uncompressed format. For example... Figure 2As shown, the decoding motion image generation unit 112 of the control unit 101 generates motion image A and motion image B by decoding motion image a and motion image b, and stores motion image A and motion image B in the storage unit 106. More specifically, the decoding motion image generation unit 112 refers to tables Pa and Pb stored in RAM 103, and sequentially inputs the frame data of motion image a and motion image b stored in the storage unit 106 to the decoding unit 104. The decoding motion image generation unit 112 multiplexes the uncompressed frame data output from the decoding unit 104 together with the camera timing information obtained by referring to tables Pa and Pb, and stores the multiplexing result in the storage unit 106. In the following, it is assumed that motion image A is obtained by decoding motion image a, and motion image B is obtained by decoding motion image b. Furthermore, the motion image decoding unit 112 generates a table PA and stores it in RAM 103. Table PA includes the frame number of motion image A, location information indicating the storage location of frame data, and location information indicating the storage location of the recording time. Similarly, the motion image decoding unit 112 generates a table PB and stores it in RAM 103. Table PB includes the frame number of motion image B, location information indicating the storage location of frame data, and location information indicating the storage location of the recording time. Figure 4 An example of the frame structure for moving image A and moving image B is shown. Figure 4 In the image, the total number of frames in motion image A is n, and the total number of frames in motion image B is m. The frame pairs indicated by the dashed lines (e.g., A1 and B2, A2 and B5, and A3 and B8) are frame pairs with the same shooting time information, which means that the images of such frames were captured at the same timing.

[0046] Next, a description of the processing for generating high-definition images according to this embodiment will be given. This processing is broadly divided into two types: "supervised data candidate acquisition processing" and "high-definition moving image generation processing". Figure 3 This is a diagram illustrating the operation of functional blocks in image processing within the image processing apparatus 100 according to the first embodiment. (As used...) Figure 2The motion image A and motion image B are stored in storage unit 106, and tables PA and PB are stored in RAM 103. The "supervisory data candidate acquisition process" is performed by candidate acquisition unit 113. On the other hand, the "high-definition motion image generation process" is performed by supervisory data extraction unit 114, learning unit 151, and inference unit 152. Candidate acquisition unit 113 generates a candidate database D1 by extracting candidate frame pairs to be used as supervisory data from frame groups of motion image A and frame groups of motion image B. Frame By, as a high-resolution object and a high-definition object, is obtained from frame groups of motion image B. To generate a learning model suitable for inferring the high-frequency components of frame By, supervisory data extraction unit 114 extracts more suitable supervisory data from the supervisory data candidates registered in candidate database D1 and generates a supervisory data database D2. Learning unit 151 of learning inference unit 105 generates a learning model M for frame By using supervisory data database D2. Inference unit 152 performs high-resolution processing on frame By, which is a high-resolution object, by inputting frame By into the learning model M generated by learning unit 151. The following is a more detailed explanation of "supervised data candidate acquisition processing" and "high-resolution motion image generation processing".

[0047] (Supervision data candidate acquisition and processing)

[0048] In the supervision data candidate acquisition process, the control unit 101 (candidate acquisition unit 113) generates a candidate database D1. In the first embodiment, the candidate acquisition unit 113 obtains pairs of frames of motion image A and frames of motion image B that have the same shooting time from the frames of motion image A and motion image B as supervision data candidates. Specifically, it obtains each pair of frames ( ) in motion image A and motion image B that have the same shooting time. Figure 4 The frames indicated by the dashed lines are used as candidates for supervision data. Before performing the learning process described later, the candidate acquisition unit 113 investigates which frames can be used as supervision data, constructs a candidate database D1, and registers the results of the investigation in the candidate database D1. Figure 6 An exemplary data structure for candidate database D1 is shown. Frame groups TA (which can be used as supervisory data) from frame group A of motion picture A and frame groups TB (which can be used as supervisory data) from frame group B are registered in candidate database D1, with their frame numbers in the respective motion picture files. Here, pairs of frames with the same shooting time (pairs of their frame numbers) are registered in association with each other using a unique index I within candidate database D1. For example, in… Figure 4 In the motion images A and B shown, the frame pairs A1 and B2, A2 and B5, and A3 and B8 (subsequent pairs are omitted) are combinations of frames captured at the same time. Figure 6In the candidate database D1 shown, these pairs are stored using frame numbers, and a unique index I is appended to these pairs. In this way, the obtained supervisory data candidates are managed by the candidate database D1.

[0049] use Figure 5 The flowchart below further explains the details of the above-described supervisory data candidate acquisition process. In S501, the candidate acquisition unit 113 selects a frame from the frames of the motion image A and obtains the time information corresponding to the selected frame from the table PA. In this embodiment, frames are selected sequentially starting from the beginning of the motion image A stored in the storage unit 106. More specifically, the candidate acquisition unit 113 selects a frame sequentially starting from the beginning of the motion image A stored in the storage unit 106. Hereinafter, the selected frame is referred to as frame Ax. The candidate acquisition unit 113 reads the time information corresponding to frame Ax from the storage unit 106 with reference to the table PA stored in the RAM 103 and transmits the time information to the RAM 103.

[0050] In S502, the candidate acquisition unit 113 compares the time information of frame Ax read in S501 with the time information of each frame of the moving image B. Specifically, the candidate acquisition unit 113 refers to the position information of the camera time stored in table PB to sequentially obtain the camera time information of each frame of the moving image B from storage unit 106, and compares this camera time information with the time information of frame Ax. In S503, the candidate acquisition unit 113 obtains a frame of the moving image B that has the same camera time as frame Ax, and uses the obtained frame as frame Bx.

[0051] In S504, the candidate acquisition unit 113 appends the unique index Ix from the candidate database D1 to the combination of frames Ax and Bx, and registers them in the candidate database D1. Specifically, the candidate acquisition unit 113 issues the unique index Ix to the combination of frames Ax and Bx, and registers the index Ix, the frame number of frame Ax in moving image A, and the frame number of frame Bx in moving image B in the candidate database D1.

[0052] In S505, the candidate acquisition unit 113 determines whether the processing described in S501 to S504 has been completed for each frame of the motion image A. If it is determined that the processing has been completed ("Yes" in S505), the processing ends. If it is determined that the processing has not been completed ("No" in S505), the processing returns to S501, and the above processing is performed for the next frame of the motion image A. As a result of the above processing, a candidate database D1 is generated.

[0053] Note that in this embodiment, although the frame pairs to be registered with the candidate database D1 are determined by comparing the camera snapshot times in S502, this is not intended to be limited to this. For example, it is permissible to reduce frame Ax to resolution XB, use an index to indicate the similarity between the images of the reduced frame Ax and the individual frames of the moving image B to determine similarity, and use the result of this determination to select the frame pairs to be registered with the candidate database D1. In this case, the candidate acquisition unit 113 has a similarity determination function that determines similarity by comparing two or more image data. Note that, for example, the structural similarity index (SSIM) can be used as an index to indicate the similarity between images. Furthermore, although the image of frame Ax is reduced to resolution XB when obtaining the index used to indicate similarity, this is not intended to be limited to this. The image of frame Ax may not be reduced, and the reduced resolution may be anything other than XB.

[0054] (High-definition motion image generation and processing)

[0055] Next, a description of the high-definition moving image generation process performed by the control unit 101 (supervisory data extraction unit 114) and the learning and inference unit 105 (learning unit 151 and inference unit 152) will be given. First, refer to Figure 3 This section provides an overview of the high-definition moving image generation process. The supervised data extraction unit 114 selects supervised data suitable for the learning model used in "inferring the object frame By" from the candidate database D1, and generates a supervised data database D2. Figure 3 (This will be referenced later) Figure 7 (Details will be provided for S702 and S703). The learning unit 151 uses the extracted supervised data to generate a learning model (S704). Then, the inference unit 152 performs high-definition processing by inferring the high-frequency components of the inference object frame By using the learning model (S705), and obtains a frame (image) Cy as the result of making the inference object frame By high-definition. Note that before the high-definition moving image generation process begins, the control unit 101 generates a moving image C in the storage unit 106. At the start of high-definition moving image generation, the moving image C does not contain frame data and is in a blank state. The inference unit 152 sequentially stores the generated frame Cy into the moving image C.

[0056] The following will be referenced Figure 7The flowchart below will illustrate the high-definition moving image generation process in more detail. In S701, the supervisory data extraction unit 114 reads a frame from the moving image B as a high-definition object frame. In this embodiment, the supervisory data extraction unit 114 reads the frames sequentially from the beginning of the moving image B stored in the storage unit 106. Hereinafter, the frame read in S701 will be referred to as frame By. More specifically, the supervisory data extraction unit 114 refers to table PB to read the frame data and camera timing information of frame By from the storage unit 106 and transmits them to RAM 103.

[0057] In S702, the supervisory data extraction unit 114 extracts frames from the supervisory data candidate TB registered in the candidate database D1, and registers the extracted frames in the supervisory data database D2. These frames have a shooting time that differs from the shooting time of frame By by an amount smaller than a threshold predetermined by the system. For example, the single-frame display period of motion image A (with a frame rate XA as the single-frame display period) can be used as the threshold. The structure of the supervisory data database D2 is similar to that of the candidate database D1 (see...). Figure 6 The structure is similar. Specifically, firstly, the supervisory data extraction unit 114 obtains the time information of frame group TB registered in candidate database D1 by referring to the position information in table PB. The supervisory data extraction unit 114 compares the obtained time information with the shooting time of frame By, extracts frames from frame group TB that differ from the shooting time by an amount smaller than a threshold, and registers the extracted frames in supervisory data database D2 in RAM 103. In the following, the frame group of motion image B that has been registered in supervisory data database D2 as a result of the aforementioned processing is referred to as UB. Note that in this embodiment, although frame groups with shooting times that differ from the shooting time of frame By by an amount smaller than a threshold are extracted from candidate database D1 when constructing supervisory data database D2, it is not intended to be limited to this. Frame group UB can be extracted using indicators that indicate the similarity to frame By. For example, the supervisory data extraction unit 114 can use SSIM to extract frame groups from frame group TB that have an index of similarity to frame By that is higher than a threshold predetermined by the system, and register the extracted frame groups as frame group UB.

[0058] In S703, the supervisory data extraction unit 114 registers frames from the candidate database D1 that are paired with frames of frame group UB in frame group TA in the supervisory data database D2. Specifically, referring to the candidate database D1 in RAM 103, the supervisory data extraction unit 114 registers frames from frame group TA that are associated with frames of frame group UB using index I in the supervisory data database D2. At this time, the combination of the two associated frames remains unchanged, and a unique index J from the supervisory data database D2 is appended to each combination. In the following text, the frame group of motion image A that has been registered in the supervisory data database D2 is referred to as UA.

[0059] In S704, the learning unit 151 uses the supervision data (frame group UA and frame group UB) registered in the supervision data database D2 to learn and generate a learning model M. Figure 8 This diagram schematically illustrates the learning model generation function of learning unit 151. The learning model generation function includes a learning process and an inference process; furthermore, the inference process can be categorized into a feature extraction process using filters including a CNN and a reconstruction process. First, in the feature extraction process, learning unit 151 inputs an image from frame group UB into a CNN (considered image E) and uses the CNN to extract convolutional features, thereby generating a large number of feature maps. Next, in the reconstruction process, learning unit 151 generates predicted high-frequency components by upsampling all feature maps via transposed convolution. In the reconstruction process, learning unit 151 further performs image reconstruction by adding the predicted high-frequency components to image E', which is the result of upscaling image E using a bicubic method, thereby generating a predicted high-resolution image G. In the learning process, learning unit 151 compares the predicted high-resolution image G generated in the above inference process with an image H included in frame group UA that corresponds to image E, and fine-tunes the learning model M based on a backpropagation method using the difference between the two. Learning unit 151 improves inference accuracy by repeating the above processing for the same image E a predetermined number of times. By performing the above processing sequence on each image of frame group UB, a learning model M suitable for inference processing of frame group UB was constructed.

[0060] As described above, the learning unit 151 reads frame data of frame pairs that have been registered as supervision data from the storage unit 106 with reference to the supervised data database D2, table PA, and table PB, and executes the learning model generation function described above. The learning unit 151 stores the learning model M generated by the learning model generation function into RAM 103.

[0061] In S705, the inference unit 152 generates a high-resolution frame Cy from frame By using the inference of the learning model M generated in S704. Specifically, first, the inference unit 152 reads the learning model M stored in RAM 103. Next, the inference unit 152 inputs the frame data (image) of frame By, which has been stored in RAM 103 in S701, into the CNN of the learning model M, and generates "the expected high-frequency components when the image of frame By has been enlarged to resolution XA". The inference unit 152 generates an image of high-resolution frame Cy with resolution XA by adding the generated high-frequency components to "the image obtained by linearly enlarging the image of frame By to resolution XA", and stores the generated image in RAM 103. Note that the above processing performed on frame By from the inference of high-frequency components to the generation of the high-resolution image is the same as the above process using... Figure 8 The inference process (feature extraction and reconstruction) described previously is handled similarly. Inference unit 152 additionally writes the frame data of the high-resolution frame Cy, already stored in RAM 103, to the end of the high-resolution motion image C in storage unit 106. Furthermore, a copy of the capture time information of By is generated and multiplexed with the motion image C as the capture time of the high-resolution frame Cy, and stored in the motion image C.

[0062] In S706, the control unit 101 determines whether the frames within the inference target range for the motion image B (which may be all frames of the motion image B, or a subset of those frames) have completed the aforementioned processing. If it is determined that the aforementioned processing has not been completed ("No" in S706), the process proceeds to S701, the supervisory data extraction unit 114 selects the next frame of the motion image B as frame By, and repeats the aforementioned processing. On the other hand, if it is determined that the aforementioned processing has been completed ("Yes" in S706), this process ends. As described above, once the high-definition motion image generation process has ended, the high-definition motion image C with resolution XA and frame rate FB is stored in the storage unit 106 in an uncompressed format.

[0063] Note that although the functional blocks are described above as being implemented solely by control unit 101 or solely by learning inference unit 105, this is not intended to be limiting. For example, the functional blocks can be implemented through coordinated operation of control unit 101 and learning inference unit 105. For example, the function of inference unit 152 can be implemented by control unit 101 and learning inference unit 105, while the processing for storing high-definition frames Cy and camera moments in motion image C in storage unit 106 can be performed by control unit 101.

[0064] Furthermore, in this embodiment, although the supervised data candidate acquisition process is performed before the learning process for the entire motion picture and the high-definition motion picture generation process, the supervised data candidate acquisition process can be performed in parallel with the execution of the high-definition motion picture generation process. Moreover, although in this embodiment a new learning model M is generated for each inference object frame in S704 and the previously generated learning model M is discarded, this is not intended to be limited to this. For example, a learning model M' that has been pre-trained externally can be loaded, and additional learning using frame group UA and frame group UB can be performed on the loaded learning model M' in S704.

[0065] As described above, the first embodiment uses a learning model M trained on an image set of near-high-resolution object images from an image set captured during the same shooting period, thus enabling high-resolution image upscaling with high accuracy. Furthermore, since image pairs from the same time points in both image sets are used as supervision data, learning can be performed with even higher accuracy.

[0066] <Second Embodiment>

[0067] In the supervisory data candidate acquisition process of the first embodiment, a combination of frames of motion image A and frames of motion image B that have the same shooting time is registered in the candidate database D1. In the case of obtaining motion image A and motion image B from motion images simultaneously captured using the same image sensor in a single camera device, as... Figure 4 As illustrated, frames with the same capture time can be obtained from moving image A and moving image B. However, this method has the potential to fail to adequately extract supervisory data candidates when moving image A and moving image B are moving images captured by multiple image sensors during the same capture time period. This is because, as... Figure 9 As illustrated, motion image B does not always include frames with the same capture time as frames of motion image A. Note that examples of structures that utilize multiple image sensors to capture motion images A and B include structures captured using a camera device including multiple image sensors, and structures captured using multiple camera devices including one or more image sensors. According to the supervisory data candidate acquisition process of the second embodiment, even if frames of motion image A and frames of motion image B do not have the same capture time, the above-mentioned problem is solved by registering combinations of frames with a time difference smaller than a pre-determined threshold in the candidate database D1.

[0068] The second embodiment is similar to the first embodiment in terms of the structure of the image processing device 100 and the high-definition image generation processing, but differs from the first embodiment in that it involves a portion of the supervision data candidate acquisition processing. Figure 10This is a flowchart illustrating the process of obtaining supervision data candidates according to the second embodiment. The following mainly describes the supervision data candidate acquisition process compared to the first embodiment. Figure 5 The differences between them.

[0069] The processing of S1001 and S1002 is the same as in the first embodiment ( Figure 5 S501 and S502 are similar to those in the first embodiment. In S1003, the candidate acquisition unit 113 obtains a frame Bx from the frames of the moving image B whose shooting time differs from that of a frame Ax of the moving image A by an amount smaller than a pre-determined threshold, and registers frame Bx in the candidate database D1 in RAM 103. Note that, for example, the display period per frame of the moving image B at a frame rate XB can be used as such a threshold. The subsequent processing in S1004 and S1005 is the same as in the first embodiment. Figure 5 The S504 and S505 are similar.

[0070] As described above, according to the second embodiment, even if motion image A and motion image B are obtained through multiple image sensors, supervisory data candidates can be appropriately extracted.

[0071] <Third Embodiment>

[0072] In the first and second embodiments, motion image A and motion image B were captured at least during the same shooting time period. Therefore, in the first and second embodiments, as... Figure 11 As shown, when motion images A and B are captured by the same or multiple cameras at different times (their capturing periods do not overlap), the process for obtaining supervised data candidates cannot produce supervised data candidates. This will be discussed regarding the use of... Figure 11 The third embodiment is illustrated by a supervisory data candidate acquisition process in which supervisory data candidates are appropriately obtained for moving images A and B. In the supervisory data candidate acquisition process of the third embodiment, an index indicating frame similarity between frames of moving image A and frames of moving image B is calculated, and frame pairs with an index equal to or greater than a threshold predetermined by the system are registered in a candidate database D1. Note that, for example, as described above, SSIM can be used as the index indicating frame similarity. Note that when determining similarity, the image of a frame of moving image A is downscaled to a resolution XB, and the downscaled image and the images of discrete frames of moving image B are used to calculate the index indicating similarity. However, in this case, the image of a frame of moving image A may not be downscaled, and the downscaled resolution may be anything other than XB.

[0073] Figure 12 This is a flowchart illustrating the process of obtaining supervisory data candidates according to the third embodiment. (Reference) Figure 10The flowchart below mainly describes the process of obtaining candidate supervision data in the first embodiment. Figure 5 The differences between them.

[0074] In S1201, the candidate acquisition unit 113 selects a frame from the frames of the motion image A and reads the frame data of the selected frame. The candidate acquisition unit 113 selects a frame sequentially from the beginning of the motion image A stored in the storage unit 106 (hereinafter, the selected frame is referred to as frame Ax), and transfers the frame data of frame Ax from the storage unit 106 to the RAM 103 with reference to the table PA stored in the RAM 103.

[0075] In S1202, the candidate acquisition unit 113 calculates the similarity between frame Ax, which has been read in S1201, and discrete frames of motion image B. More specifically, referring to the position information (related to frame data) in table PB, the candidate acquisition unit 113 sequentially retrieves the frame data of each frame of motion image B from storage unit 106 back to RAM 103. Then, using the similarity index calculation function (SSIM in this embodiment), the candidate acquisition unit 113 calculates the similarity index between frame Ax and discrete frames and stores these indices in RAM 103. In S1203, the candidate acquisition unit 113 obtains the frame of motion image B with the similarity index having the highest value among the indices calculated in S1202 as frame Bx. The subsequent processing in S1204 and S1205 is the same as in the first embodiment ( Figure 5 The S504 and S505 are similar.

[0076] As described above, according to the third embodiment, even if the shooting time periods of the two image groups (moving image A and moving image B) do not overlap, appropriate supervisory data candidates can be obtained.

[0077] <Fourth Embodiment>

[0078] The fourth embodiment will be described by further improving the performance of the learning model M by taking into account image similarity in the learning processing of the first to third embodiments. As described in the first embodiment, for Figure 7 The selected frame By in S701 is used to extract appropriate supervision data, and this supervision data is used in S704 to generate / update the learning model M. During the generation / update of the learning model M, as... Figure 8As shown, backpropagation is used to tune the network parameters. In the fourth embodiment, the intensity of the tuning using backpropagation is controlled based on the frame (image E) used in the learning process and the attributes of the frame By used as a high-resolution and high-definition object (e.g., the time of capture), or based on the images of these frames. More specifically, during the learning process, the learning unit 151 determines coefficients such that the higher the similarity between the frames of the sequentially input frame group UB and the frame By, the greater the influence on the update of the network parameters through backpropagation, and the lower the similarity between them, the smaller the influence. Here, image similarity can be obtained simply based on the time difference between the frame By and the input image E, or it can be obtained by comparing the images of the two frames using SSIM or the like. For example, in the case of using the former (the method using the time difference), the following configuration can be adopted: if the time difference is less than a threshold, the tuning intensity is multiplied by a coefficient of 1, and if the time difference is equal to or greater than the threshold, the tuning intensity is multiplied by a coefficient of 0.5.

[0079] if (ABS(the time difference between By and E) < threshold {

[0080] coefficient = 1

[0081] }else{

[0082] Coefficient = 0.5

[0083] }

[0084] On the other hand, when using the latter (using a similar approach), the configuration of using the Structural Similarity Index (SSIM) as the coefficient for tuning intensity can be adopted as follows.

[0085] Coefficient = SSIM(By and E) [0 ≤ SSIM(x) ≤ 1]

[0086] Note that examples of methods to increase or decrease the degree of influence include: multiplying the update rate of the network parameters through backpropagation by the coefficient mentioned above; multiplying the number of learning loops on the input image E by the coefficient instead of multiplying the update rate of the parameters by the coefficient; and so on.

[0087] <Fifth Embodiment>

[0088] The first to third embodiments have been described with respect to the following configuration: A pair of frames from moving image A and a pair of frames from moving image B are extracted as supervisory data candidates and registered in the candidate database D1. In the fifth embodiment, moving image A' is generated by converting moving image A to the resolution XB of moving image B, and the candidate acquisition unit 113 uses moving image A and moving image A' to obtain supervisory data candidates. That is, in the fifth embodiment, the candidate acquisition unit 113 extracts frame Ax' from moving image A' that has the same frame number as frame Ax of moving image A, and registers the pair of frames Ax and Ax' as supervisory data candidates in the candidate database D1. The fifth embodiment will be described in detail below.

[0089] (Explanation of the structure of the image processing device 100)

[0090] The hardware and functional structures of the image processing device 100 are the same as those in the first embodiment. Figure 1 The hardware and functional structures are similar to those in the previous embodiment. However, the control unit 101 of the fifth embodiment also has a resolution conversion function for applying bicubic downscaling and conversion to the image resolution. When performing resolution downscaling processing on image data stored in RAM 103, this resolution conversion function calculates the pixel values ​​of the pixels to be interpolated by referring to the pixels surrounding the image data.

[0091] (The data stored in storage unit 106, and the method for decoding and expanding the data)

[0092] In the first embodiment, motion images a and b stored in storage unit 106 are converted into an uncompressed format, and the results of decoding motion image a and motion image b are stored in storage unit 106 as motion image A and motion image B, respectively. In the fifth embodiment, motion image A' is further generated by converting motion image A into a resolution XB of motion image B. More specifically, referring to table PA stored in RAM 103, control unit 101 sequentially inputs the frame data of the frames of motion image A (hereinafter referred to as frame K) stored in storage unit 106 to the resolution conversion function of control unit 101. Then, the resolution conversion function outputs frames (hereinafter referred to as frame K') including frame data with a resolution XB. Control unit 101 refers to table PA to multiplex these frames together with the shooting time information of frame K already read from storage unit 106, and stores them in storage unit 106 as frames of motion image A'. In addition, table PA', used to maintain the frame number, frame data position information, and camera timing data position information of each frame of the motion image A', is stored in RAM 103. Figure 13Examples of these moving images A, B, and A' are shown. Images (A1' to An') are generated by downscaling the individual frames of moving image A to a resolution of XB, and these images are stored as moving image A' in storage unit 106. Note that although the resolution of moving image A is downscaled to XB in the aforementioned example, this is not intended to be limiting. Moving image A' is simply converted to an image with a lower resolution than moving image A, which is sufficient. However, by using an image converted to the same resolution as the high-definition object image, a learning model more suitable for high-definition object images can be constructed.

[0093] (Supervision data candidate acquisition and processing)

[0094] Figure 14 The structure and operation of functional blocks in image processing within the image processing apparatus 100 of the fifth embodiment are illustrated. For discrete frames of moving images A and A', the candidate acquisition unit 113 obtains combinations of frames with the same frame number and registers them in the candidate database D1. More specifically, for each frame of moving image A described in table PA, the candidate acquisition unit 113 refers to table PA' to search for frames with the same frame number in moving image A'. The candidate acquisition unit 113 appends a unique index I to the combination of frames of moving images A and A' with the same frame number and registers this combination in the candidate database D1. The group of frames of moving image A and the group of frames of moving image A' already registered in the candidate database D1 will be referred to as TA and TA', respectively.

[0095] (High-definition motion image generation and processing)

[0096] refer to Figure 15 The flowchart below mainly describes the processing compared to the first embodiment. Figure 6 The differences between them.

[0097] The processing of S1501 is the same as that in the first embodiment ( Figure 7Similar to S701. In S1502, the supervisory data extraction unit 114 extracts frames from the candidate supervisory data frame group TA' registered in the candidate database D1 that have a shooting time that differs from the shooting time of frame By by an amount smaller than a threshold predetermined by the system. For example, the single-frame display period of motion image A (in the form of a single-frame display period of frame frequency XA) can be used as the threshold. The supervisory data extraction unit 114 registers the extracted frames in the supervisory data database D2. Specifically, first, the supervisory data extraction unit 114 refers to table PA' to obtain the time information of the frames registered as frame group TA'. The supervisory data extraction unit 114 registers the frames with time information in the obtained frame group TA' that differ from the shooting time of frame By by an amount smaller than the threshold in the supervisory data database D2 in RAM 103. Hereinafter, the frame group of motion image A' registered in the supervisory data database D2 is referred to as frame group UA'. Note that in this embodiment, although frames with a shooting time that differs from the shooting time of frame By by an amount smaller than a predetermined threshold are extracted from the candidate database D1, this is not intended to be limited to this. For example, it is permissible to extract frames from frame group TA' whose similarity index (e.g., SSIM) between the images of each frame in frame group TA' and the image of frame By is higher than a threshold predetermined by the system, and to register the extracted frames in the supervisory data database D2.

[0098] In S1503, the supervisory data extraction unit 114 registers the frames of frame group TA, which are associated with frames of frame group UA' using index I, in the supervisory data database D2. Specifically, referring to the candidate database D1 in RAM 103, the supervisory data extraction unit 114 registers the frames of frame group TA, which are associated with frames of frame group UB using index I, in the supervisory data database D2. At this time, the associated combinations (frame pairs) remain unchanged, and the unique index J in the supervisory data database D2 is appended to each combination. In the following text, the frame group of motion image A that has been registered in the supervisory data database D2 is referred to as frame group UA.

[0099] In S1504, learning unit 151 refers to supervised data database D2 to learn using frame group UA and frame group UA', and generates a learning model M. Specifically, first, learning unit 151 refers to supervised data database D2 and tables PA and PA' to read frame data from storage unit 106, and inputs this frame data into the learning model generation function. Learning unit 151 uses the read frame data to learn through the learning model generation function, and stores the generated learning model M as the learning result in RAM 103. Details of the training of the learning model are as described above. Figure 8 The subsequent processing in S1505 and S1506 is the same as that in the first embodiment. Figure 7(The processing is similar to that of S705 and S706.)

[0100] As described above, according to the various embodiments, the supervision data to be used in training the learning model is selected based on the high-definition object image. Therefore, the learning model trained using the selected supervision data can infer the high-frequency components of the high-definition object image with higher accuracy, and a highly accurate high-definition image can be obtained. In other words, this achieves an improvement in the accuracy of resolution enhancement techniques for motion images that enable high-definition upscaling.

[0101] Note that in the above embodiments, although the images paired with the images selected from moving image A in obtaining the supervisory data candidates are images selected from moving image B based on the shooting time or image similarity, or images obtained by reducing the resolution of the selected images, this is not intended to be limited to these. It is sufficient that the image associated with the image selected from moving image A to be used as a supervisory data candidate is an image with a lower resolution than the selected image and associated with the selected image. Whether an image is associated with the image selected from moving image A can be determined, for example, based on common characteristics such as the temperature at the time of shooting, the shooting location, and the shooting direction.

[0102] Furthermore, while the above embodiments describe a two-stage process involving the generation of a candidate database D1 and then a supervisory data database D2, it is not intended to be limited thereto. For example, the supervisory data extraction unit 114 may extract frames that can be paired with supervisory data from the motion image A based on frame By, and obtain supervisory data by using the extracted frames and frames associated with the extracted frames as pairs. However, when sequentially re-enhancing multiple images of the motion image B, it is more efficient to generate a candidate database D1, extract appropriate supervisory data from the candidate database D1 based on the high-definition object images, and use the extracted supervisory data, as described in the embodiments above.

[0103] Furthermore, although moving image a and moving image b with a lower resolution than moving image a are used as processing objects in the above embodiments, this is not intended to be limiting. For example, uncompressed moving image a and moving image b obtained by compressing and decompressing moving image a can be used as processing objects. In this case, moving image a can be stored after the frames of moving image a have been interleaved. In this way, the relationship between moving image a and moving image b used as processing objects in the above embodiments is not limited to the size of the resolution, and it is sufficient that moving image a is more vivid than moving image b. That is, it is sufficient that the image group constituting moving image a (moving image A) includes more high-frequency components compared to the image group constituting moving image b (moving image B). For example, the processing of the above embodiments can be applied as long as each image of the image group of moving image a corresponds to one or more images of the image group of moving image b, and each image of the image group of moving image a includes more high-frequency components compared to the corresponding images of the image group of moving image b.

[0104] Furthermore, although moving image data has been described above for simplicity, the various embodiments described above can also be applied to devices that can generate still images at predetermined time intervals while storing moving images, for example, in the following cases. That is, this is a case where a still image is used as data equivalent to moving image a and a moving image is used as data equivalent to moving image b. For example, it is assumed that one of the above embodiments is applied to a camera device that uses an image sensor to capture images at 60fps in Raw format with a data size of 6K. Furthermore, for example, it is assumed that data stored in a format such as JPEG or HEIF is used as a still image while maintaining the 6K size through image processing and still image compression. Furthermore, for example, it is assumed that Raw data obtained by converting the 6K data already obtained by the image sensor to a 2K data size is recorded in a format such as MP4 through image processing and moving image compression, and the stored data (2K moving image data at 60fps) is used as a moving image. Under this assumption, if a user continuously captures still images by pressing a release switch while storing 2K motion image data at 60fps on the camera device, then 6K still images are generated at intervals of, for example, 10fps, with respect to the frame rate of the motion images (60fps). By applying one of the above embodiments to still images and motion images already generated in this way, for example, data with still image quality corresponding to motion images captured during a period of time when multiple still images were captured can be generated. That is, a system can be implemented that appears to obtain motion images as if they were captured at a frame rate of 60fps with a size of 6K (which is the size of the still images). Furthermore, in this case, the camera device prepares still images and motion images, and additionally, learning and inference processing is performed, and data with still image quality corresponding to the motion images is generated within the camera device.

[0105] The present invention can be implemented by providing a program for implementing one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by reading and executing the program by one or more processors of the computer of the system or device. Furthermore, the present invention can be implemented by circuitry (e.g., an ASIC) for implementing one or more of the functions.

[0106] This invention is not limited to the embodiments described above, and various changes and modifications can be made within the spirit and scope of this invention. Therefore, the appended claims have been added to inform the public of the scope of this invention.

[0107] This application claims the benefit of priority to Japanese Patent Application No. 2021-025524, filed on February 19, 2021, and Japanese Patent Application No. 2021-195652, filed on December 1, 2021, both of which are incorporated herein by reference in their entirety.

Claims

1. An image processing apparatus for using a first image group to upscale an image of a second image group, the second image group comprising fewer high-frequency components compared to the first image group, the image processing apparatus comprising: The obtaining component is used to obtain an image pair consisting of one image each freely selected from the first image group and another image that includes fewer high-frequency components compared to the first image and is related to the first image, as supervisory data candidates; The selection component is used to select, based on the high-resolution object image selected from the second image group, the supervision data candidate obtained by the acquisition component, including an image that is used as supervision data for learning, wherein the image has a shooting time that differs from the high-resolution object image by an amount smaller than a predetermined threshold. An inference component is used to learn a learning model using supervision data selected by the selection component, and to infer high-frequency components of the high-resolution object image using the learning model. as well as A generation component is used to generate a high-definition image based on the high-definition object image and the high-frequency components inferred by the inference component.

2. The image processing apparatus according to claim 1, wherein, The obtaining component obtains the supervisory data candidate by obtaining images related to the images selected from the first image group from the second image group.

3. The image processing apparatus according to claim 2, wherein, The acquiring component obtains an image from the second image group that has the same shooting time as the first image, as the other image associated with the first image.

4. The image processing apparatus according to claim 2, wherein, The obtaining component obtains from the second image group an image whose shooting time differs from that of the first image by an amount smaller than a predetermined threshold, as the other image associated with the first image.

5. The image processing apparatus according to claim 2, wherein, The obtaining component obtains from the second image group the image that exhibits the highest similarity to the first image, as the other image associated with the first image.

6. The image processing apparatus according to claim 5, wherein, The first image group has a first resolution, and the second image group has a second resolution that is lower than the first resolution. The obtaining component obtains the similarity between the image obtained by reducing the one image to the second resolution and the images in the second image group.

7. The image processing apparatus according to claim 1, wherein, The obtaining component obtains an image obtained by reducing the resolution of the first image by shrinking it, as the other image associated with the first image.

8. The image processing apparatus according to claim 7, wherein, The first image group has a first resolution, and the second image group has a second resolution that is lower than the first resolution. The other image associated with the first image is an image obtained by reducing the first image to the second resolution.

9. The image processing apparatus according to any one of claims 1 to 8, wherein, The selection component selects candidate supervisory data, including images whose similarity to the high-resolution object image is higher than a predetermined threshold, as supervisory data to be used in the learning process.

10. The image processing apparatus according to any one of claims 1 to 8, wherein, The inference component controls the updating of parameters during the learning process via backpropagation, based on the supervised data to be used in the learning and the high-resolution object image.

11. The image processing apparatus according to claim 10, wherein, The inference component determines coefficients based on the supervised data to be used in the learning process and the high-resolution object image, and controls the amount of parameter updates through the backpropagation based on the coefficients.

12. The image processing apparatus according to claim 10, wherein, The inference component determines coefficients based on the supervised data to be used in the learning process and the high-resolution object image, and controls the number of repetitions of parameter updates through the backpropagation based on the coefficients.

13. The image processing apparatus according to claim 11 or 12, wherein, The inference component determines the coefficients based on the difference between the capture time of the image of the supervised data to be used in the learning and the capture time of the high-definition object image.

14. The image processing apparatus according to claim 11 or 12, wherein, The inference component determines the coefficients based on the similarity between the image of the supervised data to be used in the learning and the high-resolution object image.

15. The image processing apparatus according to any one of claims 1 to 8, wherein, The first image group and the second image group are two image groups obtained by performing different types of image processing on a set of images captured by an image sensor included in a camera device.

16. The image processing apparatus according to any one of claims 1 to 8, wherein, The first image group and the second image group are image groups captured by two image sensors respectively.

17. The image processing apparatus according to any one of claims 1 to 8, wherein, The frame rate of the first image group is lower than that of the second image group.

18. An image processing method for using a first image group to upscale a second image group, the second image group comprising fewer high-frequency components compared to the first image group, the image processing method comprising: The obtaining step is used to obtain an image pair consisting of one image selected by each of the first image groups and another image that includes fewer high-frequency components compared to the first image and is related to the first image, as supervisory data candidates; The selection step is used to select, based on the high-resolution object image selected from the second image group, a supervisory data candidate from the supervisory data candidate obtained in the acquisition step, which includes an image as supervisory data to be used in the learning, wherein the image has a shooting time that differs from the high-resolution object image by an amount smaller than a predetermined threshold. An inference step is used to learn a learning model using the supervision data selected in the selection step, and to infer high-frequency components of the high-resolution object image using the learning model. as well as The generation step is used to generate a high-definition image based on the high-definition object image and the high-frequency components inferred in the inference step.

19. A computer storage medium storing a program, said program being executed by a computer to cause the computer to perform the image processing method according to claim 18.

20. A computer program product comprising a program that, when executed by a computer, causes the computer to perform the image processing method according to claim 18.

Images