Image processing device, method, and program

Abstract

To provide an image processing apparatus that improves definition of an image with high accuracy by using machine learning.SOLUTION: An image processing apparatus calculates, for each of a plurality of partial areas obtained by dividing an image for which the definition is to be improved selected from a second image group with a smaller number of high frequency component than images in a first image group, similarity with a corresponding partial area of an image for which the definition is to be improved in a previous time, couples sets each composed of one or more partial areas with the similarity equal to or more than a threshold to one local area, takes partial areas with the similarity smaller than the threshold as other local areas, and determines a plurality of local areas from the image for which the definition is to be improved, creates a learning model for inferring the high frequency component for each of the plurality of local areas by using training data selected on the basis of the image for which the definition is to be improved, and infers the high frequency component for each of the plurality of local areas.SELECTED DRAWING: Figure 20

Description

[Technical Field]

[0001] The present invention relates to an image processing device, method, and program for increasing the resolution of a group of images using machine learning. [Background technology]

[0002] Super-resolution technology using machine learning generates high-resolution images by using machine learning to infer high-frequency components that cannot be compensated for by linear interpolation of pixel values ​​when enlarging an image and performing resolution conversion. In super-resolution technology, a learning model is first generated using a group of images G and degraded images of each image in G, degraded by a given method, as training data. The learning model learns the pixel value differences between the original image and the degraded image and updates its own super-resolution processing parameters. When an image H lacking high-frequency components is input to the learning model, the high-frequency components are acquired through inference using the learning model. A high-resolution image can be generated by overlaying the high-frequency components acquired through inference on image H. When performing super-resolution processing on a video, a high-resolution video can be generated by inputting all frames one by one into the learning model.

[0003] Generally, when a product or service using a learning model is provided, the developer collects training data and generates a learning model, and then provides the generated learning model to the user. Therefore, the content of the video input by the user is unknown at the time of the training process. Therefore, the developer prepares a large amount of diverse images with no bias in image patterns as training data and repeatedly trains the system to ensure uniform accuracy for all target videos.

[0004] For example, Patent Document 1 describes a technology for performing super-resolution processing on a video using a learning model that has learned from a variety of images. However, because the training data is diverse, there is very little training data that is highly similar to the inference target video Q specified by the user. When such a learning model is used, the results of learning images that are low in similarity to the inference target video Q are reflected in the inference processing. As a result, the improvement is limited to an improvement in resolution due to the emphasis on the edges of the subject, and it is difficult to accurately infer high-frequency components such as the fine patterns of the subject, and the inference accuracy cannot be said to be high.

[0005] As an example of a system for solving such problems, Patent Document 2 describes a method in which the user learns using only images that are similar in location and conditions to the video to be inferred as training data, thereby obtaining videos with higher resolution than when learning from a variety of images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-204167 [Patent Document 2] Japanese Patent Application Publication No. 2019-129328 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 2, learning is performed using training data captured at the same location but at different times. More specifically, video previously captured along a section S of a bus route is accumulated and learned, and the resulting learning model is used to perform inference on real-time video of section S. In this case, the training data is limited to images captured within section S, which are a group of images with a relatively high similarity to the inference target, and thus improved inference accuracy can be expected. However, even among the videos captured within section S, for example, the video at the start of section S and the video at the end of section S were captured in different locations, and the subjects depicted are completely different, making them less similar. This reduces the inference accuracy for the entire section S. In addition, the subjects depicted in the past video serving as training data and the real-time video of the inference target are different, even when captured at the same location. Accurate inference cannot be performed for subjects that have not been trained, which also reduces inference accuracy.

[0008] Patent Document 2 also describes a method for classifying past video footage into multiple groups based on shooting conditions such as weather, independently learning data from each group to generate multiple learning models, and then switching the learning model used depending on the shooting conditions of the real-time video. This technology also makes it possible to suppress a decrease in inference accuracy due to differences in shooting conditions. However, even if conditions such as weather are the same, even slight differences in values ​​such as illuminance will result in differences in the frequency components of the training data and the inference target, so it cannot be said that a decrease in inference accuracy can be sufficiently suppressed. Due to these factors, even the technology described in Patent Document 2 does not provide sufficient inference accuracy for high-frequency components.

[0009] According to one aspect of the present invention, an image processing device is provided that can use machine learning to improve the resolution of an image with high accuracy. [Means for solving the problem]

[0010] An image processing device according to one aspect of the present invention has the following arrangement: 1. An image processing device that uses a first image group to increase the resolution of images of a second image group that have fewer high frequency components than images of the first image group, a selection means for selecting training data to be used for learning from a plurality of training data including an image included in the first image group as one of a pair of images, based on an image to be refined selected from the second image group; a calculation means for calculating a similarity between each of a plurality of partial regions obtained by dividing the image to be refined and a corresponding partial region of the image that was the previous target of refinement; a determination means for determining a plurality of local regions from the image to be refined, wherein a set of one or more partial regions having a similarity equal to or greater than a threshold is combined into one local region, and each partial region having a similarity less than the threshold is treated as a separate local region; a model generation means for generating a learning model for inferring high frequency components for each of the plurality of local regions using the teacher data selected by the selection means; inference means for inferring high-frequency components of each of the plurality of local regions using the learning model; The image generating device includes an image generating unit that generates a high-definition image based on the image to be refined and the high-frequency components inferred by the inference unit. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image processing device that can use machine learning to improve the resolution of an image with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating the functional configuration of an image processing apparatus according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a frame configuration of an input moving image according to the first embodiment. [Figure 4]FIG. 1 is a diagram illustrating the functional configuration of an image processing apparatus according to a first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of the data configuration of a candidate database according to the first embodiment. [Figure 6] 10 is a flowchart of a teacher data candidate acquisition process according to the first embodiment. [Figure 7] 4 is a flowchart of high-definition video generation processing according to the first embodiment. [Figure 8] FIG. 2 is a schematic diagram illustrating a learning / inference process according to the first embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a frame configuration of an input moving image in the second embodiment. [Figure 10] 10 is a flowchart of a teacher data candidate acquisition process according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a frame configuration of an input moving image according to the third embodiment. [Figure 12] 11 is a flowchart of a teacher data candidate acquisition process according to the third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a frame configuration of a moving image according to the fifth embodiment. [Figure 14] FIG. 11 is a diagram for explaining the functional configuration of an image processing apparatus according to a fifth embodiment. [Figure 15] 13 is a flowchart of high-definition moving image generation processing in the fifth embodiment. [Figure 16] 10 is a flowchart of high-definition moving image generation processing in the sixth, seventh, eighth, and ninth embodiments. [Figure 17] FIG. 20 is a diagram showing an example of learning / inference processing in the sixth embodiment. [Figure 18] 13 is a flowchart of high-definition moving image generation processing in the eighth embodiment. [Figure 19] FIG. 23 is a diagram showing an example of region selection of training data according to the ninth embodiment. [Figure 20] 23 is a flowchart of a local region extraction process according to the tenth embodiment. [Figure 21] 20A to 20C are views for explaining the concept of local region extraction processing in the tenth embodiment.

Best Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0014] <First Embodiment> (Overview of the Image Processing Apparatus) The image processing apparatus according to the first embodiment takes as inputs two videos A and B that are simultaneously captured by the same imaging device. The relationship between the resolution XA and frame rate FA of video A and the resolution XB and frame rate FB of video B is "XA > XB and FA < FB". The image processing apparatus has a function (high-definition video generation function) of generating a learning model using the frames of videos A and B and generating video C with resolution XA and frame rate FB from video B by inference using the generated learning model.

[0015] (Description of the Configuration of the Image Processing Apparatus) FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus 100 according to the first embodiment. The control unit 101 is an arithmetic device such as a Central Processing Unit (hereinafter, CPU). The control unit 101 realizes various functions by expanding and executing a program stored in a Read Only Memory (hereinafter, ROM) 102 in a work area of a Random Access Memory (hereinafter, RAM) 103. The control unit 101 can function as each functional block such as an analysis unit 211 and a decoded video generation unit 212 described later with reference to FIG. 2, and a candidate acquisition unit 413 and a teacher data extraction unit 414 described later with reference to FIG. 4. The ROM 102 stores a control program executed by the control unit 101. The RAM 103 is used for a work memory for the control unit 101 to execute a program and a temporary storage area for various data.

[0016] The decoding unit 104 decodes video or image data compressed in an encoding format defined by the Moving Picture Experts Group (hereinafter abbreviated as MPEG) into uncompressed data. The learning and inference unit 105 has a functional block (learning unit 451, described later in FIG. 4) that receives training data and generates / updates a learning model. The learning and inference unit 105 also has a functional block (inference unit 452, described later in FIG. 4) that analyzes an input image using the learning model generated by learning and infers high-frequency components to generate a high-resolution image of the input image. In this embodiment, a CNN model for super-resolution processing based on a convolutional neural network (hereinafter abbreviated as CNN) is used as the learning model. This model enlarges the input image by linear interpolation, generates high-frequency components to be added to the enlarged image, and then adds and combines the two.

[0017] The recording unit 106 is composed of a recording medium, such as a hard disk drive (HDD) or a memory card, detachably connected to the image processing device 100, and a recording medium control device that controls the recording medium. The recording medium control device controls initialization of the recording medium and data transfer between the recording medium and RAM 103 for reading and writing data, in accordance with commands from the control unit 101. The bus 107 is an information communication path connecting each function. The control unit 101, ROM 102, RAM 103, decoding unit 104, learning and inference unit 105, and recording unit 106 are connected to each other so that they can communicate with each other.

[0018] Note that the hardware blocks described in this embodiment and the functional blocks realized therein do not necessarily have to be configured as described above. For example, two or more of the control unit 101, the decoding unit 104, and the learning and inference unit 105 may be realized by a single piece of hardware. Furthermore, the function of one functional block or the functions of multiple functional blocks may be performed by the coordinated operation of several pieces of hardware. Furthermore, each functional block may be realized by a CPU executing a computer program deployed in memory, or by dedicated hardware. Furthermore, some of the functional blocks may reside on a cloud server, and processing result data may be transferred via communication. For example, the decoding unit 104 may be realized by the same CPU as the control unit 101, or by a different CPU. Alternatively, the decoding unit 104 may be realized by a GPU that operates in response to instructions from the control unit 101. Alternatively, the decoding unit 104 may be realized by hardware processing using an electronic circuit configured for combined processing. Furthermore, for example, the learning and inference unit 105 may be realized by the same CPU as the control unit 101, or by a different CPU. Alternatively, the learning and inference unit 105 may be realized by a graphics processing unit (GPU) that operates in response to instructions from the control unit 101. Alternatively, the learning and inference unit 105 may be realized by hardware processing using an electronic circuit configured for learning and inference.

[0019] (Data recorded on the recording medium and its decryption and expansion methods) FIG. 2 is a diagram illustrating functional blocks for the control unit 101 (analysis unit 211 and decoded video generation unit 212) that perform decompression processing of compressed video data. The recording unit 106 stores video a and video b, which are input data for the high-definition video generation processing. Here, video refers to one or more temporally continuous image data. In this embodiment, video a and video b are simultaneously captured by an imaging device having an image sensor and compressed using the MPEG format. Video a and video b may be generated by thinning and reducing images captured by a single image sensor, or by capturing the same subject using image sensors with different resolutions and frame rates. Hereinafter, video a and video b are assumed to be two groups of images obtained by performing different image processing on a single image captured by a single image sensor in a single imaging device. The video data for video a and video b are each compressed using the MPEG format, multiplexed with shooting time information, and stored in MP4 format. Note that other formats may be used as long as the image data and corresponding shooting time information can be acquired as a pair from the recording unit 106.

[0020] The analysis unit 211 has the function of parsing video data (in this example, an MP4 format file) recorded in the recording unit 106 and calculating the storage locations within the file of the included compressed image data and the time information registered as metadata. In the MP4 format, location information indicating the recording location within the file of each frame of data and shooting time information is recorded in the Moov portion. The analysis unit 211 extracts the Moov portion of video a from the recording unit 106 into RAM 103 and parses it, generating table Pa having frame numbers in video a, location information indicating the storage location of the frame data, and location information indicating the storage location of the shooting time. The analysis unit 211 also parses the Moov portion of video b in the same way and generates table Pb having frame numbers in video b, location information indicating the storage location of the frame data, and location information indicating the storage location of the shooting time. Tables Pa and Pb are stored in RAM 103.

[0021] To use the moving images a and b in the high-definition moving image generation process, the moving images a and b must be converted into uncompressed formats. As shown in FIG. 2, the decoded moving image generation unit 212 of the control unit 101 decodes the moving images a and b to generate moving images A and B, which are then recorded in the recording unit 106. More specifically, the decoded moving image generation unit 212 references tables Pa and Pb stored in RAM 103 and sequentially inputs the frame data of the moving images a and b stored in the recording unit 106 to the decoding unit 104. The decoded moving image generation unit 212 multiplexes the uncompressed frame data output by the decoding unit 104 with shooting time information obtained by reference to tables Pa and Pb, and records the multiplexed data in the recording unit 106. Hereinafter, it is assumed that moving image A is obtained by decoding moving image a, and moving image B is obtained by decoding moving image b. The decoded moving image generation unit 212 also generates table PA, which contains frame numbers in moving image A, location information indicating the storage locations of the frame data, and location information indicating the storage locations of the shooting times, and records the table PA in RAM 103. Similarly, the decoded video generation unit 212 generates a table PB having frame numbers in video B, location information indicating the storage location of frame data, and location information indicating the storage location of the shooting time, and records this in RAM 103. An example of the frame structure of video A and video B is shown in Figure 3. In Figure 3, the total number of frames of video A is n, and the total number of frames of video B is m. Frame pairs indicated by dashed lines (multiple pairs of images A1 and B2, A2 and B5, A3 and B8, etc.) are paired frames with the same shooting time information, indicating that the images of these frames were shot at the same timing. Also, as mentioned above, the relationship between the resolution XA of video A and the resolution XB of video B is "XA>XB", and the relationship between the frame rate FA of video A and the frame rate FB of video B is "FA <FB」である。

[0022] Next, a process for generating a high-definition image according to this embodiment will be described. This process is roughly divided into two parts: a "training data candidate acquisition process" and a "high-definition video generation process."

[0023] FIG. 4 is a diagram illustrating the configuration and operation of functional blocks in image processing by the image processing device 100 of the first embodiment. As described in FIG. 2, video A and video B are stored in the recording unit 106, and tables PA and PB are stored in RAM 103. The "trainer data candidate acquisition process" is executed by the candidate acquisition unit 413. The "high-definition video generation process" is executed by the trainer data extraction unit 414, learning unit 451, and inference unit 452. The candidate acquisition unit 413 extracts pairs of frames that are candidates for training data from the frame group of video A and the frame group of video B as trainer data candidates, and generates a trainer data candidate database (hereinafter, candidate database D1). The frame By to be subjected to high resolution and high definition is acquired from the frame group of image B. The trainer data extraction unit 414 extracts trainer data further suitable for training from the trainer data candidates registered in the candidate database D1 in order to generate a training model suitable for inferring high-frequency components of frame By. The teacher data extraction unit 414 generates a teacher data database (hereinafter referred to as teacher database D2) using the extracted teacher data. The learning unit 451 of the learning and inference unit 105 generates a learning model M for frame By using the teacher database D2. The inference unit 452 inputs the frame By to be high-resolution-enhanced to the learning model M generated by the learning unit 451, and performs high-resolution processing on frame By. The "teacher data candidate acquisition processing" and "high-resolution video generation processing" will be explained in more detail below.

[0024] (Processing to obtain candidate training data) In the teacher data candidate acquisition process, the control unit 101 (candidate acquisition unit 413) generates a candidate database D1. In the first embodiment, the candidate acquisition unit 413 acquires pairs of frames from video A and video B that have the same shooting time as teacher data candidates, from among the frames of video A and video B. Specifically, all frame pairs that have a common shooting time in video A and video B (frame pairs indicated by dashed lines in FIG. 3) are acquired as teacher data candidates. Before the learning process described below is performed, the candidate acquisition unit 413 investigates which frames can be used as teacher data, constructs a candidate database D1, and registers the investigation results.

[0025] FIG. 5 shows an example of the data structure of the candidate database D1. The candidate database D1 stores the frame numbers within each video file of a frame group TA from video A that can be used as training data, and a frame group TB from video B that can be used as training data. Pairs of frames (pairs of frame numbers) that are captured at the same time are associated and registered using a unique index I within the candidate database D1. For example, in the video A and video B shown in FIG. 3, the frame pairs A1 and B2, A2 and B5, and A3 and B8 (omitted below) are combinations of frames captured at the same time. The candidate database D1 shown in FIG. 5 records these pairs by frame number and assigns a unique index I. In this way, the acquired training data candidates are managed by the candidate database D1.

[0026] The details of the teacher data candidate acquisition process described above will be further explained using the flowchart in FIG. 6. In S601, the candidate acquisition unit 413 selects one frame of the video A and acquires time information corresponding to the selected frame from table PA. In this embodiment, the frames are selected sequentially from the beginning of the video A recorded in the recording unit 106. More specifically, the candidate acquisition unit 413 selects one frame sequentially from the beginning of the video A recorded in the recording unit 106. Hereinafter, the selected frame will be referred to as frame Ax. The candidate acquisition unit 413 reads the time information corresponding to frame Ax from the recording unit 106 by referring to table PA recorded in RAM 103, and transfers it to RAM 103.

[0027] In S602, the candidate acquisition unit 413 compares the time information of frame Ax read out in S601 with the time information of each frame of video B. Specifically, the candidate acquisition unit 413 refers to the positional information of the shooting times recorded in table PB, sequentially acquires the shooting time information of each frame of video B from the recording unit 106, and compares it with the time information of frame Ax. In S603, the candidate acquisition unit 413 acquires a frame of video B having a shooting time that matches the time information of frame Ax, and designates this as frame Bx.

[0028] In S604, the candidate acquisition unit 413 assigns a unique index Ix to the combination of the above frame Ax and frame Bx within the candidate database D1 and registers it in the candidate database D1. Specifically, the candidate acquisition unit 413 issues a unique index Ix to the combination of frame Ax and frame Bx, and registers the index Ix, the frame number of frame Ax within video A, and the frame number of frame Bx within video B in the candidate database D1.

[0029] In S605, the control unit 101 determines whether the above-described processes of S601 to S604 have been completed for all frames of the video A. If the control unit 101 determines that the processes have been completed (YES in S605), it ends the process. If the control unit 101 determines that the processes have not been completed (NO in S605), it returns the process to S601 and executes the above-described process for the next frame of the video A. By the above-described processes, the candidate database D1 is generated.

[0030] In this embodiment, in S602, the pair of frames to be registered in the candidate database D1 is determined by comparing the shooting times, but this is not limited to this. For example, the frame Ax may be reduced to a resolution XB, and a similarity determination may be performed using an index representing the similarity between the frame Ax and each frame of the video B, and the determination result may be used to select the pair of frames to be registered in the candidate database D1. In this case, the candidate acquisition unit 413 has a similarity determination function that compares two or more image data and determines the similarity. Note that, for example, the structural similarity index (SSIM) may be used as an index representing the similarity of the images. Furthermore, in acquiring the index representing the similarity, the image of the frame Ax is reduced to a resolution XB, but this is not limited to this. The image of the frame Ax may not be reduced, or the reduced resolution may be a resolution other than XB.

[0031] (High-definition video generation processing) Next, the high-definition video generation process performed by the control unit 101 (teacher data extraction unit 414) and the learning and inference unit 105 (learning unit 451, inference unit 452) will be described. First, an overview of the high-definition video generation process will be described with reference to FIG. 4. The teacher data extraction unit 414 selects teacher data suitable for learning a "learning model for the inference target frame By" from the candidate database D1 and generates a teacher database D2 (FIG. 4) (details will be described later with reference to S702 to S703 in FIG. 7). The learning unit 451 generates a learning model using the extracted teacher data (S704). Then, the inference unit 452 infers high-frequency components of the inference target frame By using the learning model and performs high-definition processing (S705), thereby obtaining a high-definition frame (image) Cy of the inference target frame By. Note that before the high-definition video generation process begins, the control unit 101 generates a video C on the recording unit 106. At the start of high-definition video generation, the video C does not contain any frame data and is empty. The inference unit 452 records the generated frames Cy in the moving image C in sequence.

[0032] The above-mentioned high-definition video generation process will be described in more detail below with reference to the flowchart in Fig. 7. In S701, the teacher data extraction unit 414 reads one frame from video B as a frame to be refined. In this embodiment, the teacher data extraction unit 414 reads frames one by one in order from the beginning of video B recorded in the recording unit 106. Hereinafter, the frame read in S701 will be referred to as frame By. More specifically, the teacher data extraction unit 414 reads the frame data and shooting time information of frame By from the recording unit 106 with reference to table PB, and transfers them to RAM 103.

[0033] In S702, the teacher data extraction unit 414 extracts frames from the teacher data candidates TB registered in the candidate database D1 whose difference in shooting time with respect to frame By is shorter than a threshold value predefined by the system, and registers these frames in the teacher database D2. For example, the threshold value can be the display period of one frame of video A (the display period of one frame at frame rate XA). The structure of the teacher database D2 is similar to that of the candidate database D1 (FIG. 5). Specifically, the teacher data extraction unit 414 first acquires time information for each of the frames TB registered in the candidate database D1 by referring to the position information in table PB. The teacher data extraction unit 414 compares each of the acquired time information with the shooting time of frame By, extracts frames whose difference in time is shorter than the threshold value from the frame group TB, and registers these frames in the teacher database D2 on RAM 103. Hereinafter, the frame group of video B registered in the teacher database D2 by the above process is referred to as UB. In this embodiment, when constructing the teacher database D2, frame groups whose shooting time with respect to frame By is shorter than the threshold value are extracted from the candidate database D1, but this is not limited thereto. The frame group UB may be extracted using an index representing the similarity with the frame By. For example, the teacher data extraction unit 414 may use SSIM to extract from the frame group TB a frame group whose similarity index with the frame By is higher than a threshold value predetermined in advance for the system, and register the frame group UB.

[0034] In S703, the teacher data extraction unit 414 registers in the teacher database D2 the frames of the frame group TA that are paired with each frame of the frame group UB in the candidate database D1. Specifically, the teacher data extraction unit 414 references the candidate database D1 on RAM 103 and registers in the teacher database D2 the frames of the frame group TA that are associated with each frame of the frame group UB by index I. At this time, the combination of the two associated frames is not changed, and a unique index J is assigned to each combination within the teacher database D2. Hereinafter, the frame group of the video A registered in the teacher database D2 will be referred to as UA.

[0035] In S704, the learning unit 451 performs learning using the training data (frame group UA and frame group UB) registered in the training database D2, and generates a training model M.

[0036] FIG. 8 is a diagram illustrating the learning model generation function of the learning unit 451. The learning model generation function includes a learning process and an inference process. The inference process is further divided into a feature extraction process using a filter including a CNN and a reconstruction process. First, in the feature extraction process, the learning unit 451 inputs one image from the frame group UB (referred to as image E) into the CNN, extracts convolution features using the CNN, and generates multiple feature maps. Next, in the reconstruction process, the learning unit 451 upsamples all feature maps using transposed convolution to generate predicted high-frequency components. In the reconstruction process, the learning unit 451 further reconstructs the image by adding the predicted high-frequency components to image E', which is obtained by enlarging image E using a bicubic algorithm or the like, to generate a predicted high-resolution image G. In the learning process, the learning unit 451 compares the predicted high-resolution image G generated in the inference process with image H, which corresponds to image E in the frame group UA, and fine-tunes the learning model M using the backpropagation algorithm using the difference between them. The learning unit 451 improves the inference accuracy by repeating these processes a predetermined number of times for the same image E. By performing the above series of processes for each image of the frame group UB, a learning model M suitable for inference processing of the frame group UB is constructed.

[0037] As described above, the learning unit 451 references the teacher database D2 and tables PA and PB, reads the frame data of the frame pairs registered as teacher data from the recording unit 106, and executes the learning model generation function described above. The learning unit 451 stores the learning model M generated by the learning model generation function in RAM 103.

[0038] In S705, the inference unit 452 generates a high-resolution frame Cy from the frame By by inference using the learning model M generated in S704. Specifically, first, the inference unit 452 reads the learning model M stored in RAM 103. Next, the inference unit 452 inputs the frame data (image) of frame By stored in RAM 103 in S701 to the CNN of the learning model M, and generates "high-frequency components expected when the image of frame By is enlarged to resolution XA." The inference unit 452 generates an image of a high-resolution frame Cy with resolution XA by adding the generated high-frequency components to "an image obtained by linearly enlarging the image of frame By to resolution XA," and records this in RAM 103. Note that the above-mentioned processing from inferring high-frequency components performed for frame By to generating a high-resolution image is the same as the inference process described above with reference to FIG. 8. The inference unit 452 appends the frame data of the high-definition frame Cy recorded in the RAM 103 to the end of the high-definition video C on the recording unit 106. The inference unit 452 also duplicates the shooting time information of By, multiplexes it as the shooting time of the high-definition frame Cy, and records it in the video C.

[0039] In S706, the control unit 101 determines whether the above processing has been completed for the frames in the inference target range of video B (which may be all frames of video B or some of the frames). If the control unit 101 determines that the processing has not been completed (NO in S706), the processing proceeds to S701, the next frame of video B is selected as frame By by the teacher data extraction unit 414, and the above processing is repeated. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S706), the control unit 101 terminates this processing. As described above, when the high-definition video generation processing is completed, the recording unit 106 records high-definition video C with a resolution of XA and a frame rate of FB in an uncompressed format.

[0040] In the above description, each functional block is described as being realized only by the control unit 101 or only by the learning and inference unit 105, but this is not limiting. For example, each functional block may be realized by cooperation between the control unit 101 and the learning and inference unit 105. For example, the function of the inference unit 452 may be realized by the control unit 101 and the learning and inference unit 105, and the process of recording the high-resolution frame Cy and the shooting time in the video C on the recording unit 106 may be executed by the control unit 101.

[0041] In addition, in this embodiment, the training data candidate acquisition process was performed before the learning process for the entire video and the high-definition video generation process were performed, but this may also be performed in parallel with the execution of the high-definition video generation process. In this embodiment, a new learning model M was created for each frame to be inferred in S704 and the previously generated model was discarded, but this is not limited to this. For example, a learning model M' previously trained externally may be loaded, and additional learning may be performed on the loaded learning model M' using the frame group UA and the frame group UB in S704.

[0042] As described above, according to the first embodiment, the learning model M is used, which is trained using a group of images similar to the image to be refined among groups of images captured during the same shooting period, so that the image can be refined with high accuracy. Furthermore, a pair of images from the two image groups captured at the same time is used as training data, so that more accurate learning is possible.

[0043] Second Embodiment In the first embodiment, the candidate training data acquisition process registers in the candidate database D1 a combination of frames from video A and video B with the same capture time. When video A and video B are acquired from videos captured simultaneously using the same image sensor in a single imaging device, frames captured at the same capture time can be acquired from video A and video B, as illustrated in FIG. 3. However, this method may result in inappropriate extraction of candidate training data when video A and video B are captured during the same capture period using multiple image sensors. This is because, as illustrated in FIG. 9, video B does not necessarily contain a frame with a capture time that matches the capture time of a frame in video A. Examples of configurations for capturing video A and video B using multiple image sensors include a configuration in which video A is captured using an imaging device with multiple image sensors and a configuration in which video B is captured using multiple imaging devices with one or more image sensors. In the second embodiment, the candidate training data acquisition process solves the above problem by registering in the candidate database D1 a combination of frames whose capture time difference is shorter than a predetermined threshold, even if the capture times of frames in video A and video B do not match.

[0044] In the second embodiment, the configuration of the image processing device 100 and the high-resolution image generation process are the same as those in the first embodiment, but the process of acquiring the teacher data candidates is partially different. Figure 10 is a flowchart explaining the process of acquiring the teacher data candidates according to the second embodiment. Below, the parts that differ from the process of acquiring the teacher data candidates in the first embodiment (Figure 6) will be mainly explained.

[0045] The processes of S1001 to S1002 are the same as S601 to S602 in the first embodiment (FIG. 6), respectively. In S1003, the candidate acquisition unit 413 acquires, from among the frames of moving image B, a frame whose shooting time difference with respect to the shooting time of one frame Ax of moving image A is shorter than a predetermined threshold, as frame Bx, and registers the frame in the candidate database D1 on RAM 103. Note that an example of such a threshold is the display period per frame at the frame rate XB of moving image B. The subsequent processes of S1004 to S1005 are the same as S604 to S605 in the first embodiment (FIG. 6), respectively.

[0046] As described above, according to the second embodiment, even if the moving images A and B are acquired by a plurality of image sensors, it is possible to appropriately extract training data candidates.

[0047] Third Embodiment In the first and second embodiments, video A and video B were shot during at least the same shooting period. Therefore, in the teacher data candidate acquisition process of the first and second embodiments, when video A and video B were shot at different times (without overlapping shooting periods) by the same or multiple imaging devices, as shown in Figure 11, it is not possible to acquire teacher data candidates. In the third embodiment, a teacher data candidate acquisition process for appropriately acquiring teacher data candidates for video A and video B as shown in Figure 11 will be described.

[0048] In the process of acquiring training data candidates in the third embodiment, an index representing the frame similarity between frames of video A and video B is calculated, and pairs of frames having an index equal to or greater than a threshold value previously set in the system are registered in the candidate database D1. As described above, the index representing frame similarity can be, for example, SSIM. In determining the similarity, the image of the frame of video A is reduced to a resolution XB, and an index representing the similarity is calculated using this and each frame image of video B. However, in this case, the image of the frame of video A does not need to be reduced, and the resolution after reduction may be other than XB.

[0049] Fig. 12 is a flowchart illustrating the process of acquiring candidate teacher data according to the third embodiment. Below, differences from the process of acquiring candidate teacher data according to the first embodiment (Fig. 6) will be mainly described with reference to the flowchart in Fig. 12.

[0050] In S1201, the candidate acquisition unit 413 selects one frame from the moving image A and reads the frame data of the selected frame. The candidate acquisition unit 413 selects one frame in order from the beginning of the moving image A recorded in the recording unit 106 (hereinafter, the selected frame will be referred to as frame Ax). The candidate acquisition unit 413 transfers the frame data of the selected frame Ax from the recording unit 106 to the RAM 103 by referring to table PA recorded in the RAM 103.

[0051] In S1202, the candidate acquisition unit 413 calculates the similarity between frame Ax read out in S1201 and each frame of moving image B. More specifically, the candidate acquisition unit 413 references the position information (related to frame data) in table PB and sequentially acquires frame data for each frame of moving image B from the recording unit 106 to RAM 103. The candidate acquisition unit 413 then calculates a similarity index between frame Ax and each frame using a similarity index calculation function (SSIM in this embodiment), and records the similarity index in RAM 103. In S1203, the candidate acquisition unit 413 acquires the frame of moving image B with the highest numerical value among the similarity indexes calculated in S1202 as frame Bx. The subsequent processes of S1204 to S1205 are the same as S604 to S605 in the first embodiment (FIG. 6).

[0052] As described above, according to the third embodiment, appropriate training data candidates can be acquired even if the shooting periods of the two image groups (moving image A and moving image B) do not overlap.

[0053] <Fourth embodiment> In the fourth embodiment, we will explain how to improve the performance of the learning model M by taking image similarity into account in the learning process of the first to third embodiments. As described in the first embodiment, appropriate training data is extracted for the selected frame By in S701 of FIG. 7, and the training model M is generated / updated using the training data in S704. When generating / updating this learning model M, network parameters are tuned using backpropagation, as shown in FIG. 8. In the fourth embodiment, the strength of the backpropagation tuning is controlled based on the attributes (e.g., capture time) of the frame (image E) used in the learning and the frame By to be enhanced in resolution and definition, or the image of those frames. More specifically, in the learning process, the learning unit 451 determines coefficients so that the stronger the similarity between each frame of the sequentially input frame group UB and frame By is, the stronger the influence of backpropagation on the network parameter update, and the weaker the similarity is, the weaker the influence is. Here, frame similarity may be calculated simply based on the time difference between frame By and input image E, or by comparing the images of both frames using SSIM or the like. For example, when using the former method (method using time difference), the tuning strength can be multiplied by a coefficient of 1 if the time difference is smaller than the threshold, and by a coefficient of 0.5 if the time difference is greater than or equal to the threshold, as shown below. if(ABS(time difference between By and E)<threshold){coefficient=1}else {coefficient=0.5}

[0054] On the other hand, when the latter method (method using similarity) is used, the structural similarity index (SSIM) can be used as a coefficient of tuning strength, as shown below. Coefficient = SSIM(By and E) [0 ≤ SSIM(x) ≤ 1]

[0055] The strength of influence can be determined, for example, by multiplying the update rate of the network parameters in the error backpropagation in the learning process by the above-mentioned coefficient, or by multiplying the number of learning loops for the input image E by a coefficient without multiplying the parameter update rate by a coefficient.

[0056] Fifth Embodiment In the first to third embodiments, a configuration was described in which a pair of a frame from video A and a frame from video B is extracted as teacher data candidates and registered in the candidate database D1. In the fifth embodiment, a video A' is generated by converting video A to the resolution XB of video B, and the candidate acquisition unit 413 acquires teacher data candidates using video A and video A'. That is, the candidate acquisition unit 413 of the fifth embodiment extracts frame Ax', which has the same frame number as frame Ax in video A, from video A' and registers the pair of frame Ax and frame AX' in the candidate database D1 as teacher data candidates. The fifth embodiment will be described in detail below.

[0057] (Explanation of the configuration of the image processing device 100) The hardware configuration and functional configuration of the image processing device 100 are the same as those of the first embodiment (FIG. 1). However, the control unit 101 of the fifth embodiment also has a resolution conversion function for reducing the resolution of an image using bicubic digitization. When performing resolution reduction processing on image data recorded in the RAM 103, this resolution conversion function calculates the pixel values ​​of pixels that require interpolation by referring to their surrounding pixels.

[0058] (Data recorded in the recording unit 106 and its decryption and decompression methods) In the first embodiment, the moving images a and b stored in the recording unit 106 are converted into an uncompressed format, and the decoded moving image a is recorded in the recording unit 106 as moving image A and the decoded moving image b is recorded in the recording unit 106 as moving image B. In the fifth embodiment, moving image A' is generated by converting moving image A to a resolution XB of moving image B. More specifically, the control unit 101 refers to table PA recorded in RAM 103 and sequentially inputs frame data of frames of moving image A (hereinafter, frame K) stored in the recording unit 106 to a resolution conversion function of the control unit 101. The resolution conversion function then outputs a frame of frame data with resolution XB (hereinafter, frame K'). The control unit 101 refers to table PA and multiplexes the frame data with shooting time information of frame K read from the recording unit 106, and records the multiplexed frame in the recording unit 106 as a frame of moving image A'. In addition, table PA' is recorded in RAM 103, which holds the frame number of each frame of moving image A', location information indicating the storage location of the frame data, and location information indicating the storage location of the shooting time data.

[0059] An example of these videos A, B, and A' is shown in FIG. 13. For each frame image (A1 to An) of video A, images (A1' to An') reduced to a resolution XB are generated and recorded in the recording unit 106 as video A'. In the above example, the resolution of video A is reduced to XB, but this is not limited to this. Video A' may be any image converted to have a resolution lower than that of video A. However, by using an image converted to the same resolution as the image to be refined, a learning model that is more suitable for the image to be refined can be constructed.

[0060] (Processing to obtain candidate training data) FIG. 14 shows the configuration and operation of functional blocks in image processing by the image processing device 100 of the fifth embodiment. The candidate acquisition unit 413 acquires combinations of frames having the same frame number for each frame of moving image A and moving image A', and registers them in the candidate database D1. More specifically, the candidate acquisition unit 413 searches for frames in moving image A' that have the same frame number as each frame of moving image A listed in table PA by referring to table PA'. The candidate acquisition unit 413 assigns a unique index I to combinations of frames from moving image A and moving image A' that have the same frame number, and registers them in the candidate database D1. The frame group of moving image A registered in the candidate database D1 is designated as TA, and the frame group of moving image A' is designated as TA'.

[0061] (High-definition video generation processing) The following mainly describes the differences from the processing of the first embodiment (FIG. 7), with reference to the flowchart of FIG.

[0062] The processing of S1501 is the same as S701 in the first embodiment (FIG. 7). In S1502, the teacher data extraction unit 414 extracts frames from the teacher data candidate frame group TA' registered in the candidate database D1, the frames having a difference in shooting time from frame By that is shorter than a threshold value predetermined by the system. The threshold value can be, for example, the display period of one frame of video A (the display period of one frame at frame rate XA). The teacher data extraction unit 414 registers the extracted frames in the teacher database D2.

[0063] Specifically, the teacher data extraction unit 414 first refers to table PA' to acquire the time information of the frames registered in the frame group TA'. The teacher data extraction unit 414 registers, among the time information of the acquired frame group TA', frames whose time difference with frame By is shorter than a threshold in the teacher database D2 on RAM 103. Hereinafter, the frame group of the moving image A' registered in the teacher database D2 is referred to as frame group UA'. Note that in this embodiment, frames whose difference with the shooting time of frame By is shorter than a predetermined threshold are extracted from the candidate database D1, but this is not limited to this. For example, frames whose index (e.g., SSIM) representing the similarity between the image of each frame of the frame group TA' and the image of frame By is higher than a threshold predetermined for the system may be extracted from the frame group TA' and registered in the teacher database D2.

[0064] In S1503, the teacher data extraction unit 414 registers in the teacher database D2 the frames of the frame group TA that are associated with each frame of the frame group UA' by index I. Specifically, the teacher data extraction unit 414 refers to the candidate database D1 on the RAM 103, and registers in the teacher database D2 the frames of the frame group TA that are associated with each frame of the frame group UA' by index I. At this time, the associated combinations (frame pairs) do not change, and each combination is assigned a unique index J in the teacher database D2. Hereinafter, the frame group of the video A registered in the teacher database D2 will be referred to as the frame group UA.

[0065] In S1504, the learning unit 451 performs learning using the frame group UA and the frame group UA' by referring to the teacher database D2, and generates a learning model M. Specifically, first, the learning unit 451 refers to the teacher database D2 and tables PA and PA', reads frame data from the recording unit 106, and inputs the frame data to the learning model generation function. The learning unit 451 performs learning using the frame data read by the learning model generation function, and stores the learning model M generated as the learning result in RAM 103. Details of learning the learning model are as described above with reference to FIG. 8. The subsequent processing of S1505 and S1506 is the same as that of the first embodiment (the processing of S705 and S706 in FIG. 7).

[0066] As described above, according to each of the above embodiments, training data used for training a training model is selected based on an image to be refined. Therefore, a training model trained using the selected training data can infer high-frequency components of the image to be refined with higher accuracy, thereby obtaining a highly refined image. In other words, the accuracy of video super-resolution technology for enhancing the resolution of videos is improved.

[0067] In the above embodiments, when acquiring candidate teacher data, the image paired with the image selected from video A is an image selected from video B based on the shooting time or image similarity, or an image with a lower resolution than the selected image. However, this embodiment is not limited to this. An image related to an image selected from video A used as candidate teacher data may have a lower resolution than the selected image and may be an image related to the selected image. Whether an image is related to an image selected from video A may be determined based on commonalities such as the temperature at the time of shooting, the shooting location, and the shooting direction, for example.

[0068] Furthermore, in the above embodiments, a two-stage process of generating the candidate database D1 and then the teacher database D2 has been described, but this is not limiting. For example, the teacher data extraction unit 414 may extract frames from video A based on the frame By that can be paired with the teacher data, and obtain teacher data by using the extracted frames and frames related to the extracted frames as pairs. However, when sequentially increasing the resolution of multiple images in video B, it is more efficient to create the candidate database D1 as in the above embodiments and then extract and use appropriate teacher data from the candidate database D1 depending on the image to be increased in resolution.

[0069] Furthermore, in the above embodiments, moving image a and moving image b having a lower resolution than moving image a are processed, but this is not limited to this. For example, uncompressed moving image a and moving image b obtained by compressing and then restoring moving image a may be processed. In this case, moving image a may be saved with frames thinned out. As such, the relationship between moving images a and b to be processed in the above embodiments is not limited to a difference in resolution, as long as moving image a is clearer than moving image b. In other words, the group of images constituting moving image a (movie A) only needs to contain more high-frequency components than the group of images constituting moving image b (movie B). For example, the processing of the above embodiments can be applied if each image in the group of images in moving image a corresponds to one or more images in the group of images in moving image b, and each image in the group of images in moving image a has more high-frequency components than the corresponding image in the group of images in moving image b.

[0070] For simplicity, the above description focuses on video data. However, for example, if the device is capable of generating still images at predetermined timings while recording a video, the above embodiments can also be applied to the following cases. That is, a still image is used as data corresponding to video a, and a video is used as data corresponding to video b. For example, assume that one of the above embodiments is applied to an imaging device that captures 6K raw data at 60 fps using an image sensor. Assume, for example, that the still image data is recorded in a format such as JPEG or HEIF after undergoing processing and still-image compression at 6K size. Assume, for example, that the video data is recorded in a format such as MP4 after undergoing processing and video compression after converting 6K data acquired by an image sensor to 2K data size (60 fps 2K video data). Under these assumptions, if a user presses the release switch to continuously capture still images while the imaging device is recording 2K video data at 60 fps, 6K still images are generated at intervals of, for example, 10 fps relative to the video frame rate (60 fps). By applying any of the above-described embodiments to the still images and videos generated in this manner, data with still image quality corresponding to the video captured during a period in which multiple still images were captured can be generated. That is, a system can be realized that obtains a video with a still image size of 6K and a frame rate of 60 fps, as if the video had been captured. In this case, the imaging device prepares still images and videos, and performs learning and inference processing within the imaging device to generate still image quality data corresponding to the video.

[0071] Sixth Embodiment In the sixth embodiment, an improvement in the learning performance and inference performance in the learning process and inference process of the first embodiment will be described, taking into account image similarity.

[0072] In the first embodiment, appropriate training data is extracted for the selected frame By in S701 of FIG. 7, and a learning model M is generated or updated using the training data in S704. Then, in S705, high-frequency components are inferred using the learning model M, and a high-resolution frame Cy is generated. However, with this method, if the frame By contains various textures, such as people, buildings, plants, and the sea, the amount of information to be learned in one go becomes large, and the desired learning performance may not be achieved. This is because a single frame contains high-frequency components with various patterns. Therefore, in the learning process of the sixth embodiment, the above problem is solved by extracting a portion of a frame, generating a learning model for each local region, performing inference using the learning model for each local region, and generating and combining high-resolution images for each local region.

[0073] In the sixth embodiment, the hardware configuration and functional configuration of the image processing device 100 are the same as those in the first embodiment (FIG. 1). The extracted teacher data may be from any of the first to fifth embodiments. Since the learning process and subsequent steps are different, a specific description will be given using the flowchart in FIG. 16 and an example of the learning and inference process in FIG. 17.

[0074] The processing from S1601 to S1603 is the same as S701 to S703 in the first embodiment (FIG. 7).

[0075] In S1604, the inference unit 452 cuts out the inference target frame By into a local region (determines a local region) and stores the local region in the RAM 103. Hereinafter, the cut-out local region (local image) is referred to as a local region Byn1701.

[0076] Next, in S1605, the learning unit 451 selects local areas UAn1702 and UBn1703 that correspond to the same coordinate position as the local area Byn of the inference target frame By from the training data (frame group UA, UB) registered in the training database D2 (local area selection). The learning unit 451 stores the selected local area UAn1702 and local area UBn1703 in RAM 103. In this embodiment, the training data includes a pair of local areas, but the training data may include multiple pairs of local areas. Note that this group of local areas is, for example, a rectangular area of ​​equal size, measuring several tens of pixels on each side, but is not limited to this.

[0077] Note that the "local area corresponding to the same coordinate position" of the local area Byn1701 to be inferred, as described here, is an area indicated by the exact same coordinates as the local area of ​​the inference target frame By, in the case of the frame group UB. In other words, if the local area coordinates of the inference target frame By are (sx, sy), then the local area coordinates of the local area UBn1703 are also (sx, sy). On the other hand, in the frame group UA, the ratio of the resolution XA of video A to the resolution XB of video B is taken into consideration. For example, if XA:XB has a 2:1 relationship in both width and height, then if the local area coordinates of the inference target frame By are (sx, sy), then the local area coordinates of the local area UAn1702 are (sx*2, sy*2). Hereinafter, the expression "local area corresponding to the same coordinate position" refers to the above.

[0078] In S1606, the learning unit 451 uses the local area UAn1702 and the local area UBn1703 to generate a learning model Mn1704 (local area learning model) using the learning model generation function shown in Fig. 8. The learning unit 451 reads out the frame data of the frame pair registered as teacher data from the recording unit 106, inputs it to the learning model generation function for each local area, and saves the generated learning model Mn1704 in the RAM 103.

[0079] In S1607, the inference unit 452 performs inference on the local region Byn1701 using the learning model Mn1704 generated in S1606, and generates a local region Cyn1705 (local high-frequency component) of a high-resolution frame. First, the inference unit 452 reads out the learning model Mn1704 saved in RAM 103 in S1606. Next, the inference unit 452 inputs the local region Byn1701 saved in RAM 103 in S1604 to the CNN of the learning model Mn1704, and generates "high-frequency components expected when the local region Byn1701 is expanded to the local region UAn1702." The inference unit 452 generates the local region Cyn1705 by adding the generated high-frequency components to "an image obtained by linearly expanding the image of the local region Byn1701 to the local region UAn1702," and records this in RAM 103. The process from the inference of high-frequency components performed for the local region Byn1701 to the generation of a high-definition image is the same as the inference process shown in FIG.

[0080] Next, in S1608, the inference unit 452 combines the local region Cyn1705 of the high-resolution frame Cy1706 recorded in the RAM 103 based on the coordinate position information of the frame to generate the high-resolution frame Cy1706 and stores it in the RAM 103. Note that 1705 indicated by the dashed line in Fig. 17 is the local region Cyn, and 1706 indicated by the solid line is the high-resolution frame Cy.

[0081] In S1609, the control unit 101 determines whether the above processing has been completed for all local regions in frame By. If the control unit 101 determines that the processing has not been completed (NO in S1609), the process proceeds to S1605, and the above processing is repeated for the next local region in frame By. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S1609), the process proceeds to S1610.

[0082] In S1610, the inference unit 452 appends the frame data of the high-resolution frame Cy1706 recorded in the RAM 103 to the end of the high-resolution image C on the recording unit 106. In addition, the shooting time information of By is duplicated, multiplexed as the shooting time of the high-resolution frame Cy1706, and recorded in the moving image C.

[0083] In S1611, the control unit 101 determines whether the above processing has been completed for all frames of moving image B. If the control unit 101 determines that the processing has not been completed (NO in S1611), the process proceeds to S1601, and the above processing is repeated with the next frame of moving image B as By. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S1611), the control unit 101 ends this processing. When the high-definition moving image generation processing is completed as described above, a high-definition moving image C with a resolution of XA and a frame rate of FB is recorded in the recording unit 106 in an uncompressed format.

[0084] As described above, according to the sixth embodiment, for an image to be refined that contains a large amount of information and includes various textures, the amount of information to be learned at one time can be reduced by learning for each local region, thereby enabling more accurate learning, and thus enabling the generation of a higher-resolution image.

[0085] Seventh Embodiment In the seventh embodiment, an example will be described in which the learning process for each local region in the sixth embodiment is changed to improve the super-resolution performance.

[0086] In the method of the sixth embodiment, a learning model is generated by using an area in a frame different from the inference target that is at the same position as the inference target area for learning. However, this method has the problem that when the subject moves rapidly, the objects captured in the inference area and the training data differ, which may make it impossible to achieve the desired super-resolution performance.

[0087] Therefore, in the learning process of the seventh embodiment, a similarity evaluation function is provided to search for areas that have a high similarity to the inference area from among the candidate teacher data, and areas that have a high similarity are used for learning, thereby solving the above problem.

[0088] (High-definition video generation processing) The seventh embodiment differs from the sixth embodiment only in the processing of S1605 in the flowchart of the high-definition moving image generation processing shown in Fig. 16. Therefore, only the processing of S1605 in the seventh embodiment will be described.

[0089] In S1605, the inference unit 452 extracts a portion of the inference target frame By and stores it as a local region in RAM 103. Note that this local region is, for example, a rectangular region of uniform size, measuring several tens of pixels square, but is not limited to this. The control unit 101 uses SSIM, which is provided as a means for realizing a similarity evaluation function, to search for a region UBn having the highest similarity to the local region of the inference target frame By from the frame group UB of the teacher data registered in the teacher database D2, and stores the region UBn in RAM 103. The learning unit 451 selects a frame from the frame group UA that is paired with the frame to which the local region UBn stored in RAM 103 belongs, and from among the frame group UA, stores a local region UAn that is located in the same relative position as the local region UBn in RAM 103. Note that the similarity evaluation may also be performed using PSNR (Peak Signal to Noise Ratio), SNR (Signal to Noise Ratio), or MSE (Mean Square Error). In the above, the region UBn with the highest similarity is searched for from all frames included in the frame group UB, but this is not limited to this. For example, the region UBn with the highest similarity may be searched for for each frame included in the frame group UB. In this case, the same number of pairs of local regions UBn and local regions UAn as the number of frames included in the frame group UB are obtained.

[0090] As described above, according to the seventh embodiment, learning is performed using an area that has a high similarity to the inference area, so that it is possible to generate a higher-resolution image even for a video in which the subject moves vigorously.

[0091] Eighth Embodiment In the eighth embodiment, a method for solving the problem in the sixth embodiment shown in the seventh embodiment, which is different from that in the seventh embodiment, will be described.

[0092] In the eighth embodiment, a method will be described in which a motion vector related to an inference region is used to identify a region with high similarity. However, it is assumed that the video b in the eighth embodiment is compressed in the MPEG-4 AVC format using inter-frame prediction. Note that MPEG-4 AVC is an abbreviation for ISO / IEC 14496-10 "MPEG-4 Part 10: Advanced Video Coding."

[0093] The following describes the eighth embodiment, focusing mainly on the differences from the sixth embodiment.

[0094] (Data recorded on the recording medium and its decryption and expansion methods) In the processing of the analysis unit 211 in the eighth embodiment, in addition to the processing of parsing video data recorded in the recording unit 106 (described in the first embodiment), the following processing is performed. The analysis unit 211 parses the MP4 file storing video b to obtain an avcC box. The analysis unit 211 then obtains a sequence parameter set (hereinafter referred to as SPS) and a picture parameter set (hereinafter referred to as PPS) included in the avcC box, and stores them in RAM 103.

[0095] (High-definition video generation processing) The difference between the high-definition moving image generation process in the eighth embodiment and the sixth embodiment is the processes in S1605 to S1607 in the flowchart of Fig. 16. Therefore, the processes in S1605 to S1607 in the eighth embodiment will be described using the flowchart of Fig. 18.

[0096] In the sixth embodiment, in step S1604, the inference unit 452 cuts out the local region Byn of the inference target frame By into a rectangular region of equal size, 16 pixels on each side.

[0097] In S1801, if the inference target frame By is an I picture, the control unit 101 proceeds to S1803. If the inference target frame By is a P picture or a B picture, the control unit 101 proceeds to S1802. Whether the inference target frame is an I picture, a P picture, or a B picture can be determined by referring to the SPS and PPS, for example.

[0098] In S1802, the control unit 101 acquires a macroblock layer from the local region Byn of the inference target frame By. Furthermore, if a sub-macroblock is used, a sub-macroblock prediction is acquired. Otherwise, a macroblock prediction is acquired.

[0099] The control unit 101 derives a unit block area Bynb of prediction in a macroblock from the sub-macroblock prediction or macroblock prediction of the macroblock to which the local area Byn of the inference target frame By belongs. The unit block area Bynb of prediction is either a macroblock, each block of a partitioned macroblock, each block of a sub-macroblock, or each block of a partitioned sub-macroblock. These blocks are units of motion compensation.

[0100] The control unit 101 derives the motion vector, reference frame, mbPartIdx, and subMbPardIdx of the block area Bynb from the SPS, PPS, macroblock prediction, or sub-macroblock prediction.

[0101] Here, the control unit 101 generates six pieces of information for each block area Bynb: "mbPartIdx," "subMbPartIdx," "presence or absence of motion vector," "motion vector," "reference / referenced frame," and "reference direction," and stores them in the RAM 103. "mbPartIdx" and "subMbPartIdx" are information that identifies which block area within a macroblock the block area Bynb belongs to. "Motion vector" indicates the temporal and spatial movement of the block area Bynb, and more specifically, indicates the referenced block in the referenced frame. "Presence or absence of motion vector" indicates whether the block area Bynb has such a motion vector. "Reference / referenced frame" indicates the referenced frame referenced in decoding the inference target frame By from which the block area Bynb is extracted, and the reference frame that references the block area Bynb. When the "reference / referenced frame" is generated in S1802, the referenced frame is recorded. The "reference direction" refers to the direction in which the motion vector points from the macroblock of the local region Byn of the inference target frame By, and the direction in which the macroblock in another frame points to the local region Byn of the inference target frame By is the reference direction. Hereinafter, the collection of the above six pieces of information will be referred to as motion vector information.

[0102] The control unit 101 checks whether the frame specified by the "reference / referenced frame" in the generated motion vector information exists in the candidate training data. If the frame specified by the "reference / referenced frame" exists in the candidate training data, the control unit 101 sets the "presence or absence of motion vector" in the motion vector information to "present," and if not, sets the "presence or absence of motion vector" to "absence."

[0103] Also, for example, if the inference target frame By is a B picture and the block has two motion vectors, the referenced frame that is closer in time to the inference target frame By is used. If the difference in temporal distance from the inference target frame By is the same for both, the motion vector and referenced frame information that is closer in spatial distance indicated by the motion vector is used. If the temporal distance and spatial distance are the same, either referenced frame may be used.

[0104] In S1803, for a block area Bynb for which the "presence or absence of motion vector" in the motion vector information is "absent," the control unit 101 searches for a block that references the block area Bynb from among the training data candidates. Hereinafter, a block that references the block area Bynb is also referred to as a reference source block. Note that the method of obtaining the motion vector and reference frame information required to determine whether a block is a reference source block for the block area Bynb has been described in S1802, and will not be repeated here.

[0105] If a block that references block area Bynb (a block that references block area Bynb) is found, the "presence of motion vector" in the motion vector information of block area Bynb is updated to "presence." Furthermore, the frame containing the block that references block area Bynb is stored as the reference frame in "referenced / referenced frame." The range of frames to be searched is limited to three frames before and after the frame containing block area Bynb. Furthermore, the range of macroblocks to be searched is limited to MaxVmvR for each level specified by MPEG-4 AVC. MaxVmvR is derived from the SPS of video b. The range of frames and macroblocks to be searched are not limited to this example.

[0106] In S1804, the inference unit 452 acquires, from the frame group UB, a reference-target or reference-source block area UBXnb for each block area Bynb for which the "presence or absence of motion vector" in the motion vector information is "present," and stores each in RAM 103. The inference unit 452 also acquires, from the frame group UA, a block area UAXnb corresponding to the same coordinate position as the block area UBXnb acquired from the motion vector information of each block area Bynb recorded in RAM 103, and stores each in RAM 103. That is, the inference unit 452 acquires a block area UAXnb corresponding to the same coordinate position as the block area UBXnb from a frame of the frame group UA that is paired with the frame to which the block area UBXnb belongs. The inference unit 452 then stores the block area UAXnb in RAM 103 in association with the block area UBXnb.

[0107] In S1805, the control unit 101 determines whether the "presence or absence of motion vector" of the motion vector information in all block areas Bynb included in the local area Byn of the inference target frame By is "present" or "absent." If the control unit 101 determines that the "presence or absence of motion vector" of all block areas Bynb is "present" (YES in S1805), the control unit 101 proceeds to S1806.

[0108] In S1806, the inference unit 452 generates a local area UBXn by combining the block areas UBXnb recorded in the RAM 103 based on the coordinate position information of each block area Bynb. The inference unit 452 stores the generated local area UBXn in the RAM 103.

[0109] The inference unit 452 also generates a local area UAXn by combining block areas UAXnb corresponding to the same coordinate positions as the block area UBXnb recorded in the RAM 103 based on the coordinate position information of each block area Bynb. The inference unit 452 stores the generated local area UAXn in the RAM 103.

[0110] The learning unit 451 then generates a learning model Mn using the local area UAXn and local area UBXn recorded in RAM 103 and the learning model generation function shown in Fig. 8. Note that the local area UBXn is training data corresponding to the same coordinate position as the local area UAXn in the paired frame. The learning unit 451 reads out this training data from RAM 103, executes the learning model generation function, and stores the generated learning model Mn in RAM 103.

[0111] In S1807, the inference unit 452 performs inference on the local region Byn of the frame By using the learning model Mn generated in S1806, and generates a local region Cyn of the high-resolution frame.

[0112] First, the inference unit 452 reads out the learning model Mn stored in RAM 103 in S1806. Next, the inference unit 452 inputs the local region Byn of the frame By stored in RAM 103 into the CNN of the learning model Mn, and generates "high-frequency components expected in the local region Byn when the inference target frame By is enlarged to resolution XA." The inference unit 452 generates a local region Cyn by adding the generated high-frequency components to "the local region Byn linearly enlarged based on the ratio between resolution XB and resolution XA," and records this in RAM 103. Note that the processing from inferring high-frequency components performed on the local region Byn to generating a high-resolution image is the same as the inference process shown in FIG. 8.

[0113] On the other hand, in S1805, if the control unit 101 determines that the local region Byn includes a block region Bynb for which the "presence or absence of motion vector" is "absent" (NO in S1805), the control unit 101 proceeds to S1808. In S1808, the control unit 101 determines whether the "presence or absence of motion vector" in the motion vector information is "present" or "absent" for each block region Bynb included in the local region Byn. If the control unit 101 determines that the "presence or absence of motion vector" is "present" (YES in S1808), the control unit 101 proceeds to S1809. On the other hand, in S1808, if the control unit 101 determines that the "presence or absence of motion vector" is "absent" (NO in S1808), the control unit 101 proceeds to S1811.

[0114] In S1809, the learning unit 451 generates a learning model Mnb using the block region Bynb and the local region UBXnb, using the learning model generation function shown in FIG.

[0115] More specifically, in S1809, the learning unit 451 generates a learning model Mnb for inferring the block area Bynb using the local area UBXnb and local area UAXnb recorded in RAM 103 and the learning model generation function shown in FIG. 8. The local area UBXnb is training data corresponding to the same coordinate position as the local area UAXnb in the paired frame. The learning unit 451 reads this training data from RAM 103 and inputs it to the learning model generation function, and stores the generated learning model Mnb in RAM 103.

[0116] In S1810, the inference unit 452 performs inference on the block region Bynb using the learning model Mnb to generate a block region Cynb of a high-resolution frame. First, the inference unit 452 reads the learning model Mnb stored in RAM 103 in S1809. Next, the inference unit 452 inputs the block region Bynb stored in RAM 103 into the CNN of the learning model Mnb to generate "high-frequency components expected in the local region Bynb when the inference target frame By is enlarged to resolution XA." The inference unit 452 generates a block region Cynb of a high-resolution frame by adding the generated high-frequency components to "a local region Bynb linearly enlarged based on the ratio between resolution XB and resolution XA," and records this in RAM 103. Note that the processing from inferring high-frequency components performed on the block region Bynb to generating a high-resolution image is the same as the inference process shown in FIG. 8.

[0117] In S1811, the control unit 101 linearly enlarges a block area Bynb in which the presence or absence of a motion vector in the motion vector information is "absent" based on the ratio between the resolutions XA and XB to obtain a block area Cynb of the high-definition frame Cy, and stores the result in the RAM 103. Note that the method of enlargement is not limited to linear enlargement, as long as it can be performed based on the ratio between the resolutions XA and XB.

[0118] In S1812, the control unit 101 determines whether the above processing has been completed for all block areas Bynb. If the control unit 101 determines that the processing has not been completed (NO in S1812), the control unit 101 proceeds to S1807, where it performs processing on the incomplete block areas Bynb. If the control unit 101 determines that the processing has been completed (YES in S1812), the control unit 101 proceeds to S1813. In S1813, the control unit 101 reads out the block areas Cynb stored in the RAM 103 in S1810 and S1811, and combines them based on the coordinate position information of the corresponding block areas Bynb to generate a local area Cyn of the high-resolution frame. The generated local area Cyn is stored in the RAM 103. In S1608 of FIG. 16, the local area Cyn generated as described above is used as the local area Cyn 1705.

[0119] As described above, according to the eighth embodiment, learning is performed using motion vectors that refer to / are referenced by areas with high similarity to the inference area, so it is possible to generate higher-resolution images even for videos in which the subject moves rapidly.

[0120] Ninth Embodiment In the ninth embodiment, a method for solving the problem in the sixth embodiment shown in the seventh embodiment, which is different from the seventh and eighth embodiments, will be described.

[0121] The ninth embodiment will be described below, focusing mainly on the differences from the sixth embodiment.

[0122] (High-definition video generation processing) The ninth embodiment differs from the sixth embodiment only in the processes of S1605 and S1606 in the flowchart of the high-definition moving image generation process shown in Fig. 16. Therefore, the processes of S1605 and S1606 in the ninth embodiment will be described below.

[0123] In S1605, the control unit 101 selects local regions (assumed to be UAn5 and UBn5) corresponding to the same coordinate position as the local region Byn in the inference target frame By from the pair of frames of the frame group UA and UB, and stores them in RAM 103. In addition, the control unit 101 stores eight regions of the same size as UBn5 that are adjacent to UBn5 in RAM 103. Similarly, eight regions of the same size as UAn5 that are adjacent to UAn5 are saved in RAM 103. An example of region selection for frames included in the frame group UB is shown in FIG. 19. Note that in this embodiment, the region at the same position coordinate as the local region Byn and its eight adjacent regions are selected as the inference target region, but the means and number of region selections are not limited to this.

[0124] Next, the control unit 101 evaluates the similarity between the local region Byn of the inference target frame By and each of UBn1 to UBn9, and obtains a similarity evaluation value. The control unit 101 then determines the number of times of learning for each of UBn1 to UBn9 based on the similarity evaluation value, and stores the result in the RAM 103 as learning information. The learning information includes, for example, information for identifying one of UBn1 to UBn9, a similarity evaluation value with the local region Byn, and the number of times of learning. If the similarity evaluation value with the local region Byn in the learning information is less than a threshold value predefined by the system, the control unit 101 updates the number of times of learning for the learning information to 0. For regions whose similarity evaluation value is equal to or greater than the threshold, the control unit 101 determines the number of times of learning based on the ratio of the similarity evaluation values ​​between regions whose similarity evaluation value is equal to or greater than the threshold, and updates the learning information. For example, assume that the similarity evaluation values ​​of UBn4, UBn5, and UBn6 are equal to or greater than the threshold, the ratio is 2:5:3, and the total number of times of learning is set to 1,000. In this case, the number of times of learning in the learning information of UBn4 to UBn6 is 200, 500, and 300, respectively. Note that in the method of determining the number of times of learning in this embodiment, the number of times of learning is linearly allocated to areas having similarity evaluation values ​​exceeding the threshold, but the method is not limited to this.

[0125] In S1606, the learning unit 451 uses a pair of an image of the local region (any of UBn1 to 9) indicated by the learning information and an image of the corresponding local region (any of UAn1 to 9) in the frame group UA as training data to generate a learning model Mn. The learning unit 451 performs training on each piece of training data using the learning model generation function shown in FIG. 8 the number of times of training indicated by the learning information, and generates a learning model Mn. The generated learning model Mn is saved in RAM 103.

[0126] The processing from S1607 onwards is the same as in the sixth embodiment, and therefore a description thereof will be omitted.

[0127] As described above, according to the ninth embodiment, a plurality of regions having a high similarity to the inference region are used for learning according to their similarities to the inference region, which makes it possible to generate a higher-resolution image even for a video in which the subject moves vigorously.

[0128] As described above, according to the sixth to ninth embodiments, a local region can be determined from an image to be refined, and the amount of information used for training the learning model can be reduced. Furthermore, according to the sixth to ninth embodiments, a local region of the training data that is highly correlated with the local region determined from the image to be refined can be selected and used for training the learning model. This makes it possible to more accurately infer the high-frequency components of the image to be refined, and to obtain a highly accurate high-resolution image. In other words, the accuracy of video super-resolution technology for enhancing the resolution of videos can be improved.

[0129] Tenth Embodiment In the tenth embodiment, an example will be described in which the learning process for each local region in the sixth embodiment is modified to reduce the learning process load. In the method of the sixth embodiment, one frame is divided into multiple local regions, and a learning model is generated for each local region, followed by inference processing, thereby improving super-resolution performance. However, this method has the problem of requiring the generation of a learning model for each local region, resulting in a high learning process load. Therefore, in the learning process of the tenth embodiment, a similarity evaluation function is provided to detect movement for each local region, and local regions determined to have no movement are combined to form a new combined local region, thereby reducing the number of local regions. This reduces the number of generated learning models and the learning process load.

[0130] The difference between the tenth embodiment and the sixth embodiment is the process of S1604 (processing of cutting out a local region from frame By) in the flowchart of the high-definition video generation process shown in Fig. 16. Therefore, the following mainly describes the process of S1604 in the tenth embodiment.

[0131] The local region extraction processing of frame By in S1604 of the tenth embodiment will be described with reference to Fig. 20 and Fig. 21. Fig. 20 is a flowchart showing the local region extraction processing according to the tenth embodiment. Fig. 21 is a diagram illustrating the concept of the local region extraction processing according to the tenth embodiment.

[0132] In FIG. 21, 2100 is the inference target frame By. 2110 is an image diagram in which semantic region division is performed on the inference target frame By, and the semantic regions are displayed as frames. Frames 2101 and 2102 are the "tree" region, frames 2103 and 2104 are the "ground" region, and frame 2105 is the "person" region. At this time, each of the multiple frames obtained is treated as a different semantic region even if they have the same meaning. For example, frame 2101 and frame 2102 are regions with the same meaning ("tree"), but are treated as different semantic regions.

[0133] 2120 is an image diagram in which the inference target frame By is divided into equally sized rectangular partial regions Byn' as in the sixth embodiment, and whether or not there is motion in each partial region is determined. In this embodiment, it is assumed that there is a lot of movement in the image of a "person" and little movement in the other images. In 2120, the shaded partial regions By1' to By9', By13' to By16', By20' to By23', By27' to By30', and By34' to By35' are partial regions determined to have little movement.

[0134] 2130 is an illustration of a local region Byn extracted in this embodiment. The local region Byn is basically the same as the partial region Byn'. However, in this embodiment, using the results of the division into semantic regions (2110) and the results of determining the amount of motion of each local region (2120), partial regions Byn' determined to have "no motion" within the same semantic region are combined to form a single local region (combined local region). The local regions indicated by diagonal lines in 2130 correspond to the combined local regions. That is, partial regions By1', By2', By8', and By9' are combined to form a single local region By1, and partial regions By6', By7', By13', and By14' are combined to form a single local region By5. Furthermore, partial regions By22' and By23' are combined to form a single local region By16, and partial regions By27' and By28' are combined to form a single local region By20. Furthermore, a partial region Byn' that does not meet the above conditions is cut out as a single local region.

[0135] Next, the processing of the tenth embodiment will be described with reference to the flowchart of FIG. 20. In S2001, the learning and inference unit 105 performs processing to divide the image of the inference target frame By into semantic regions, and stores the processing result in RAM 103. Here, the semantic region segmentation can be realized by inference using a CNN model such as Mask-R CNN. Therefore, the learning and inference unit 105 performs semantic region segmentation by switching the CNN model used from a CNN model for super-resolution to a CNN model for semantic region segmentation (e.g., Mask-R CNN). Alternatively, a learning and inference unit dedicated to semantic region segmentation may be added separately from the learning and inference unit 105.

[0136] In S2002, the control unit 101 extracts a partial region Byn' from the inference target frame By and stores it in the RAM 103. In this embodiment, the partial region Byn' is a rectangular region (square region) of uniform size, for example, several tens of pixels on each side, but is not limited to this. For example, the partial region Byn' may also be a rectangular region.

[0137] In S2003, the control unit 101 determines whether or not there is motion in the image of each partial region Byn′ extracted in S2002 from the previous inference target frame. The control unit 101 stores information indicating the partial regions determined to have no motion from the previous inference target frame in the RAM 103. Here, determining whether or not there is motion in the image of a partial region can be achieved, for example, by a similarity evaluation function using SSIM. The control unit 101 acquires the similarity between the partial regions at the same coordinates in the inference target frame By and the previous inference target frame using SSIM, and if the acquired similarity is greater than a specific threshold, the control unit 101 determines that the partial region has no motion. If the acquired similarity is equal to or less than a specific threshold, the control unit 101 determines that the partial region has motion. Note that, although SSIM is used for the similarity evaluation, this is not limiting. For example, PSNR (Peak Signal to Noise Ratio), SNR (Signal to Noise Ratio), MSE (Mean Square Error), etc. may also be used.

[0138] In S2004, the control unit 101 selects partial regions that are within the same semantic region calculated in S2001 and that are determined to have no motion in S2003, and stores the selected partial regions in RAM 103. In this embodiment, if a partial region is entirely contained within a single semantic region, the partial region is considered to exist within the single semantic region. However, this is not limited to this. For example, if a predetermined percentage or more of a partial region is contained within a single semantic region, the partial region may be treated as existing within the single semantic region. In S2005, the control unit 101 combines the partial regions selected in S2004 and stores the combined local region in RAM 103. In this embodiment, partial regions that are contained within the same semantic region are treated as a single local region even if they are not contiguous, but this is not limited to this. For example, partial regions that are determined to have no motion and that are contiguous vertically and horizontally within the same semantic region in frame By may be combined into a single local region.

[0139] In S2006, the inference unit 452 extracts the combined local region Byn stored in RAM 103 in S2005 as a single local region and stores it in RAM 103. Furthermore, the inference unit 452 extracts each of the partial regions Byn' stored in RAM 103 in S2002 that were not selected as regions to be combined in S2005 as a single local region Byn and stores it in RAM 103. In the example of FIG. 21, the partial regions that divide the image into 42 are extracted as 34 local regions by the combining process of S2005. These 34 local regions are used in the processes from S1605 onwards in FIG. 16.

[0140] As described above, according to the tenth embodiment, multiple "motionless" partial regions are combined into a single local region, thereby reducing the number of subsequent learning model generation processes and reducing the learning processing load while maintaining super-resolution performance.

[0141] In this embodiment, local regions are combined within the same semantic region acquired in S2001, but this is not necessarily the case. For example, the control unit 101 may combine all “still” local regions in frame By into a single local region, regardless of their semantic regions. Furthermore, for example, the control unit 101 may combine adjacent “still” partial regions in the front, back, left, and right directions, regardless of their semantic regions. In this case, for example, the set of “still” partial regions shown in 2120 of FIG. 21 is cut out as a single local region. Furthermore, for example, the control unit 101 may combine “still” partial regions so that the combined local region has a rectangular shape. For example, if a “still” partial region such as the one shown in 2120 of FIG. 21 is obtained, three combined local regions (e.g., 5 × 2 local regions on the left and right, and a 1 × 3 local region in the middle) are cut out.

[0142] The tenth embodiment has been described above based on the sixth embodiment, but it is clear that the local regions combined by the tenth embodiment can also be used in the processes described in the seventh to ninth embodiments. Furthermore, it goes without saying that the training data extracted for learning may be that of any of the first to fifth embodiments.

[0143] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0144] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention. [Explanation of symbols]

[0145] 101: control unit, 102: ROM, 103: RAM, 104: decoding unit, 105: learning and inference unit, 106: recording unit, 107: bus, 211: analysis unit, 212: decoded video generation unit, 413: candidate acquisition unit, 414: teacher data extraction unit, 451: learning unit, 452: inference unit

Claims

1. 1. An image processing device that uses a first image group to increase the resolution of images of a second image group having fewer high frequency components than images of the first image group, a selection means for selecting training data to be used for learning from a plurality of training data including an image included in the first image group as one of a pair of images, based on an image to be refined selected from the second image group; a calculation means for calculating a similarity between each of a plurality of partial regions obtained by dividing the image to be refined and a corresponding partial region of the image that was the previous target of refinement; a determination means for determining a plurality of local regions from the image to be refined, wherein a set of one or more partial regions having a similarity equal to or greater than a threshold is combined into one local region, and each partial region having a similarity less than the threshold is treated as a separate local region; a model generation means for generating a learning model for inferring high frequency components for each of the plurality of local regions using the teacher data selected by the selection means; inference means for inferring high-frequency components of each of the plurality of local regions using the learning model; an image generating unit configured to generate a high-definition image based on the image to be refined and the high-frequency components inferred by the inference unit;

2. The image processing device according to claim 1, characterized in that the determining means combines partial regions, among the partial regions whose similarity is equal to or greater than the threshold, that are continuous in either the up, down, left, or right direction in the image to be refined into a single local region.

3. further comprising a dividing means for dividing the image to be refined into semantic regions; 3. The image processing apparatus according to claim 1, wherein the determining means combines partial regions that belong to the same semantic region obtained by the dividing means and whose similarity is equal to or greater than the threshold value.

4. The method further includes acquiring means for acquiring a pair of a first image selected from the first image group and a third image having fewer high frequency components than the first image and related to the first image as candidates for the training data, 4. The image processing apparatus according to claim 1, wherein the selection means selects the training data to be used in the learning from candidates of the training data.

5. 5. The image processing apparatus according to claim 4, wherein the acquisition means acquires the candidate training data by acquiring the third image from the second image group.

6. 6. The image processing apparatus according to claim 5, wherein the acquisition means acquires, as the third image, an image having the same photographing time as the first image from the second image group.

7. The image processing device according to claim 5, wherein the acquisition means acquires, as the third image, an image from the second image group having a shooting time whose difference from the shooting time of the first image is smaller than a predetermined threshold value.

8. 6. The image processing apparatus according to claim 5, wherein the acquisition means acquires, as the third image, an image that has the greatest similarity to the first image from the second image group.

9. 9. The image processing apparatus according to claim 8, wherein the acquisition means determines a similarity between an image obtained by reducing the first image to a resolution of the second image group and an image of the second image group.

10. 5. The image processing apparatus according to claim 4, wherein the acquisition means acquires, as the third image, an image obtained by reducing the resolution of the first image by reducing the size of the first image.

11. 11. The image processing apparatus according to claim 10, wherein the third image is an image obtained by reducing the resolution of the first image to that of the second image group.

12. The image processing device according to any one of claims 4 to 11, characterized in that the selection means selects candidate teacher data including images whose shooting time difference from the shooting time of the image to be enhanced in resolution is smaller than a predetermined threshold as teacher data to be used for the learning.

13. The image processing device according to any one of claims 4 to 11, characterized in that the selection means selects candidate training data including an image whose similarity to the image to be refined is greater than a predetermined threshold as training data to be used for the learning.

14. 14. The image processing device according to claim 1, wherein the inference means controls updating of parameters by error backpropagation in the learning, based on teacher data used in the learning and the image to be refined.

15. 15. The image processing device according to claim 14, wherein the inference means determines coefficients based on training data used in the learning and the image to be refined, and controls the amount of parameter update by the error backpropagation based on the coefficients.

16. 15. The image processing device according to claim 14, wherein the inference means determines coefficients based on training data used in the learning and the image to be refined, and controls the number of iterations of updating the parameters by the error backpropagation based on the coefficients.

17. 17. The image processing apparatus according to claim 15, wherein the inference means determines the coefficient based on a difference between the photographing time of an image of the training data used in the learning and the photographing time of the image to be subjected to high definition enhancement.

18. 17. The image processing apparatus according to claim 15, wherein the inference means determines the coefficients based on a similarity between an image of training data used in the learning and the image to be refined.

19. the model generation means extracts pairs of images corresponding to the plurality of local regions from the training data selected by the selection means, and generates a learning model for each local image of the plurality of local regions using the extracted pairs of images; the inference means infers a local high frequency component of the local image using a learning model for the local image; 19. The image processing device according to claim 1, wherein the image generating means generates a high-definition image of a local region using the local high-frequency component and the local image, and combines the high-definition images generated for each local region.

20. 20. The image processing apparatus according to claim 19, wherein the model generating means extracts a pair of images of an area corresponding to the same coordinate position as the local area from the training data selected by the selecting means.

21. 21. The image processing apparatus according to claim 20, wherein the image generating means generates a high-definition image of the image to be subjected to high-definition processing by combining high-definition images for each local region based on coordinate position information.

22. 20. The image processing apparatus according to claim 19, wherein the model generating means extracts a pair of images having the highest similarity to the local image from the training data selected by the selecting means.

23. The image processing device described in claim 19, characterized in that the model generation means extracts a pair of images corresponding to the local region from the training data selected by the selection means based on a motion vector set for a block, which is a unit of motion compensation, included in the local region, or a motion vector that references the block.

24. The model generation means extracting a plurality of pairs of images corresponding to a plurality of regions identified based on the positions of the local regions from the training data selected by the selection means; 20. The image processing device according to claim 19, wherein the number of times of learning performed using each of the plurality of pairs of images in generating the learning model is determined based on the similarity between the local image and each of the plurality of pairs of images.

25. 25. The image processing apparatus according to claim 24, wherein the plurality of regions include a first region corresponding to the position of the local region and a second region adjacent to the first region.

26. 26. The image processing apparatus according to claim 24, wherein the model generating means does not perform learning using a pair of images whose similarity with the local image is equal to or less than a threshold value.

27. 27. The image processing device according to claim 1, wherein the first image group and the second image group are two image groups obtained by performing different image processing on a single image captured by a single image sensor included in a single imaging device.

28. 27. The image processing apparatus according to claim 1, wherein the first image group and the second image group are image groups captured by two image sensors, respectively.

29. 29. The image processing apparatus according to claim 1, wherein a frame rate of the first group of images is lower than a frame rate of the second group of images.

30. 1. An image processing method for increasing the resolution of images of a second image group, which has fewer high frequency components than images of the first image group, by using a first image group, comprising: a selection step of selecting training data to be used for learning from a plurality of training data including an image included in the first image group as one of a pair of images, based on an image to be refined selected from the second image group; a calculation step of calculating a similarity between each of a plurality of partial regions obtained by dividing the image to be refined and a corresponding partial region of the image that was the previous target of refinement; a determination step of determining a plurality of local regions from the image to be refined, combining a set of one or more partial regions whose similarity is equal to or greater than a threshold into one local region, and each partial region whose similarity is less than the threshold into a separate local region; a model generation step of generating a learning model for inferring high-frequency components for each of the plurality of local regions using the training data selected in the selection step; an inference step of inferring high-frequency components of each of the plurality of local regions using the learning model; an image generation step of generating a high-resolution image based on the image to be refined and the high-frequency components inferred in the inference step; An image processing method comprising:

31. 30. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 29.

Images