Image processing apparatus and method, program

Abstract

To provide an image processing apparatus and method, and a program for improving the definition of an image with high accuracy.SOLUTION: An image processing apparatus uses a first image group to improve the definition of images of a second image group with less high frequency component than that of the first image group, selects, from pieces of training data including an image included in the first image group as one of a pair of images, training data used for learning based on an image to be improved in definition selected from the second image group, learns a learning model by using the selected training data and infers a high frequency component of the image to be improved in definition by using the learning model, and creates an image with improved definition based on the image to be improved in definition and the inferred high frequency component.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and method, and a program for enhancing an image group using machine learning.

Background Art

[0002] Super-resolution technology using machine learning is a technology for generating a high-definition image by inferring high-frequency components that cannot be compensated by linear interpolation of pixel values when enlarging an image and performing resolution conversion. In super-resolution technology, first, a learning model is generated using an image group G and a degraded image obtained by degrading each image in the image group G by an arbitrary method as teacher data. The learning model is generated by learning the difference between the pixel values of the original image and the degraded image and updating the super-resolution processing parameters it holds. When an image H lacking high-frequency components is input to the thus-generated learning model, high-frequency components are obtained by inference using the learning model. By superimposing the high-frequency components obtained by inference on the image H, a high-definition image can be generated. When performing super-resolution processing on a video, a high-definition video can be generated by inputting all the frames one by one into the learning model.

[0003] Generally, when products or services utilizing learning models are provided, the process of collecting training data and generating the learning model is performed by the developer, and the generated learning model is then provided to the user. Therefore, the content of the video input by the user is unknown at the time of the learning process. To achieve consistent accuracy in inference for all target videos, the developer prepares a large amount of diverse images with no bias in image patterns as training data and repeatedly trains the model. For example, Patent Document 1 describes a technique for performing super-resolution processing on a video using a learning model trained on diverse images. However, because the training data is diverse, only a very small amount of training data has a high similarity to the target video Q specified by the user. When such a learning model is used, the results of learning images with low similarity to the target video Q are reflected in the inference process. As a result, the improvement in resolution is limited to the enhancement of edges of the subject, and it is difficult to accurately infer high-frequency components such as fine patterns of the subject, so the inference accuracy cannot be said to be high. As an example of a system to solve such problems, Patent Document 2 describes a method in which the user learns by using only images that are similar to the video to be inferred and similar in terms of shooting location and shooting conditions as training data, thereby obtaining a higher-resolution video compared to when a variety of images are used for training. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-204167 [Patent Document 2] Japanese Patent Publication No. 2019-129328 [Overview of the project] [Problems that the invention aims to solve]

[0005] Patent Document 2 describes training using training data where the shooting location is the same but the shooting time is different. More specifically, video footage previously shot in a section S of a bus route is accumulated and used for training, and inference is performed on real-time video footage of section S using the resulting trained model. In this case, the training data is limited to images shot in section S, and since it is a group of images with a relatively high degree of similarity to the inference target, an improvement in inference accuracy can be expected. However, even among the video footage shot in section S, for example, the video at the starting point of section S and the video at the ending point of section S are shot in different locations, so the subjects shown are completely different and cannot be said to be highly similar. This is a factor that reduces the inference accuracy for the entire section S. In addition, even if the video is from the same location, the subjects shown in the past video footage used as training data and the real-time video footage used for inference are different. Accurate inference cannot be performed for subjects that have not been trained, so this is also a factor that reduces the inference accuracy.

[0006] Furthermore, Patent Document 2 describes classifying past video footage into multiple groups based on shooting conditions such as weather, independently training each group's data to generate multiple learning models, and switching the learning model used depending on the shooting conditions of real-time video. Such a technique can suppress the decrease in inference accuracy due to different shooting conditions. However, even if conditions such as weather are the same, if values ​​such as illuminance differ even slightly, the frequency components of the training data and the target of inference will differ, so it cannot be said that the decrease in inference accuracy can be sufficiently suppressed. Due to these factors, even with the technique in Patent Document 2, the inference accuracy for high-frequency components cannot be said to be sufficient.

[0007] According to the present invention, a technology is provided for increasing the resolution of images with high precision. [Means for solving the problem]

[0008] An image processing apparatus according to one aspect of the present invention has the following configuration. That is, An image processing apparatus that uses a first group of images to enhance the resolution of a second group of images having fewer high-frequency components than the first group of images, The image included in the first group of images is considered one of a pair of images. The other image in the pair of images has fewer high-frequency components than the images included in the first image group, and is associated with the images included in the first image group. A selection means for selecting the training data to be used for learning from the training data included therein, based on the image to be high-resolution selected from the second group of images, An inference means that learns a learning model using the training data selected by the selection means, and infers the high-frequency components of the image to be high-resolution using the learning model, The system comprises a generation means for generating a high-resolution image based on the image to be high-resolution and the high-frequency components inferred by the inference means. [Effects of the Invention]

[0009] According to the present invention, images can be made high-resolution with high precision. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the configuration of the image processing apparatus according to the first embodiment. [Figure 2] A diagram illustrating the functional configuration of an image processing apparatus according to the first embodiment. [Figure 3] A diagram illustrating the functional configuration of an image processing apparatus according to the first embodiment. [Figure 4] A diagram showing an example of the frame structure of an input video according to the first embodiment. [Figure 5] Flowchart of the process for acquiring candidate training data according to the first embodiment. [Figure 6] A diagram showing an example of the data structure of a candidate database according to the first embodiment. [Figure 7] Flowchart of the high-definition video generation process according to the first embodiment. [Figure 8] A schematic diagram illustrating the learning / inference process according to the first embodiment. [Figure 9] A diagram showing an example of the frame structure of the input video in the second embodiment. [Figure 10] Flowchart of the process for acquiring candidate training data according to the second embodiment. [Figure 11]A diagram showing an example of the frame configuration of an input video according to the third embodiment. [Figure 12] A flowchart of teacher data candidate acquisition processing according to the third embodiment. [Figure 13] A diagram showing an example of the frame configuration of a video according to the fifth embodiment. [Figure 14] A diagram for explaining the functional configuration of an image processing apparatus according to the fifth embodiment. [Figure 15] A flowchart of high-definition video generation processing in the fifth embodiment.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate explanations are omitted.

[0012] <First Embodiment> (Outline Explanation of Image Processing Apparatus) The image processing apparatus of the first embodiment takes as input two videos A and B that are simultaneously captured by the same imaging device. The relationship between the resolution XA / frame rate FA of video A and the resolution XB / frame rate FB of video B is "XA > XB and FA < FB". The image processing apparatus has a function (high-definition video generation function) of generating a learning model using the frames of videos A and B and generating video C with a resolution of XA / frame rate FB from video B by inference using the generated learning model.

[0013] (Explanation of the Configuration of the Image Processing Apparatus) Figure 1 is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment. The control unit 101 is an arithmetic unit such as a CPU, and it realizes various functions by loading and executing programs stored in the ROM 102 into the work area of ​​the RAM 103. The control unit 101 can function as, for example, the analysis unit 111 and the decoded video generation unit 112, which will be described later in Figure 2, the candidate acquisition unit 113, and the training data extraction unit 114, which will be described later in Figure 3. The ROM 102 stores the control program executed by the control unit 101. The RAM 103 is used as a work memory for the control unit 101 to execute programs, and as a temporary storage area for various data.

[0014] The decoding unit 104 decodes video or image data compressed according to the encoding format defined by the Moving Picture Experts Group (hereinafter abbreviated as MPEG) into uncompressed data. The learning and inference unit 105 has a functional block (learning unit 151, described later in Figure 3) that generates / updates a learning model by inputting training data. The learning and inference unit 105 also has a functional block (inference unit 152, described later in Figure 3) that generates a high-resolution image of the input image by analyzing the input image using the learning model generated by learning and inferring high-frequency components. In this embodiment, a super-resolution processing CNN model based on a Convolutional Neural Network (CNN) is used as the learning model. This includes scaling the input image by linear interpolation, generating high-frequency components to be added to the scaled image, and additive synthesis of the two.

[0015] The recording unit 106 consists of a recording medium, such as a hard disk drive (HDD) or memory card, which is detachably connected to the image processing device 100, and a recording medium control device that controls the recording medium. The recording medium control device controls the initialization of the recording medium and the data transfer between the recording medium and RAM 103 for reading and writing data, according to commands from the control unit 101. The bus 107 is an information communication path that connects each function. The control unit 101, ROM 102, RAM 103, decoding unit 104, learning and inference unit 105, and recording unit 106 are connected so that they can communicate with each other.

[0016] It should be noted that the hardware blocks and functional blocks executed within them described in this embodiment do not necessarily have to be configured as described above. For example, two or more of the control unit 101, decoding unit 104, and learning inference unit 105 may be implemented by a single piece of hardware. Also, the functionality of one functional block or the functionality of multiple functional blocks may be executed by the coordinated operation of several pieces of hardware. Furthermore, each functional block may be implemented by a CPU executing a computer program loaded into memory, or it may be implemented by dedicated hardware. In addition, a part of each functional block may reside on a cloud server, and the processing result data may be transferred via communication. For example, the decoding unit 104 may be implemented by the same CPU as the control unit 101, or by a different CPU. Alternatively, the decoding unit 104 may be implemented by a GPU that operates under instructions from the control unit 101. Alternatively, the decoding unit 104 may be implemented by hardware processing using electronic circuits configured for complex processing. Also, for example, the learning inference unit 105 may be implemented by the same CPU as the control unit 101, or by a different CPU. Alternatively, the learning inference unit 105 may be implemented by a GPU that operates in response to instructions from the control unit 101. Alternatively, the learning inference unit 105 may be implemented by hardware processing using electronic circuits configured for learning inference.

[0017] (Data recorded on a recording medium and its decoding and extraction methods) Figure 2 is a diagram illustrating the functional blocks that perform the decompression processing of compressed video data by the control unit 101 (analysis unit 111 and decoded video generation unit 112). The recording unit 106 stores video a and video b, which are input data for the high-definition video generation process. Here, video refers to one or more temporally continuous image data. In this embodiment, video a and video b are captured simultaneously by an imaging device having an image sensor and compressed using the MPEG method. Video a and video b may be generated by applying decimation and reduction processing to images captured by a single image sensor, or they may be generated by capturing the same subject with image sensors of different resolutions and frame rates. Hereinafter, video a and video b are assumed to be two groups of images obtained by performing different image processing on a single image captured by a single image sensor of a single imaging device. The video data of video a and video b are each compressed using the MPEG method, multiplexed together with the capture time information, and stored in MP4 format. Note that other formats are also acceptable as long as the image data and corresponding capture time information can be obtained as a pair from the recording unit 106.

[0018] The analysis unit 111 has the function of parsing the video data (in this example, an MP4 file) recorded in the recording unit 106 and calculating the storage location within the file of the compressed image data and time information registered as metadata. In the MP4 format, the Moov section contains location information indicating the recording location within the file for each frame data and shooting time information. The analysis unit 111 unpacks the Moov section of video a from the recording unit 106 into the RAM 103 and parses it to generate a table Pa containing the frame number, frame data location information, and shooting time location information for video a. Similarly, the analysis unit 111 parses the Moov section of video b and generates a table Pb containing the frame number, frame data location information, and shooting time location information for video b. Tables Pa and Pb are stored in the RAM 103.

[0019] To use them in high-definition video generation processing, it is necessary to convert video a and video b into an uncompressed format. As shown in Figure 2, the decoded video generation unit 112 of the control unit 101 decodes video a and video b to generate video A and video B and records them in the recording unit 106. More specifically, the decoded video generation unit 112 refers to tables Pa and Pb held in RAM 103 and sequentially inputs the frame data of video a and video b stored in the recording unit 106 to the decoder 104. The decoded video generation unit 112 multiplexes the uncompressed frame data output by the decoder 104 with the shooting time information obtained by referring to tables Pa and Pb and records it in the recording unit 106. Hereinafter, it is assumed that video A is obtained by decoded video a, and video B is obtained by decoded video b. The decoded video generation unit 112 also generates a table PA containing the frame number in video A, position information indicating the storage location of the frame data, and position information indicating the storage location of the shooting time, and records it in RAM 103. Similarly, the decoded video generation unit 112 generates a table PB containing the frame number in video B, location information indicating the storage location of the frame data, and location information indicating the storage location of the shooting time, and records it in RAM 103. An example of the frame configuration of video A and video B is shown in Figure 4. In Figure 4, the total number of frames in video A is n, and the total number of frames in video B is m. Pairs of frames shown by dashed lines (A1 and B2, A2 and B5, A3 and B8, etc.) are pairs of frames that have the same shooting time information, indicating that the images of these frames were taken at the same time.

[0020] Next, the process for generating high-resolution images according to this embodiment will be described. This process is broadly divided into two parts: "training data candidate acquisition process" and "high-resolution video generation process". Figure 3 is a diagram illustrating the operation of functional blocks in image processing by the image processing apparatus 100 of the first embodiment. As explained in Figure 2, video A and video B are held in the recording unit 106, and tables PA and PB are held in the RAM 103. The "training data candidate acquisition process" is executed by the candidate acquisition unit 113. The "high-resolution video generation process" is executed by the training data extraction unit 114, the learning unit 151, and the inference unit 152. The candidate acquisition unit 113 extracts pairs of frames that will be candidates for training data from the frame group of video A and the frame group of video B as training data candidates, and generates a candidate database D1. Frame By, which is the target for high resolution and high-resolution, is obtained from the frame group of image B. The training data extraction unit 114 extracts further training data suitable for learning from the candidate training data registered in the candidate database D1, in order to generate a training model suitable for inferring the high-frequency components of frame By, and generates a training data database D2. The learning unit 151 of the learning inference unit 105 generates a training model M for frame By using the training data database D2. The inference unit 152 inputs frame By, which is the target of high-resolution processing, into the training model M generated by the learning unit 151 and performs high-resolution processing on frame By. The "training data candidate acquisition process" and the "high-resolution video generation process" will be explained in more detail below.

[0021] (Process for acquiring candidate training data) In the training data candidate acquisition process, the control unit 101 (candidate acquisition unit 113) generates a candidate database D1. In the first embodiment, the candidate acquisition unit 113 acquires pairs of video A and video B frames with matching shooting times as training data candidates. Specifically, all frame pairs with common shooting times in video A and video B (frame pairs shown by dashed lines in Figure 4) are acquired as training data candidates. Before the learning process described later is performed, the candidate acquisition unit 113 investigates which frames can be used as training data, constructs the candidate database D1, and registers the investigation results. Figure 6 shows an example of the data structure of the candidate database D1. The candidate database D1 registers the frame numbers within each video file for the frame group TA of video A that can be used as training data, and the frame group TB of video B that can be used as training data. Here, pairs of frames with matching shooting times (pairs of frame numbers) are associated and registered using a unique index I in the candidate database D1. For example, in video A and video B shown in Figure 4, the pairs of frames A1 and B2, A2 and B5, A3 and B8 (and so on) are combinations of frames taken at the same time. The candidate database D1 shown in Figure 6 shows how these pairs are recorded by frame number and assigned a unique index I. In this way, the acquired training data candidates are managed by the candidate database D1.

[0022] The details of the training data candidate acquisition process described above will be further explained using the flowchart in Figure 5. In S501, the candidate acquisition unit 113 selects one frame from video A and acquires the time information corresponding to the selected frame from table PA. In this embodiment, frames are selected sequentially from the beginning of video A recorded in the recording unit 106. More specifically, the candidate acquisition unit 113 selects one frame at a time from the beginning of video A recorded in the recording unit 106. Hereinafter, the selected frame will be referred to as frame Ax. The candidate acquisition unit 113 reads the time information corresponding to frame Ax from the recording unit 106 by referring to table PA recorded in RAM 103 and transfers it to RAM 103.

[0023] In S502, the candidate acquisition unit 113 compares the time information of frame Ax read in S501 with the time information of each frame of video B. Specifically, the candidate acquisition unit 113 refers to the shooting time position information recorded in table PB, sequentially acquires the shooting time information of each frame of video B from the recording unit 106, and compares it with the time information of frame Ax. In S503, the candidate acquisition unit 113 acquires a frame of video B that has a shooting time that matches the time information of frame Ax, and sets this as frame Bx.

[0024] In S504, the candidate acquisition unit 113 assigns a unique index Ix to the combination of frame Ax and frame Bx within the candidate database D1 and registers it in the candidate database D1. Specifically, the candidate acquisition unit 113 issues a unique index Ix to the combination of frame Ax and frame Bx, and registers the index Ix, the frame number of frame Ax in video A, and the frame number of frame Bx in video B in the candidate database D1.

[0025] In S505, the candidate acquisition unit 113 determines whether the processes described in S501 to S504 have been completed for all frames of video A. If it is determined that the processes have been completed (YES in S505), the process is terminated. If it is determined that the processes have not been completed (NO in S505), the process returns to S501, and the processes described above are executed for the next frame of video A. Through these processes, the candidate database D1 is generated.

[0026] In this embodiment, in S502, pairs of frames to be registered in the candidate database D1 were determined by comparing their capture times, but this is not limited to this. For example, frame Ax could be reduced to resolution XB, a similarity determination could be performed using an index representing the similarity between it and the images of each frame in video B, and pairs of frames to be registered in the candidate database D1 could be selected using the determination results. In this case, the candidate acquisition unit 113 would have a similarity determination function that determines the similarity by comparing two or more image data. As an index representing the similarity of images, for example, the Structural Similarity Index (SSIM) could be used. Also, in acquiring the index indicating similarity, the image of frame Ax was reduced to resolution XB, but this is not limited to this. The image of frame Ax does not need to be reduced, and the resolution after reduction may be other than XB.

[0027] (High-definition video generation processing) Next, the high-definition video generation process performed by the control unit 101 (training data extraction unit 114) and the learning inference unit 105 (learning unit 151, inference unit 152) will be explained. First, an overview of the high-definition video generation process will be explained with reference to Figure 3. The training data extraction unit 114 selects training data suitable for training the "learning model for the frame By to be inferred" from the candidate database D1 and generates the training data database D2 (Figure 3) (details will be described later by referring to S702 to S703 in Figure 7). The learning unit 151 generates a learning model using the extracted training data (S704). Then, the inference unit 152 uses the learning model to infer the high-frequency components of the frame By to be inferred and performs high-definition processing (S705) to obtain the high-definition frame (image) Cy of the frame By to be inferred. Before the start of the high-definition video generation process, the control unit 101 generates video C on the recording unit 106. At the start of high-definition video generation, video C does not have frame data and is empty. The inference unit 152 sequentially records the generated frames Cy into the video C.

[0028] The process for generating high-definition video described above will be explained in more detail below with reference to the flowchart in Figure 7. In S701, the training data extraction unit 114 reads one frame from video B as the frame to be high-definition. In this embodiment, the training data extraction unit 114 reads one frame at a time from the beginning of video B recorded in the recording unit 106. Hereinafter, the frame read in S701 will be referred to as frame By. More specifically, the training data extraction unit 114 reads the frame data and shooting time information of frame By from the recording unit 106 by referring to table PB and transfers it to RAM 103.

[0029] In S702, the training data extraction unit 114 extracts frames from the candidate training data TB registered in the candidate database D1 in which the difference in shooting time with frame By is shorter than a threshold predetermined by the system, and registers these frames in the training data database D2. As the threshold, for example, the display period of one frame of video A (the display period of one frame at frame rate XA) can be used. The structure of the training data database D2 is the same as that of the candidate database D1 (Figure 6). Specifically, first, the training data extraction unit 114 obtains the time information for each frame group TB registered in the candidate database D1 by referring to the position information of table PB. The training data extraction unit 114 compares each of the obtained time information with the shooting time of frame By, extracts frames from the frame group TB in which the difference between the two is shorter than the threshold, and registers them in the training data database D2 on RAM 103. Hereafter, the frame group of video B registered in the training data database D2 by the above process will be referred to as UB. In this embodiment, when constructing the training data database D2, a group of frames whose capture time with respect to frame By is shorter than a threshold is extracted from the candidate database D1, but this is not the only method. The frame group UB may be extracted using an index that represents the similarity with respect to frame By. For example, the training data extraction unit 114 may use SSIM to extract a group of frames from the frame group TB whose similarity index with respect to frame By is higher than a threshold predetermined by the system, and register it as the frame group UB.

[0030] In S703, the training data extraction unit 114 registers the frames of frame group TA that are paired with each frame of frame group UB in the candidate database D1 into the training data database D2. Specifically, the training data extraction unit 114 refers to the candidate database D1 on RAM 103 and registers the frames of frame group TA that are associated with each frame of frame group UB by index I into the training data database D2. At this time, the combination of two associated frames is not changed, and a unique index J is assigned to each combination within the training data database D2. Hereinafter, the frame group of video A registered in the training data database D2 will be referred to as UA.

[0031] In S704, the learning unit 151 performs training using the training data (frame group UA and frame group UB) registered in the training data database D2 to generate a learning model M. Figure 8 is a schematic diagram showing the learning model generation function by the learning unit 151. The learning model generation function has a learning process and an inference process, and the inference process can be further classified into a feature extraction process using a filter including a CNN and a reconstruction process. First, in the feature extraction process, the learning unit 151 inputs one image from frame group UB into the CNN (let's call it image E), and the CNN extracts convolutional features to generate a number of feature maps. Next, in the reconstruction process, the learning unit 151 upsamples all feature maps by transpose convolution to generate predicted high-frequency components. In the reconstruction process, the learning unit 151 further reconstructs the image by adding image E', which is obtained by expanding image E using a bicubic method or the like, with the predicted high-frequency components to generate a predicted high-resolution image G. In the learning process, the learning unit 151 compares the predicted high-resolution image G generated in the inference process described above with the corresponding image H from the frame group UA, and performs minute tuning of the learning model M using backpropagation with the difference between them. The learning unit 151 improves the inference accuracy by repeating these processes a predetermined number of times for the same image E. By performing the above series of processes for each image in the frame group UB, a learning model M suitable for the inference processing of the frame group UB is constructed.

[0032] As described above, the learning unit 151 reads frame data of frame pairs registered as training data from the recording unit 106 by referring to the training data database D2 and tables PA and PB, and executes the learning model generation function described above. The learning unit 151 saves the learning model M generated by the learning model generation function to RAM 103.

[0033] In S705, the inference unit 152 generates a high-resolution frame Cy from frame By by performing inference using the learning model M generated in S704. Specifically, first, the inference unit 152 reads the learning model M stored in RAM 103. Next, in S701, the inference unit 152 inputs the frame data (image) of frame By held in RAM 103 into the CNN of the learning model M and generates "high-frequency components expected when the image of frame By is enlarged to resolution XA". The inference unit 152 adds the generated high-frequency components to "the image obtained by linearly enlarging the image of frame By to resolution XA" to generate a high-resolution frame Cy image with resolution XA, and records this in RAM 103. The above process, from inference of high-frequency components to generation of a high-resolution image performed on frame By, is the same as the inference process (feature extraction process and reconstruction process) described above using Figure 8. The inference unit 152 appends the frame data of the high-resolution frame Cy recorded in the RAM 103 to the end of the high-resolution video C on the recording unit 106. It also duplicates the shooting time information of By, multiplexes it as the shooting time of the high-resolution frame Cy, and records it in video C.

[0034] In S706, the control unit 101 determines whether the above process has been completed for the frames within the inference target range of video B (which may be all frames of video B or some frames). If it is determined that the process is not completed (NO in S706), the process proceeds to S701, and the next frame of video B is selected as frame By by the training data extraction unit 114, and the above process is repeated. On the other hand, if it is determined that the process is completed (YES in S706), this process ends. As described above, when the high-definition video generation process is completed, the recording unit 106 records a high-definition video C with resolution XA and frame rate FB in uncompressed format.

[0035] Although the above explanation describes each functional block as being implemented by the control unit 101 alone or by the learning inference unit 105 alone, this is not the only way to do so. For example, each functional block may be implemented through the cooperation of the control unit 101 and the learning inference unit 105. For instance, the functions of the inference unit 152 may be implemented by the control unit 101 and the learning inference unit 105, while the process of recording the high-definition frame Cy and the shooting time on the video C on the recording unit 106 is performed by the control unit 101.

[0036] Furthermore, in this embodiment, the process of acquiring candidate training data was performed before the learning process for the entire video and the high-definition video generation process, but it may also be performed in parallel with the execution of the high-definition video generation process. Also, in this embodiment, a new learning model M was created for each frame to be inferred in S704 and the previously generated model was discarded, but this is not the only way. For example, a learning model M' that has been trained externally in advance may be loaded, and additional learning using the frame group UA and frame group UB may be performed on the loaded learning model M' in S704.

[0037] As described above, according to the first embodiment, since the learning model M is trained using a group of images similar to the image to be refinished from among the image group taken during the same shooting period, the image can be refinished with high accuracy. Furthermore, since a pair of images from the same time from two image groups is used as training data, even higher accuracy learning is possible.

[0038] <Second Embodiment> In the training data candidate acquisition process of the first embodiment, combinations of frames from video A and video B with matching shooting times are registered in the candidate database D1. When video A and video B are acquired from videos captured simultaneously using the same image sensor of a single imaging device, frames with the same shooting time can be obtained from video A and video B, as illustrated in Figure 4. However, with this method, if video A and video B are videos captured during the same shooting period by multiple image sensors, the extraction of training data candidates may not be performed appropriately. This is because, as illustrated in Figure 9, there are not necessarily frames in video B with shooting times that match the shooting times of frames in video A. Configurations for capturing video A and video B with multiple image sensors include configurations that use an imaging device with multiple image sensors, and configurations that use multiple imaging devices with one or more image sensors. In the training data candidate acquisition process of the second embodiment, even if the shooting times of frames from video A and video B do not match, the above problem is solved by registering combinations of frames in the candidate database D1 where the time difference is shorter than a predetermined threshold.

[0039] In the second embodiment, the configuration of the image processing device 100 and the high-resolution image generation process are the same as in the first embodiment, but some parts of the training data candidate acquisition process differ. Figure 10 is a flowchart illustrating the training data candidate acquisition process according to the second embodiment. The following mainly describes the parts that differ from the training data candidate acquisition process of the first embodiment (Figure 5).

[0040] The processes in S1001 to S1002 are the same as those in S501 to S502 of the first embodiment (Figure 5). In S1003, the candidate acquisition unit 113 acquires frames from among the frames of video B in which the difference between the shooting time of the frame and the shooting time of one frame Ax of video A is shorter than a predetermined threshold, and registers them as frames Bx in the candidate database D1 on RAM 103. For example, such a threshold could be the display period per frame based on the frame rate XB of video B. The processes in S1004 to S1005 are the same as those in S504 to S505 of the first embodiment (Figure 5).

[0041] As described above, according to the second embodiment, even if video A and video B are acquired by multiple image sensors, it is possible to appropriately extract candidate training data.

[0042] <Third Embodiment> In the first and second embodiments, video A and video B were filmed during at least the same filming period. Therefore, in the training data candidate acquisition process of the first and second embodiments, training data candidates cannot be acquired when videos A and B are filmed at different times (without overlapping filming periods) by the same or multiple imaging devices, as shown in Figure 11. In the third embodiment, a training data candidate acquisition process for appropriately acquiring training data candidates for videos A and B as shown in Figure 11 will be described. In the training data candidate acquisition process of the third embodiment, an index representing the frame similarity between the frames of video A and the frames of video B is calculated, and pairs of frames with an index above a predetermined threshold are registered in the candidate database D1. As mentioned above, SSIM can be used as the index representing frame similarity. In the similarity determination, the image of the frame of video A is reduced to a resolution of XB, and an index representing the similarity is calculated using this and the image of each frame of video B. However, the image of the frame of video A does not need to be reduced, and the resolution after reduction may be other than XB.

[0043] Figure 12 is a flowchart illustrating the training data candidate acquisition process according to the third embodiment. The following explanation will primarily refer to the flowchart in Figure 10 to describe the differences from the training data candidate acquisition process of the first embodiment (Figure 5).

[0044] In S1201, the candidate acquisition unit 113 selects one frame from video A and reads the frame data of the selected frame. The candidate acquisition unit 113 selects one frame from the beginning of video A recorded in the recording unit 106 (hereinafter, the selected frame will be referred to as frame Ax), and transfers the frame data of frame Ax from the recording unit 106 to the RAM 103 by referring to table PA recorded in the RAM 103.

[0045] In S1202, the candidate acquisition unit 113 calculates the similarity between frame Ax, which was read in S1201, and each frame of video B. More specifically, the candidate acquisition unit 113 refers to the position information (related to frame data) of table PB and sequentially acquires the frame data of each frame of video B from the recording unit 106 to the RAM 103. Then, the candidate acquisition unit 113 calculates the similarity index between frame Ax and each frame using the similarity index calculation function (SSIM in this embodiment) and records it in the RAM 103. In S1203, the candidate acquisition unit 113 acquires the video B frame with the highest value among the similarity indexes calculated in S1202 as frame Bx. The processing in S1204 to S1205 is the same as S504 to S505 in the first embodiment (Figure 5).

[0046] As described above, according to the third embodiment, appropriate candidate training data can be obtained even if the shooting periods of the two image groups (video A and video B) do not overlap.

[0047] <Fourth Embodiment> In the fourth embodiment, we will describe how to improve the performance of the learning model M by taking image similarity into account in the learning process of the first to third embodiments. As described in the first embodiment, appropriate training data is extracted for the frame By selected in S701 in Figure 7, and the learning model M is generated / updated using this training data in S704. When generating / updating the learning model M, tuning of the network parameters is performed using backpropagation, as shown in Figure 8. In the fourth embodiment, the intensity of tuning by backpropagation is controlled based on the attributes (e.g., shooting time) of the frame By that is the target of high resolution and high definition enhancement, or the images of those frames, for the frame (image E) used for learning. More specifically, in the learning process, the learning unit 151 determines coefficients such that the stronger the similarity between each frame in the sequentially input frame group UB and frame By, the stronger the influence of backpropagation on updating the network parameters, and the weaker the similarity. Here, image similarity may be simply determined based on the time difference between frame By and the input image E, or it may be determined by comparing the images of both frames using SSIM or the like. For example, when using the former method (the method using time differences), one possible configuration is to multiply the tuning intensity by a coefficient of 1 if the time difference is less than the threshold, and by a coefficient of 0.5 if the time difference is greater than or equal to the threshold. if(ABS(Time difference between By and E)<threshold){ Coefficient = 1 else { Coefficient = 0.5 }

[0048] On the other hand, when using the latter method (the method using similarity), one possible configuration is to use the structural time similarity index (SSIM) as a coefficient for tuning intensity, as shown below. Coefficient = SSIM(By and E) [0 ≤ SSIM(x) ≤ 1]

[0049] Regarding how to adjust the strength of the influence, examples include multiplying the network parameter update rate in backpropagation by the aforementioned coefficient, or multiplying the number of training loops for the input image E by the coefficient without multiplying the parameter update rate by the coefficient.

[0050] <Fifth Embodiment> In the first to third embodiments, a configuration was described in which pairs of frames from video A and video B are extracted as candidate training data and registered in the candidate database D1. In the fifth embodiment, video A' is generated by converting video A to the resolution XB of video B, and the candidate acquisition unit 113 acquires candidate training data using video A and video A'. That is, the candidate acquisition unit 113 of the fifth embodiment extracts frame Ax' with the same frame number as frame Ax of video A from video A', and registers the pair of frame Ax and frame AX' as candidate training data in the candidate database D1. The fifth embodiment will be described in detail below.

[0051] (Description of the configuration of the image processing device 100) The hardware and functional configuration of the image processing device 100 is the same as in the first embodiment (Figure 1). However, the control unit 101 of the fifth embodiment also has a resolution conversion function for reducing the resolution of an image using bicubic interpolation. When performing a resolution reduction process on image data recorded in RAM 103, this resolution conversion function calculates the pixel value of a pixel that requires interpolation by referring to its surrounding pixels.

[0052] (Data recorded in the recording unit 106 and its decoding and decompression method) In the first embodiment, video a and video b stored in the recording unit 106 are converted to an uncompressed format, and the decoded video a is recorded as video A, and the decoded video b is recorded as video B in the recording unit 106. In the fifth embodiment, video A' is generated by converting video A to the resolution XB of video B. More specifically, the control unit 101 refers to table PA recorded in RAM 103 and sequentially inputs the frame data of the frames of video A (hereinafter referred to as frame K) stored in the recording unit 106 to the resolution conversion function of the control unit 101. Then, the resolution conversion function outputs a frame of frame data with resolution XB (hereinafter referred to as frame K'). The control unit 101 refers to table PA and multiplexes it with the shooting time information of frame K read from the recording unit 106 and records it in the recording unit 106 as a frame of video A'. In addition, table PA' which holds the frame number of each frame of video A', the position information of the frame data, and the position information of the shooting time data is recorded in RAM 103. An example of these videos A, B, and A' is shown in Figure 13. For each frame image (A1 to An) of video A, reduced images (A1' to An') with a resolution of XB are generated and recorded in the recording unit 106 as video A'. In the above example, the resolution of video A is reduced to XB, but this is not the only option. Video A' can be any image converted to have a lower resolution than video A. However, by using an image converted to the same resolution as the image to be high-resolution, a learning model more suitable for the image to be high-resolution can be constructed.

[0053] (Process for acquiring candidate training data) Figure 14 shows the configuration and operation of the functional blocks in image processing by the image processing apparatus 100 of the fifth embodiment. The candidate acquisition unit 113 acquires combinations of frames with the same frame number for each frame of video A and video A', and registers them in the candidate database D1. More specifically, the candidate acquisition unit 113 searches for frames in video A' that have matching frame numbers for each frame of video A listed in table PA, by referring to table PA'. The candidate acquisition unit 113 assigns a unique index I to combinations of frames from video A and video A' that have the same frame number, and registers them in the candidate database D1. The group of frames from video A registered in the candidate database D1 is called TA, and the group of frames from video A' is called TA'.

[0054] (High-definition video generation processing) The following explanation will mainly describe the differences from the process of the first embodiment (Figure 6) with reference to the flowchart in Figure 15.

[0055] The processing in S1501 is the same as in S701 in the first embodiment (Figure 7). In S1502, the training data extraction unit 114 extracts frames from the frame group TA' of candidate training data registered in the candidate database D1 in which the difference in shooting time with frame By is shorter than a threshold predetermined by the system. As the threshold, for example, the display period of one frame of video A (the display period of one frame at frame rate XA) can be used. The training data extraction unit 114 registers the extracted frames in the training data database D2. Specifically, first, the training data extraction unit 114 obtains the time information of the frames registered in the frame group TA' by referring to table PA'. The training data extraction unit 114 registers the frames from the acquired time information of the frame group TA' in the training data database D2 on RAM 103 in which the difference in time with frame By is shorter than the threshold. Hereinafter, the frame group of video A' registered in the training data database D2 will be referred to as frame group UA'. In this embodiment, frames whose difference in capture time from frame By is shorter than a predetermined threshold are extracted from the candidate database D1, but this is not limited to this. For example, frames in frame group TA' in which an index representing the similarity between the image of each frame of frame group TA' and the image of frame By (e.g., SSIM) is higher than a threshold predetermined by the system may be extracted from frame group TA' and registered in the training data database D2.

[0056] In S1503, the training data extraction unit 114 registers the frames of frame group TA that are associated with each frame of frame group UA' by index I in the training data database D2. Specifically, the training data extraction unit 114 refers to the candidate database D1 on RAM 103 and registers the frames of frame group TA that are associated with each frame of frame group UB by index I in the training data database D2. At this time, the associated combinations (pairs of frames) are not changed, and each combination is assigned a unique index J within the training data database D2. Hereinafter, the frame group of video A registered in the training data database D2 will be referred to as frame group UA.

[0057] In S1504, the learning unit 151 performs learning using frame group UA and frame group UA' by referring to the training data database D2 and generates a learning model M. Specifically, first, the learning unit 151 reads frame data from the recording unit 106 by referring to the training data database D2 and tables PA and PA', and inputs it into the learning model generation function. The learning unit 151 performs learning using the frame data read by the learning model generation function, and saves the learning model M generated as a result of the learning to RAM 103. Details of the learning of the learning model are as described above with reference to Figure 8. The processing in S1505 and S1506 thereafter is the same as in the first embodiment (processing in S705 and S706 in Figure 7).

[0058] As described above, according to each embodiment, training data used to train the learning model is selected based on the image to be re-resolution. Therefore, the learning model trained using the selected training data can infer the high-frequency components of the image to be re-resolution with higher accuracy, and a highly accurate re-resolution image can be obtained. In other words, the accuracy of video super-resolution technology for re-resolution video is improved.

[0059] In the above embodiment, when acquiring candidate training data, the image paired with the image selected from video A is an image selected from video B based on the shooting time or image similarity, or a low-resolution version of the selected image, but is not limited to these. The image associated with the image selected from video A used as candidate training data only needs to have a lower resolution than the selected image and be an image associated with the selected image. Whether or not an image is associated with the image selected from video A may be determined, for example, based on commonalities such as the temperature at the time of shooting, the shooting location, and the shooting direction.

[0060] Furthermore, although the above embodiments describe a two-step process of generating a candidate database D1 and then generating a training data database D2, the process is not limited to this. For example, the training data extraction unit 114 may extract frames from video A that can be paired with training data based on frame By, and obtain training data using the extracted frames and the frames related to the extracted frames as pairs. However, when sequentially increasing the resolution of multiple images of video B, it is more efficient to create a candidate database D1 as in each of the above embodiments and extract and use appropriate training data from the candidate database D1 according to the image to be increased in resolution.

[0061] Furthermore, while the above embodiments process video a and video b with a lower resolution than video a, the process is not limited to these two. For example, an uncompressed video a and video b obtained by compressing and then restoring video a may be used as processing targets. In this case, video a may be saved with frames downsampled. Thus, the relationship between video a and video b to be processed in each of the above embodiments is not limited to the size of their resolutions; it is sufficient that video a is clearer than video b. That is, the group of images constituting video a (video A) should contain more high-frequency components than the group of images constituting video b (video B). For example, if each image in the group of images in video a corresponds to one or more images in the group of images in video b, and each image in the group of images in video a has more high-frequency components than the corresponding image in the group of images in video b, the processing of each embodiment can be applied.

[0062] Furthermore, although the above explanation has focused on video data for simplicity, the above embodiments can also be applied to the following case, for example, if the device can generate still images at predetermined timings while recording video. That is, when still images are used as data corresponding to video a, and video is used as data corresponding to video b. For example, let's assume that one of the above embodiments is applied to an imaging device that shoots in 6K data size in Raw format at 60fps using an image sensor. Then, let's assume that the still images are data recorded in a format such as JPEG or HEIF after development processing and still image compression while maintaining the 6K size. And, let's assume that the video is data (video data of 2K size at 60fps) obtained by converting the 6K data obtained from the image sensor to a data size of 2K, and then recording the acquired Raw data in a format such as MP4 after development processing and video compression. Under these assumptions, when the user presses the release switch to take continuous shots of still images while the imaging device is recording 2K video data at 60fps, 6K still images are generated at intervals of, for example, 10fps relative to the video frame rate (60fps). By applying one of the above embodiments to the still images and videos thus generated, data of still image quality corresponding to a video taken during a period in which multiple still images were captured can be generated. In other words, a system can be realized that produces video as if it were shot at 6K resolution (the size of a still image) and a frame rate of 60fps. In this case, still images and videos are prepared in the imaging device, and learning and inference processing are performed within the imaging device to generate data of still image quality corresponding to the video.

[0063] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0064] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are attached to make the scope of the invention public. [Explanation of Symbols]

[0065] 101: Control Unit, 102: ROM, 103: RAM, 104: Decoding Unit, 105: Learning and Inference Unit, 106: Recording Unit, 107: Bus, 111: Analysis Unit, 112: Decoded Video Generation Unit, 113: Candidate Extraction Unit, 114: Training Data Extraction Unit, 151: Learning Unit, 152: Inference Unit

Claims

1. An image processing apparatus that uses a first group of images to enhance the resolution of a second group of images having fewer high-frequency components than the first group of images, A selection means for selecting training data to be used for learning from among training data that includes one image in the first image group as one of an image pair, and an image having fewer high-frequency components than the image in the first image group and related to the image in the first image group as the other of the image pair, based on the image to be high-resolution selected from the second image group, An inference means that learns a learning model using the training data selected by the selection means, and infers the high-frequency components of the image to be high-resolution using the learning model, An image processing apparatus comprising: a generation means for generating a high-resolution image based on the image to be high-resolution and the high-frequency components inferred by the inference means.

2. The image processing apparatus according to claim 1, characterized in that the selection means selects training data to be used for learning by selecting an image from a second image group that is related to an image selected from the first image group.

3. The image processing apparatus according to claim 2, characterized in that the selection means selects an image from the second group of images having the same shooting time as the selected image as an image associated with the selected image.

4. The image processing apparatus according to claim 2, characterized in that the selection means selects from the second group of images an image having a shooting time less than a predetermined threshold than the shooting time of the selected image, as an image associated with the selected image.

5. The image processing apparatus according to claim 2, characterized in that the selection means selects from the second group of images the image with the greatest similarity to the selected image as an image related to the selected image.

6. The first group of images has a first resolution, and the second group of images has a second resolution lower than the first resolution. The image processing apparatus according to claim 5, characterized in that the selection means obtains similarity between an image obtained by reducing the selected image to the second resolution and the images in the second group of images.

7. The image processing apparatus according to claim 1, characterized in that the selection means acquires an image obtained by reducing the resolution of the selected image as an image related to the selected image.

8. The first group of images has a first resolution, and the second group of images has a second resolution lower than the first resolution. The image processing apparatus according to claim 7, characterized in that the image associated with the selected image is an image obtained by reducing the selected image to the second resolution.

9. The image processing apparatus according to any one of claims 1 to 8, characterized in that the selection means selects candidate training data, which includes images taken at a time when the difference from the time of acquisition of the image to be high-resolution is less than a predetermined threshold, as training data to be used for learning.

10. The image processing apparatus according to any one of claims 1 to 8, characterized in that the selection means selects candidate training data, which includes images whose similarity to the image to be high-resolution is greater than a predetermined threshold, as training data to be used for learning.

11. The image processing apparatus according to any one of claims 1 to 10, characterized in that the inference means controls the updating of parameters by backpropagation in the learning process based on the training data used for learning and the image to be refinished.

12. The image processing apparatus according to claim 11, wherein the inference means determines coefficients based on the training data used for learning and the image to be refinished, and controls the amount of parameter update by backpropagation based on the coefficients.

13. The image processing apparatus according to claim 11, wherein the inference means determines coefficients based on the training data used for learning and the image to be refined, and controls the number of iterations of parameter updates by backpropagation based on the coefficients.

14. The image processing apparatus according to claim 12 or 13, characterized in that the inference means determines the coefficient based on the difference between the time the training data images used for learning are captured and the time the images to be high-resolution are captured.

15. The image processing apparatus according to claim 12 or 13, characterized in that the inference means determines the coefficient based on the similarity between the image of the training data used for learning and the image to be high-resolution.

16. The image processing apparatus according to any one of claims 1 to 15, characterized in that the first image group and the second image group are two image groups obtained by performing different image processing on a single image captured by a single image sensor of a single imaging device.

17. The image processing apparatus according to any one of claims 1 to 16, characterized in that the first image group and the second image group are image groups captured by each of the two image sensors.

18. The image processing apparatus according to any one of claims 1 to 17, characterized in that the frame rate of the first image group is lower than the frame rate of the second image group.

19. An image processing method that uses a first group of images to enhance the resolution of a second group of images having fewer high-frequency components than the first group of images, A selection step of selecting training data to be used for learning from among training data that includes one image in the first image group as one of an image pair, and an image having fewer high-frequency components than the image in the first image group and related to the image in the first image group as the other of the image pair, based on the image to be high-resolution selected from the second image group, An inference step is performed to train a learning model using the training data selected in the selection step, and to infer the high-frequency components of the image to be high-resolution using the learning model. An image processing method characterized by comprising a generation step of generating a high-resolution image based on the image to be high-resolution and the high-frequency components inferred by the inference step.

20. A program for causing a computer to function as each means of the image processing apparatus described in any one of claims 1 to 17.

Images