Image processing device, method, and program

Abstract

To provide an image processing device capable of performing high definition processing on an image at high accuracy by using machine learning.SOLUTION: An image processing device for performing high definition processing on an image of a second image group having less high frequency component of a corresponding frame than a first image group by using the first image group includes: a selection unit which selects a pair of pieces of teacher data used for learning from among a plurality of pairs of pieces of teacher data containing an image contained in a first image group as one among a pair of images, on the basis of an image of a high definition process object selected from a second image group; a learning model generation unit which generates a learning model by using a pair of pieces of teacher data selected by the selection unit; a deduction unit which deduces a high frequency component of an image being a high definition process object by using the generated learning model; and an image generation unit which generates a high-definition-processed image on the basis of the image being the high definition process object and the high frequency component deduced by the deduction unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, method, and program for increasing the resolution of a group of images using machine learning. [Background technology]

[0002] Super-resolution technology using machine learning generates high-resolution images by using machine learning to infer high-frequency components that cannot be compensated for by linear interpolation of pixel values ​​when enlarging an image and performing resolution conversion. In super-resolution technology, a learning model is first generated using a group of images G and degraded images of each image in G, degraded by a given method, as training data. The learning model learns the pixel value differences between the original image and the degraded image and updates its own super-resolution processing parameters. When an image H lacking high-frequency components is input to the learning model, the high-frequency components are acquired through inference using the learning model. A high-resolution image can be generated by overlaying the high-frequency components acquired through inference on image H. When performing super-resolution processing on a video, a high-resolution video can be generated by inputting all frames one by one into the learning model.

[0003] Generally, when a product or service using a learning model is provided, the developer collects training data and generates a learning model, and then provides the generated learning model to the user. Therefore, the content of the video input by the user is unknown at the time of the training process. Therefore, the developer prepares a large amount of diverse images with no bias in image patterns as training data and repeatedly trains the system to ensure uniform accuracy for all target videos.

[0004] For example, Patent Document 1 describes a technology for performing super-resolution processing on a video using a learning model that has learned from a variety of images. However, because the training data is diverse, there is very little training data that is highly similar to the inference target video Q specified by the user. When such a learning model is used, the results of learning images that are low in similarity to the inference target video Q are reflected in the inference processing. As a result, the improvement is limited to an improvement in resolution due to the emphasis on the edges of the subject, and it is difficult to accurately infer high-frequency components such as the fine patterns of the subject, and the inference accuracy cannot be said to be high.

[0005] As an example of a system for solving such problems, Patent Document 2 describes a method in which the user learns using only images that are similar in location and conditions to the video to be inferred as training data, thereby obtaining videos with higher resolution than when learning from a variety of images. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-204167 [Patent Document 2] Japanese Patent Application Publication No. 2019-129328 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 2, learning is performed using training data captured at the same location but at different times. More specifically, video previously captured along a section S of a bus route is accumulated and learned, and the resulting learning model is used to perform inference on real-time video of section S. In this case, the training data is limited to images captured within section S, which are a group of images with a relatively high similarity to the inference target, and thus improved inference accuracy can be expected. However, even among the videos captured within section S, for example, the video at the start of section S and the video at the end of section S were captured in different locations, and the subjects depicted are completely different, making them less similar. This reduces the inference accuracy for the entire section S. In addition, the subjects depicted in the past video serving as training data and the real-time video of the inference target are different, even when captured at the same location. Accurate inference cannot be performed for subjects that have not been trained, which also reduces inference accuracy.

[0008] Patent Document 2 also describes a method for classifying past video footage into multiple groups based on shooting conditions such as weather, independently learning data from each group to generate multiple learning models, and then switching the learning model used depending on the shooting conditions of the real-time video. This technology also makes it possible to suppress a decrease in inference accuracy due to differences in shooting conditions. However, even if conditions such as weather are the same, even slight differences in values ​​such as illuminance will result in differences in the frequency components of the training data and the inference target, so it cannot be said that a decrease in inference accuracy can be sufficiently suppressed. Due to these factors, even the technology described in Patent Document 2 does not provide sufficient inference accuracy for high-frequency components.

[0009] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can use machine learning to improve the resolution of images with high accuracy. [Means for solving the problem]

[0010] An image processing device according to the present invention is an image processing device that uses a first image group to increase the resolution of images of a second image group having fewer high-frequency components in corresponding frames than the first image group, an acquisition means for acquiring, as candidates for training data, a plurality of pairs of a first image selected from the first image group and a third image having fewer high frequency components than the first image and having a shooting time that differs from the shooting time of the first image by less than a predetermined threshold; The aforementioned Multiple pairs of candidate training data The image processing system is characterized by comprising a selection means for selecting a pair of training data to be used for learning from among the above based on an image to be refined selected from the second image group, a learning model generation means for generating a learning model using the pair of training data selected by the selection means, an inference means for inferring high-frequency components of the image to be refined using the generated learning model, and an image generation means for generating a high-resolution image based on the image to be refined and the high-frequency components inferred by the inference means. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image processing device that can use machine learning to improve the resolution of an image with high accuracy. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating the functional configuration of an image processing apparatus according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a frame configuration of an input moving image according to the first embodiment. [Figure 4] FIG. 1 is a diagram illustrating the functional configuration of an image processing apparatus according to a first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of the data configuration of a candidate database according to the first embodiment. [Figure 6] 10 is a flowchart of a teacher data candidate acquisition process according to the first embodiment. [Figure 7] 4 is a flowchart of high-definition video generation processing according to the first embodiment. [Figure 8] FIG. 2 is a schematic diagram illustrating a learning / inference process according to the first embodiment. [Figure 9]FIG. 11 is a diagram showing an example of a frame configuration of an input moving image in the second embodiment. [Figure 10] 10 is a flowchart of a teacher data candidate acquisition process according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a frame configuration of an input moving image according to the third embodiment. [Figure 12] 10 is a flowchart of a teacher data candidate acquisition process according to the third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a frame configuration of a moving image according to the fifth embodiment. [Figure 14] FIG. 11 is a diagram for explaining the functional configuration of an image processing apparatus according to a fifth embodiment. [Figure 15] 13 is a flowchart of high-definition video generation processing in the fifth embodiment. [Figure 16] 10 is a flowchart of high-definition moving image generation processing in the sixth, seventh, eighth, and ninth embodiments. [Figure 17] FIG. 20 is a diagram showing an example of learning / inference processing in the sixth embodiment. [Figure 18] 13 is a flowchart of high-definition moving image generation processing in the eighth embodiment. [Figure 19] FIG. 23 is a diagram showing an example of region selection of training data according to the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] First Embodiment (Outline of image processing device) The image processing apparatus according to the first embodiment takes as input two videos A and B that are simultaneously captured by the same imaging device. The relationship between the resolution XA and frame rate FA of video A and the resolution XB and frame rate FB of video B is "XA > XB and FA < FB". The image processing apparatus has a function (high-definition video generation function) of generating a learning model using the frames of videos A and B and generating a video C with resolution XA and frame rate FB from video B by inference using the generated learning model.

[0015] (Description of the configuration of the image processing apparatus) FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus 100 according to the first embodiment. The control unit 101 is an arithmetic device such as a CPU, and realizes various functions by expanding and executing the program stored in the ROM 102 in the work area of the RAM 103. The control unit 101 can function as, for example, each functional block of an analysis unit 211 and a decoded video generation unit 212 described later with reference to FIG. 2, and a candidate acquisition unit 413 and a teacher data extraction unit 414 described later with reference to FIG. 4. The ROM 102 stores a control program executed by the control unit 101. The RAM 103 is used for a work memory for the control unit 101 to execute the program and a temporary storage area for various data.

[0016] The decoding unit 104 decodes video or image data compressed in an encoding format defined by the Moving Picture Experts Group (hereinafter abbreviated as MPEG) into uncompressed data. The learning and inference unit 105 has a functional block (learning unit 451, described later in FIG. 4) that receives training data and generates / updates a learning model. The learning and inference unit 105 also has a functional block (inference unit 452, described later in FIG. 4) that analyzes an input image using the learning model generated by learning and infers high-frequency components to generate a high-resolution image of the input image. In this embodiment, a CNN model for super-resolution processing based on a convolutional neural network (hereinafter abbreviated as CNN) is used as the learning model. This model enlarges the input image by linear interpolation, generates high-frequency components to be added to the enlarged image, and then adds and combines the two.

[0017] The recording unit 106 is composed of a recording medium, such as a hard disk drive (HDD) or a memory card, detachably connected to the image processing device 100, and a recording medium control device that controls the recording medium. The recording medium control device controls initialization of the recording medium and data transfer between the recording medium and RAM 103 for reading and writing data, in accordance with commands from the control unit 101. The bus 107 is an information communication path connecting each function. The control unit 101, ROM 102, RAM 103, decoding unit 104, learning and inference unit 105, and recording unit 106 are connected to each other so that they can communicate with each other.

[0018] Note that the hardware blocks described in this embodiment and the functional blocks realized therein do not necessarily have to be configured as described above. For example, two or more of the control unit 101, the decoding unit 104, and the learning and inference unit 105 may be realized by a single piece of hardware. Furthermore, the function of one functional block or the functions of multiple functional blocks may be executed by the coordinated operation of several pieces of hardware. Furthermore, each functional block may be realized by a CPU executing a computer program deployed on memory, or may be realized by dedicated hardware. Furthermore, a configuration may be adopted in which some of the functional blocks reside on a cloud server, and processing result data is transferred via communication.

[0019] (Data recorded on the recording medium and its decryption and expansion methods) FIG. 2 is a diagram illustrating functional blocks for the control unit 101 (analysis unit 211 and decoded video generation unit 212) that perform decompression processing of compressed video data. The recording unit 106 stores video a and video b, which are input data for the high-definition video generation processing. Here, video refers to one or more temporally continuous image data. In this embodiment, video a and video b are simultaneously captured by an imaging device having an image sensor and compressed using the MPEG format. Video a and video b may be generated by thinning and reducing images captured by a single image sensor, or by capturing the same subject using image sensors with different resolutions and frame rates. Hereinafter, video a and video b are assumed to be two groups of images obtained by performing different image processing on a single image captured by a single image sensor in a single imaging device. The video data for video a and video b are each compressed using the MPEG format, multiplexed with shooting time information, and stored in MP4 format. Note that other formats may be used as long as the image data and corresponding shooting time information can be acquired as a pair from the recording unit 106.

[0020] The analysis unit 211 has the function of parsing video data (in this example, an MP4 format file) recorded in the recording unit 106 and calculating the storage locations within the file of the included compressed image data and the time information registered as metadata. In the MP4 format, location information indicating the recording location within the file of each frame of data and shooting time information is recorded in the Moov portion. The analysis unit 211 extracts the Moov portion of video a from the recording unit 106 into RAM 103 and parses it, generating a table Pa that contains the frame numbers in video a, the location information of the frame data, and the location information of the shooting time. The analysis unit 211 also parses the Moov portion of video b in the same way and generates a table Pb that contains the frame numbers in video b, the location information of the frame data, and the location information of the shooting time. Tables Pa and Pb are stored in RAM 103.

[0021] To use the moving images a and b in the high-definition moving image generation process, the moving images a and b must be converted into uncompressed formats. As shown in FIG. 2, the decoded moving image generation unit 212 of the control unit 101 decodes the moving images a and b to generate moving images A and B, which are then recorded in the recording unit 106. More specifically, the decoded moving image generation unit 212 references tables Pa and Pb stored in RAM 103 and sequentially inputs the frame data of the moving images a and b stored in the recording unit 106 to the decoding unit 104. The decoded moving image generation unit 212 multiplexes the uncompressed frame data output by the decoding unit 104 with shooting time information obtained by reference to tables Pa and Pb, and records the multiplexed data in the recording unit 106. Hereinafter, it is assumed that moving image A is obtained by decoding moving image a, and moving image B is obtained by decoding moving image b. The decoded moving image generation unit 212 also generates table PA, which contains frame numbers, frame data location information, and shooting time location information for moving image A, and records the table PA in RAM 103. Similarly, the decoded video generation unit 212 generates a table PB having frame numbers in video B, position information of frame data, and position information of shooting time, and stores this in RAM 103. An example of the frame structure of video A and video B is shown in Fig. 3. In Fig. 3, the total number of frames of video A is n, and the total number of frames of video B is m. Frame pairs indicated by dashed lines (multiple pairs of images A1 and B2, A2 and B5, A3 and B8, etc.) are paired frames with the same shooting time information, indicating that the images of these frames were shot at the same timing.

[0022] Next, a process for generating a high-definition image according to this embodiment will be described. This process is roughly divided into two parts: a "training data candidate acquisition process" and a "high-definition video generation process."

[0023] FIG. 4 is a diagram illustrating the operation of functional blocks in image processing by the image processing device 100 of the first embodiment. As described in FIG. 2, video A and video B are stored in the recording unit 106, and tables PA and PB are stored in RAM 103. The "trainer data candidate acquisition process" is performed by the candidate acquisition unit 413, and the "high-definition video generation process" is performed by the trainer data extraction unit 414, learning unit 451, and inference unit 452. The candidate acquisition unit 413 extracts pairs of frames from the frame group of video A and the frame group of video B that are candidates for training data as trainer data candidates, and generates a candidate database D1. The frame By to be subjected to high resolution and high definition is obtained from the frame group of image B. The trainer data extraction unit 414 extracts trainer data further suitable for training from the trainer data candidates registered in the candidate database D1 to generate a training model suitable for inferring high-frequency components of frame By, and generates a trainer database D2. The learning unit 451 of the learning and inference unit 105 generates a learning model M for the frame By using the teacher database D2. The inference unit 452 inputs the frame By to be high-resolution-enhanced into the learning model M generated by the learning unit 451, and performs high-resolution processing on the frame By. The "teacher data candidate acquisition processing" and "high-resolution video generation processing" will be described in more detail below.

[0024] (Processing to obtain candidate training data) In the teacher data candidate acquisition process, the control unit 101 (candidate acquisition unit 413) generates a candidate database D1. In the first embodiment, the candidate acquisition unit 413 acquires pairs of frames from video A and video B that have the same shooting time as teacher data candidates, from among the frames of video A and video B. Specifically, all frame pairs that have a common shooting time in video A and video B (frame pairs indicated by dashed lines in FIG. 3) are acquired as teacher data candidates. Before the learning process described below is performed, the candidate acquisition unit 413 investigates which frames can be used as teacher data, constructs a candidate database D1, and registers the investigation results.

[0025] FIG. 5 shows an example of the data structure of the candidate database D1. The candidate database D1 stores the frame numbers within each video file of a frame group TA from video A that can be used as training data, and a frame group TB from video B that can be used as training data. Pairs of frames (pairs of frame numbers) that are captured at the same time are associated and registered using a unique index I within the candidate database D1. For example, in the video A and video B shown in FIG. 3, the frame pairs A1 and B2, A2 and B5, and A3 and B8 (omitted below) are combinations of frames captured at the same time. The candidate database D1 shown in FIG. 5 records these pairs by frame number and assigns a unique index I. In this way, the acquired training data candidates are managed by the candidate database D1.

[0026] The details of the teacher data candidate acquisition process described above will be further explained using the flowchart in FIG. 6. In S601, the candidate acquisition unit 413 selects one frame of the video A and acquires time information corresponding to the selected frame from table PA. In this embodiment, the frames are selected sequentially from the beginning of the video A recorded in the recording unit 106. More specifically, the candidate acquisition unit 413 selects one frame sequentially from the beginning of the video A recorded in the recording unit 106. Hereinafter, the selected frame will be referred to as frame Ax. The candidate acquisition unit 413 reads the time information corresponding to frame Ax from the recording unit 106 by referring to table PA recorded in RAM 103, and transfers it to RAM 103.

[0027] In S602, the candidate acquisition unit 413 compares the time information of frame Ax read out in S601 with the time information of each frame of video B. Specifically, the candidate acquisition unit 413 refers to the positional information of the shooting times recorded in table PB, sequentially acquires the shooting time information of each frame of video B from the recording unit 106, and compares it with the time information of frame Ax. In S603, the candidate acquisition unit 413 acquires a frame of video B having a shooting time that matches the time information of frame Ax, and designates this as frame Bx.

[0028] In S604, the candidate acquisition unit 413 assigns a unique index Ix to the combination of the above frame Ax and frame Bx within the candidate database D1 and registers it in the candidate database D1. Specifically, the candidate acquisition unit 413 issues a unique index Ix to the combination of frame Ax and frame Bx, and registers the index Ix, the frame number of frame Ax within video A, and the frame number of frame Bx within video B in the candidate database D1.

[0029] In S605, the control unit 101 determines whether the above-described processes of S601 to S604 have been completed for all frames of the video A. If the control unit 101 determines that the processes have been completed (YES in S605), it ends the process. If the control unit 101 determines that the processes have not been completed (NO in S605), it returns the process to S601 and executes the above-described process for the next frame of the video A. By the above-described processes, the candidate database D1 is generated.

[0030] In this embodiment, in S602, the pair of frames to be registered in the candidate database D1 is determined by comparing the shooting times, but this is not limited to this. For example, the frame Ax may be reduced to a resolution XB, and a similarity determination may be performed using an index representing the similarity between the frame Ax and each frame of the video B, and the determination result may be used to select the pair of frames to be registered in the candidate database D1. In this case, the candidate acquisition unit 413 has a similarity determination function that compares two or more image data and determines the similarity. Note that, for example, the structural similarity index (SSIM) may be used as an index representing the similarity of the images. Furthermore, in acquiring the index representing the similarity, the image of the frame Ax is reduced to a resolution XB, but this is not limited to this. The image of the frame Ax may not be reduced, or the reduced resolution may be a resolution other than XB.

[0031] (High-definition video generation processing) Next, the high-definition video generation process performed by the control unit 101 (teacher data extraction unit 414) and the learning and inference unit 105 (learning unit 451, inference unit 452) will be described. First, an overview of the high-definition video generation process will be described with reference to FIG. 4. The teacher data extraction unit 414 selects teacher data suitable for learning a "learning model for the inference target frame By" from the candidate database D1 and generates a teacher database D2 (FIG. 4) (details will be described later with reference to S702 to S703 in FIG. 7). The learning unit 451 generates a learning model using the extracted teacher data (S704). Then, the inference unit 452 infers high-frequency components of the inference target frame By using the learning model and performs high-definition processing (S705), thereby obtaining a high-definition frame (image) Cy of the inference target frame By. Note that before the high-definition video generation process begins, the control unit 101 generates a video C on the recording unit 106. At the start of high-definition video generation, the video C does not contain any frame data and is empty. The inference unit 452 records the generated frames Cy in the moving image C in sequence.

[0032] The above-mentioned high-definition video generation process will be described in more detail below with reference to the flowchart in Fig. 7. In S701, the teacher data extraction unit 414 reads one frame from video B as a frame to be refined. In this embodiment, the teacher data extraction unit 414 reads frames one by one in order from the beginning of video B recorded in the recording unit 106. Hereinafter, the frame read in S701 will be referred to as frame By. More specifically, the teacher data extraction unit 414 reads the frame data and shooting time information of frame By from the recording unit 106 with reference to table PB, and transfers them to RAM 103.

[0033] In S702, the teacher data extraction unit 414 extracts frames from the teacher data candidates TB registered in the candidate database D1 whose difference in shooting time with respect to frame By is shorter than a threshold value predefined by the system, and registers these frames in the teacher database D2. For example, the threshold value can be the display period of one frame of video A (the display period of one frame at frame rate XA). The structure of the teacher database D2 is similar to that of the candidate database D1 (FIG. 5). Specifically, the teacher data extraction unit 414 first acquires time information for each of the frames TB registered in the candidate database D1 by referring to the position information in table PB. The teacher data extraction unit 414 compares each of the acquired time information with the shooting time of frame By, extracts frames whose difference in time is shorter than the threshold value from the frame group TB, and registers these frames in the teacher database D2 on RAM 103. Hereinafter, the frame group of video B registered in the teacher database D2 by the above process is referred to as UB. In this embodiment, when constructing the teacher database D2, frame groups whose shooting time with respect to frame By is shorter than the threshold value are extracted from the candidate database D1, but this is not limited thereto. The frame group UB may be extracted using an index representing the similarity with the frame By. For example, the teacher data extraction unit 414 may use SSIM to extract from the frame group TB a frame group whose similarity index with the frame By is higher than a threshold value predetermined in advance for the system, and register the frame group UB.

[0034] In S703, the teacher data extraction unit 414 registers in the teacher database D2 the frames of the frame group TA that are paired with each frame of the frame group UB in the candidate database D1. Specifically, the teacher data extraction unit 414 references the candidate database D1 on RAM 103 and registers in the teacher database D2 the frames of the frame group TA that are associated with each frame of the frame group UB by index I. At this time, the combination of the two associated frames is not changed, and a unique index J is assigned to each combination within the teacher database D2. Hereinafter, the frame group of the video A registered in the teacher database D2 will be referred to as UA.

[0035] In S704, the learning unit 451 performs learning using the training data (frame group UA and frame group UB) registered in the training database D2, and generates a training model M.

[0036] FIG. 8 is a diagram illustrating the learning model generation function of the learning unit 451. The learning model generation function includes a learning process and an inference process. The inference process is further divided into a feature extraction process using a filter including a CNN and a reconstruction process. First, in the feature extraction process, the learning unit 451 inputs one image from the frame group UB (image E) into the CNN and generates multiple feature maps using the CNN. Next, in the reconstruction process, the learning unit 451 upsamples all feature maps by deconvolution to generate predicted high-frequency components. In the reconstruction process, the learning unit 451 further reconstructs the image by adding the predicted high-frequency components to image E', which is obtained by enlarging image E using a bicubic algorithm or the like, to generate a predicted high-resolution image G. In the learning process, the learning unit 451 compares the predicted high-resolution image G generated in the inference process with image H, which corresponds to image E in the frame group UA, and performs fine tuning of the learning model M using the backpropagation algorithm using the difference between them. The learning unit 451 repeats these processes a predetermined number of times for the same image E to improve the inference accuracy. By performing the above series of processes on each image in the frame group UB, a learning model M suitable for inference processing of the frame group UB is constructed.

[0037] As described above, the learning unit 451 refers to the teacher database D2 and tables PA and PB, reads the frame data of the frame pairs registered as teacher data from the recording unit 106, and inputs the frame data to the learning model generation function described above. The learning unit 451 stores the learning model M generated by the learning model generation function in RAM 103.

[0038] In S705, the inference unit 452 generates a high-resolution frame Cy from the frame By by inference using the learning model M generated in S704. Specifically, first, the inference unit 452 reads the learning model M stored in RAM 103. Next, the inference unit 452 inputs the frame data (image) of frame By stored in RAM 103 in S701 to the CNN of the learning model M, and generates "high-frequency components expected when the image of frame By is enlarged to resolution XA." The inference unit 452 generates an image of a high-resolution frame Cy with resolution XA by adding the generated high-frequency components to "an image obtained by linearly enlarging the image of frame By to resolution XA," and records this in RAM 103. Note that the above-mentioned processing from inferring high-frequency components performed for frame By to generating a high-resolution image is the same as the inference process described above with reference to FIG. 8. The inference unit 452 appends the frame data of the high-definition frame Cy recorded in the RAM 103 to the end of the high-definition video C on the recording unit 106. The inference unit 452 also duplicates the shooting time information of By, multiplexes it as the shooting time of the high-definition frame Cy, and records it in the video C.

[0039] In S706, the control unit 101 determines whether the above processing has been completed for all frames of the moving image B. If the control unit 101 determines that the processing has not been completed (NO in S706), the processing proceeds to S701, the next frame of the moving image B is selected as frame By by the teacher data extraction unit 414, and the above processing is repeated. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S706), the control unit 101 ends this processing. As described above, when the high-definition moving image generation processing is completed, the recording unit 106 records the high-definition moving image C with a resolution of XA and a frame rate of FB in an uncompressed format.

[0040] In the above description, each functional block is described as being realized only by the control unit 101 or only by the learning and inference unit 105, but this is not limiting. For example, each functional block may be realized by cooperation between the control unit 101 and the learning and inference unit 105. For example, the function of the inference unit 452 may be realized by the control unit 101 and the learning and inference unit 105, and the process of recording the high-resolution frame Cy and the shooting time in the video C on the recording unit 106 may be executed by the control unit 101.

[0041] In addition, in this embodiment, the training data candidate acquisition process was performed before the learning process for the entire video and the high-definition video generation process were performed, but this may also be performed in parallel with the execution of the high-definition video generation process. In this embodiment, a new learning model M was created for each frame to be inferred in S704 and the previously generated model was discarded, but this is not limited to this. For example, a learning model M' previously trained externally may be loaded, and additional learning may be performed on the loaded learning model M' using the frame group UA and the frame group UB in S704.

[0042] As described above, according to the first embodiment, the learning model M is used, which is trained using a group of images similar to the image to be refined among groups of images captured during the same shooting period, so that the image can be refined with high accuracy. Furthermore, a pair of images from the two image groups captured at the same time is used as training data, so that more accurate learning is possible.

[0043] Second Embodiment (When recording simultaneously with multiple cameras) In the first embodiment, the candidate training data acquisition process registers in the candidate database D1 a combination of frames from video A and video B with the same capture time. When video A and video B are acquired from videos captured simultaneously using the same image sensor in a single imaging device, frames captured at the same capture time can be acquired from video A and video B, as illustrated in FIG. 3. However, this method may result in inappropriate extraction of candidate training data when video A and video B are captured during the same capture period using multiple image sensors. This is because, as illustrated in FIG. 9, video B does not necessarily contain a frame with a capture time that matches the capture time of a frame in video A. Examples of configurations for capturing video A and video B using multiple image sensors include a configuration in which video A is captured using an imaging device with multiple image sensors and a configuration in which video B is captured using multiple imaging devices with one or more image sensors. In the second embodiment, the candidate training data acquisition process solves the above problem by registering in the candidate database D1 a combination of frames whose capture time difference is shorter than a predetermined threshold, even if the capture times of frames in video A and video B do not match.

[0044] In the second embodiment, the configuration of the image processing device 100 and the high-resolution image generation process are the same as those in the first embodiment, but the process of acquiring the teacher data candidates is partially different. Figure 10 is a flowchart explaining the process of acquiring the teacher data candidates according to the second embodiment. Below, the parts that differ from the process of acquiring the teacher data candidates in the first embodiment (Figure 6) will be mainly explained.

[0045] The processes of S1001 to S1002 are the same as S601 to S602 in the first embodiment (FIG. 6), respectively. In S1003, the candidate acquisition unit 413 acquires, from among the frames of moving image B, a frame whose shooting time difference with respect to the shooting time of one frame Ax of moving image A is shorter than a predetermined threshold, as frame Bx, and registers the frame in the candidate database D1 on RAM 103. Note that an example of such a threshold is the display period per frame at the frame rate XB of moving image B. The subsequent processes of S1004 to S1005 are the same as S604 to S605 in the first embodiment (FIG. 6), respectively.

[0046] As described above, according to the second embodiment, even if the moving images A and B are acquired by a plurality of image sensors, it is possible to appropriately extract training data candidates.

[0047] Third Embodiment (When taken at a different time) In the first and second embodiments, video A and video B were shot during at least the same shooting period. Therefore, in the teacher data candidate acquisition process of the first and second embodiments, when video A and video B were shot at different times (without overlapping shooting periods) by the same or multiple imaging devices, as shown in Figure 11, it is not possible to acquire teacher data candidates. In the third embodiment, a teacher data candidate acquisition process for appropriately acquiring teacher data candidates for video A and video B as shown in Figure 11 will be described.

[0048] In the process of acquiring training data candidates in the third embodiment, an index representing the frame similarity between frames of video A and video B is calculated, and pairs of frames having an index equal to or greater than a threshold value previously set in the system are registered in the candidate database D1. As described above, the index representing frame similarity can be, for example, SSIM. In determining the similarity, the image of the frame of video A is reduced to a resolution XB, and an index representing the similarity is calculated using this and each frame image of video B. However, in this case, the image of the frame of video A does not need to be reduced, and the resolution after reduction may be other than XB.

[0049] Fig. 12 is a flowchart illustrating the process of acquiring candidate teacher data according to the third embodiment. Below, differences from the process of acquiring candidate teacher data according to the first embodiment (Fig. 6) will be mainly described with reference to the flowchart in Fig. 12.

[0050] In S1201, the candidate acquisition unit 413 selects one frame from the moving image A and reads the frame data of the selected frame. The candidate acquisition unit 413 selects one frame from the beginning of the moving image A recorded in the recording unit 106 (hereinafter, the selected frame will be referred to as frame Ax), and transfers the frame data of frame Ax from the recording unit 106 to the RAM 103 by referring to table PA recorded in the RAM 103.

[0051] In S1202, the candidate acquisition unit 413 calculates the similarity between frame Ax read out in S1201 and each frame of moving image B. More specifically, the candidate acquisition unit 413 references the position information (related to frame data) in table PB and sequentially acquires frame data for each frame of moving image B from the recording unit 106 to RAM 103. The candidate acquisition unit 413 then calculates a similarity index between frame Ax and each frame using a similarity index calculation function (SSIM in this embodiment), and records the similarity index in RAM 103. In S1203, the candidate acquisition unit 413 acquires the frame of moving image B with the highest numerical value among the similarity indexes calculated in S1202 as frame Bx. The subsequent processes of S1204 to S1205 are the same as S604 to S605 in the first embodiment (FIG. 6).

[0052] As described above, according to the third embodiment, appropriate training data candidates can be acquired even if the shooting periods of the two image groups (video A and video B) do not overlap.

[0053] <Fourth embodiment> (Improved learning performance by taking image similarity into account) In the fourth embodiment, we will explain how to improve the performance of the learning model M by taking image similarity into account in the learning process of the first to third embodiments. As described in the first embodiment, appropriate training data is extracted for the selected frame By in S701 of FIG. 7, and the training model M is generated / updated using the training data in S704. When generating / updating this learning model M, network parameters are tuned using backpropagation, as shown in FIG. 8. In the fourth embodiment, the strength of the backpropagation tuning is controlled based on the attributes (e.g., capture time) of the frame (image E) used in the learning and the frame By to be enhanced in resolution and definition, or the image of those frames. More specifically, in the learning process, the learning unit 451 determines coefficients so that the stronger the similarity between each frame of the sequentially input frame group UB and frame By is, the stronger the influence of backpropagation on the network parameter update, and the weaker the similarity is, the weaker the influence is. Here, image similarity may be calculated simply based on the time difference between frame By and input image E, or by comparing the images of both frames using SSIM or the like. For example, when using the former method (method using time difference), the tuning strength can be multiplied by a coefficient of 1 if the time difference is smaller than the threshold, and by a coefficient of 0.5 if the time difference is greater than or equal to the threshold, as shown below.

[0054] if(ABS(time difference between By and E)<threshold){coefficient=1}else {coefficient=0.5} On the other hand, when the latter method (method using similarity) is used, the structural similarity index (SSIM) can be used as a coefficient of tuning strength, as shown below.

[0055] Coefficient = SSIM(By and E) [0 ≤ SSIM(x) ≤ 1] The strength of influence can be determined, for example, by multiplying the update rate of the network parameters in the backpropagation by the above-mentioned coefficient, or by multiplying the number of learning loops for the input image E by a coefficient without multiplying the parameter update rate by a coefficient.

[0056] Fifth Embodiment In the first to third embodiments, a configuration was described in which a pair of a frame from video A and a frame from video B is extracted as teacher data candidates and registered in the candidate database D1. In the fifth embodiment, a video A' is generated by converting video A to the resolution XB of video B, and the candidate acquisition unit 413 acquires teacher data candidates using video A and video A'. That is, the candidate acquisition unit 413 of the fifth embodiment extracts a frame (referred to as Ax') having the same frame number as frame Ax of video A from video A', and registers the pair of frame Ax and frame AX' in the candidate database D1 as teacher data candidates. The fifth embodiment will be described in detail below.

[0057] (Explanation of the configuration of the image processing device 100) The hardware configuration and functional configuration of the image processing device 100 are the same as those of the first embodiment (FIG. 1). However, the control unit 101 of the fifth embodiment also has a resolution conversion function for reducing the resolution of an image using bicubic digitization. When performing resolution reduction processing on image data recorded in the RAM 103, this resolution conversion function calculates the pixel values ​​of pixels that require interpolation by referring to their surrounding pixels.

[0058] (Data recorded in the recording unit 106 and its decryption and decompression methods) In the first embodiment, the moving images a and b stored in the recording unit 106 are converted into an uncompressed format, and the decoded moving image a is recorded in the recording unit 106 as moving image A and the decoded moving image b is recorded in the recording unit 106 as moving image B. In the fifth embodiment, moving image A' is generated by converting moving image A to a resolution XB of moving image B. More specifically, the control unit 101 refers to the table PA recorded in the RAM 103 and sequentially inputs frame data of frames of moving image A (hereinafter, frame K) stored in the recording unit 106 to a resolution conversion function of the control unit 101. The resolution conversion function then outputs a frame of frame data with resolution XB (hereinafter, frame K'). The control unit 101 refers to the table PA and multiplexes the frame data with shooting time information of frame K read from the recording unit 106, and records the multiplexed frame in the recording unit 106 as a frame of moving image A'. In addition, the control unit 101 records in the RAM 103 a table PA' that holds the frame number, position information of the frame data, and position information of the shooting time data of each frame of the moving image A'.

[0059] An example of these videos A, B, and A' is shown in FIG. 13. For each frame image (A1 to An) of video A, images (A1' to An') reduced to a resolution XB are generated and recorded in the recording unit 106 as video A'. In the above example, the resolution of video A is reduced to XB, but this is not limited to this. Video A' may be any image converted to have a resolution lower than that of video A. However, by using an image converted to the same resolution as the image to be refined, a learning model that is more suitable for the image to be refined can be constructed.

[0060] (Processing to obtain candidate training data) FIG. 14 shows the configuration and operation of functional blocks in image processing by the image processing device 100 of the fifth embodiment. The candidate acquisition unit 413 acquires combinations of frames having the same frame number for each frame of moving image A and moving image A', and registers them in the candidate database D1. More specifically, the candidate acquisition unit 413 searches for frames in moving image A' that have the same frame number as each frame of moving image A listed in table PA by referring to table PA'. The candidate acquisition unit 413 assigns a unique index I to combinations of frames from moving image A and moving image A' that have the same frame number, and registers them in the candidate database D1. The frame group of moving image A registered in the candidate database D1 is designated as TA, and the frame group of moving image A' is designated as TA'.

[0061] (High-definition video generation processing) The following mainly describes the differences from the processing of the first embodiment (FIG. 7) with reference to the flowchart of FIG.

[0062] The processing of S1501 is the same as S701 in the first embodiment (FIG. 7). In S1502, the teacher data extraction unit 414 extracts frames from the teacher data candidate frame group TA' registered in the candidate database D1, the frames having a difference in shooting time from frame By that is shorter than a threshold value predetermined by the system. The threshold value can be, for example, the display period of one frame of video A (the display period of one frame at frame rate XA). The teacher data extraction unit 414 registers the extracted frames in the teacher database D2.

[0063] Specifically, the teacher data extraction unit 414 first refers to table PA' to acquire the time information of the frames registered in the frame group TA'. The teacher data extraction unit 414 registers, among the time information of the acquired frame group TA', frames whose time difference with frame By is shorter than a threshold in the teacher database D2 on RAM 103. Hereinafter, the frame group of the moving image A' registered in the teacher database D2 is referred to as frame group UA'. Note that in this embodiment, frames whose difference with the shooting time of frame By is shorter than a predetermined threshold are extracted from the candidate database D1, but this is not limited to this. For example, frames whose index (e.g., SSIM) representing the similarity between the image of each frame of the frame group TA' and the image of frame By is higher than a threshold predetermined for the system may be extracted from the frame group TA' and registered in the teacher database D2.

[0064] In S1503, the teacher data extraction unit 414 registers in D2 the frames of frame group TA that are associated with each frame of frame group UA' by index I. Specifically, the teacher data extraction unit 414 references the candidate database D1 on RAM 103 and registers in the teacher database D2 the frames of frame group TA that are associated with each frame of frame group UA' by index I. At this time, the associated combinations (frame pairs) do not change, and each combination is assigned a unique index J in the teacher database D2. Hereinafter, the frame group of video A registered in the teacher database D2 will be referred to as frame group UA.

[0065] In S1504, the learning unit 451 performs learning using the frame group UA and the frame group UA' by referring to the teacher database D2, and generates a learning model M. Specifically, first, the learning unit 451 refers to the teacher database D2 and tables PA and PA', reads frame data from the recording unit 106, and inputs the frame data to the learning model generation function. The learning unit 451 performs learning using the frame data read by the learning model generation function, and stores the learning model M generated as the learning result in RAM 103. Details of learning the learning model are as described above with reference to FIG. 8. The subsequent processing of S1505 and S1506 is the same as that of the first embodiment (the processing of S705 and S706 in FIG. 7).

[0066] As described above, according to each of the above embodiments, training data used for training a training model is selected based on an image to be refined. Therefore, a training model trained using the selected training data can infer high-frequency components of the image to be refined with higher accuracy, thereby obtaining a highly refined image. In other words, the accuracy of video super-resolution technology for enhancing the resolution of videos is improved.

[0067] In the above embodiment, when acquiring candidate teacher data, the image paired with the image selected from video A is an image selected from video B based on the shooting time or image similarity, or a low-resolution version of the selected image, but this is not limited to this. An image related to an image selected from video A used as candidate teacher data may have a lower resolution than the selected image and be related to the selected image. Whether an image is related to an image selected from video A may be determined based on commonalities such as the temperature at the time of shooting, the shooting location, and the shooting direction, for example.

[0068] Furthermore, in the above embodiment, a two-stage process of generating the candidate database D1 and then the teacher database D2 has been described, but this is not limiting. For example, the teacher data extraction unit 414 may extract frames from video A that can be paired with teacher data based on the frame By, and obtain teacher data by using the extracted frames and frames related to the extracted frames as pairs. However, when sequentially increasing the resolution of multiple images in video B, it is more efficient to create the candidate database D1 as in the above embodiments and then extract and use appropriate teacher data from the candidate database D1 depending on the image to be increased in resolution.

[0069] Sixth Embodiment (Improvement of learning and inference performance by taking image similarity into account) In the sixth embodiment, an improvement in the learning performance and inference performance in the learning process and inference process of the first embodiment will be described, taking into account image similarity.

[0070] In the first embodiment, appropriate training data is extracted for the selected frame By in S701 of FIG. 7 , a learning model M is generated or updated using the training data in S704, and a high-resolution frame Cy is generated by inference using the learning model M in S705. However, with this method, if the frame By contains various textures such as people, buildings, plants, and the sea, the amount of information to be learned at one time may be too large, resulting in a decrease in learning performance. This is because a single frame contains high-frequency components with various patterns. Therefore, the learning process of the sixth embodiment solves the above problem by extracting a portion of a frame, generating a learning model for each local region, performing inference using the learning model for each local region, and generating and combining high-resolution images for each local region.

[0071] In the sixth embodiment, the hardware configuration and functional configuration of the image processing device 100 are the same as those in the first embodiment (FIG. 1). The extracted teacher data may be from any of the first to fifth embodiments. Since the learning process and subsequent steps are different, a specific description will be given using the flowchart in FIG. 16 and an example of the learning and inference process in FIG. 17.

[0072] The processing from S1601 to S1603 is the same as S701 to S703 in the first embodiment (FIG. 7).

[0073] In S1604, the inference unit 452 cuts out the inference target frame By into a local region (determines a local region) and stores the local region in the RAM 103. Hereinafter, the cut-out local region (local image) is referred to as a local region Byn1701.

[0074] Next, in S1605, the learning unit 451 selects local areas UAn1702 and UBn1703 that correspond to the same coordinate position as the local area Byn of the inference target frame By from the teacher data (frame group UA and frame group UB) registered in the teacher database D2 (local area selection), and stores these in RAM 103. In this embodiment, the teacher data has a pair of local areas, but the teacher data may also have multiple local area pairs. Note that this group of local areas is, for example, a rectangular area of ​​equal size, measuring several tens of pixels square, but is not limited to this.

[0075] Note that the "local area corresponding to the same coordinate position" of the local area Byn1701 to be inferred, as mentioned here, is an area indicated by the exact same coordinates as the local area of ​​the inference target frame By, in the case of frame group UB. In other words, if the local area coordinates of the inference target frame By are (sx, sy), then the local area coordinates of UBn1703 are also (sx, sy). On the other hand, in the case of frame group UA, the ratio of the resolution XA of video A to the resolution XB of video B is taken into consideration. For example, if XA:XB has a 2:1 relationship in both width and height, then if the local area coordinates of the inference target frame By are (sx, sy), then the local area coordinates of UAn1702 are (sx*2, sy*2). Hereinafter, the expression "local area corresponding to the same coordinate position" refers to the above.

[0076] In S1606, the learning unit 451 uses the local area UAn1702 and the local area UBn1703 to generate a learning model Mn1704 (local area learning model) using the learning model generation function shown in Fig. 8. The learning unit 451 reads out the frame data of the frame pair registered as teacher data from the recording unit 106, inputs it to the learning model generation function for each local area, and saves the generated learning model Mn1704 in the RAM 103.

[0077] In S1607, the inference unit 452 performs inference on the local region Byn1701 using the learning model Mn1704 generated in S1606, and generates a local region Cyn1705 (local high-frequency component) of a high-resolution frame. First, the inference unit 452 reads out the learning model Mn1704 saved in RAM 103 in S1606. Next, the inference unit 452 inputs the local region Byn1701 saved in RAM 103 in S1604 to the CNN of the learning model Mn1704, and generates "high-frequency components expected when the local region Byn1701 is expanded to the local region UAn1702." The inference unit 452 generates the local region Cyn1705 by adding the generated high-frequency components to "an image obtained by linearly expanding the image of the local region Byn1701 to the local region UAn1702," and records this in RAM 103. The process from the inference of high-frequency components performed for the local region Byn1701 to the generation of a high-definition image is the same as the inference process shown in FIG.

[0078] Next, in S1608, the inference unit 452 combines the local region Cyn1705 of the high-resolution frame Cy1706 recorded in the RAM 103 based on the coordinate position information of the frame to generate the high-resolution frame Cy1706 and stores it in the RAM 103. Note that 1705 indicated by the dashed line in Fig. 17 is the local region Cyn, and 1706 indicated by the solid line is the high-resolution frame Cy.

[0079] In S1609, the control unit 101 determines whether the above processing has been completed for all local regions in frame By. If the control unit 101 determines that the processing has not been completed (NO in S1609), the process proceeds to S1605, and the above processing is repeated for the next local region in frame By. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S1609), the process proceeds to S1610.

[0080] In S1610, the inference unit 452 appends the frame data of the high-resolution frame Cy1706 recorded in the RAM 103 to the end of the high-resolution image C on the recording unit 106. In addition, the shooting time information of By is duplicated, multiplexed as the shooting time of the high-resolution frame Cy1706, and recorded in the moving image C.

[0081] In S1611, the control unit 101 determines whether the above processing has been completed for all frames of moving image B. If the control unit 101 determines that the processing has not been completed (NO in S1611), the process proceeds to S1601, and the above processing is repeated with the next frame of moving image B as By. On the other hand, if the control unit 101 determines that the processing has been completed (YES in S1611), the control unit 101 ends this processing. When the high-definition moving image generation processing is completed as described above, a high-definition moving image C with a resolution of XA and a frame rate of FB is recorded in the recording unit 106 in an uncompressed format.

[0082] As described above, according to the sixth embodiment, for an image to be refined that contains a large amount of information and includes various textures, the amount of information to be learned at one time can be narrowed down by learning for each local region, thereby enabling more accurate learning and making it possible to generate a higher-resolution image.

[0083] Seventh Embodiment (Improving super-resolution performance by using regions with high similarity to the inference target region for learning) In the seventh embodiment, an example will be described in which the learning process for each local region in the sixth embodiment is changed to improve the super-resolution performance.

[0084] In the method of the sixth embodiment, a learning model is generated by using an area in a frame different from the inference target that is at the same position as the inference target area for learning. However, this method has the problem that when the subject moves rapidly, the inference area and the training data differ, which reduces the super-resolution performance.

[0085] Therefore, in the learning process of the seventh embodiment, a similarity evaluation means is provided to search for areas that have a high similarity to the inference area from among the candidate training data, and areas that have a high similarity are used for learning, thereby solving the above problem.

[0086] (High-definition video generation processing) The seventh embodiment differs from the sixth embodiment only in the processing of S1605 in the flowchart of the high-definition moving image generation processing shown in Fig. 16. Therefore, only the processing of S1605 in the seventh embodiment will be described.

[0087] In S1605, the inference unit 452 extracts a partial region of the inference target frame By and stores it in RAM 103 as a local region. Note that this local region is, for example, a rectangular region of uniform size, measuring several tens of pixels square, but is not limited to this. The control unit 101 uses SSIM, which is provided as similarity evaluation means, to search for a region UBn having the highest similarity to the local region of the inference target frame By from the frame group UB of the teacher data registered in the teacher database D2, and stores it in RAM 103. The learning unit 451 selects a frame from the frame group UA that is paired with the frame to which the local region UBn stored in RAM 103 belongs, and from among them, stores a local region UAn that is located in the same relative position as the local region UBn in RAM 103. Note that the similarity evaluation may also be performed using PSNR (Peak Signal to Noise Ratio), SNR (Signal to Noise Ratio), or MSE (Mean Square Error).

[0088] As described above, according to the seventh embodiment, learning is performed using an area that has a high similarity to the inference area, so that it is possible to generate a higher-resolution image even for a video in which the subject moves vigorously.

[0089] Eighth Embodiment (Improving super-resolution performance by using regions with high similarity to the inference target region for learning) In the eighth embodiment, a method for solving another problem in the sixth embodiment that the seventh embodiment aims to solve will be described. The problem in the sixth embodiment is that, because the learning data for super-resolution processing of a local region of the inference target frame By is selected from a local region corresponding to the same coordinate position as the local region, super-resolution performance decreases when the subject moves rapidly, for example.

[0090] In the eighth embodiment, a method of using a motion vector related to an inference region to identify a region with high similarity will be described. However, it is assumed that the moving image b in the eighth embodiment is compressed using inter-frame prediction in the format of ISO / IEC 14496-10 "MPEG-4 Part 10: Advanced Video Coding" (hereinafter abbreviated as MPEG-4 AVC).

[0091] The eighth embodiment will be described below, focusing only on the differences from the sixth embodiment.

[0092] (Data recorded on the recording medium and its decryption and expansion methods) In the processing of the analysis unit 211 in the eighth embodiment, in addition to the processing of parsing video data recorded in the recording unit 106 (described in the first embodiment), the following processing is performed: The analysis unit 211 parses the MP4 file storing the video b, obtains the avcC box, and saves the sequence parameter set (hereinafter abbreviated as SPS) and picture parameter set (hereinafter abbreviated as PPS) included therein in the RAM 103.

[0093] (High-definition video generation processing) The difference between the high-definition moving image generation processing in the eighth embodiment and the sixth embodiment is the processing in S1605 and S1606 in the flowchart of Fig. 16. Therefore, the processing in S1605 and S1606 in the eighth embodiment will be described using the flowchart of Fig. 18.

[0094] In the sixth embodiment, in S1604, the inference unit 452 cuts out the local region of the inference target frame By into a rectangular region of equal size, 16 pixels on each side.

[0095] In S1801, if the inference target frame By is an I-picture, the control unit 101 advances the process to S1803. If the inference target frame By is a P-picture or a B-picture, the control unit 101 advances the process to S1802.

[0096] In S1802, the control unit 101 acquires a macroblock layer from a local region of the inference target frame By. Furthermore, if a sub-macroblock is used, a sub-macroblock prediction is acquired. Otherwise, a macroblock prediction is acquired.

[0097] The control unit 101 derives a prediction unit block area Bynb in a macroblock to which a local region of the inference target frame By belongs, from the sub-macroblock prediction or macroblock prediction of the macroblock. The prediction unit block area Bynb is any one of the macroblock, each block in a partitioned macroblock, each block in a sub-macroblock, or each block in a partitioned sub-macroblock.

[0098] The control unit 101 derives the motion vector, reference frame, mbPartIdx, and subMbPardIdx of the block area Bynb from the SPS, PPS, macroblock prediction, or sub-macroblock prediction.

[0099] Here, the control unit 101 generates six pieces of information for each block area Bynb: "mbPartIdx," "subMbPardIdx," "presence or absence of motion vector," "motion vector," "reference / referenced frame," and "reference direction," and stores these in RAM 103. This information identifies which block area Bynb in the macroblock in question. The reference direction is the direction pointed by the motion vector from the macroblock in the local area of ​​the inference target frame By, and the referenced direction is the direction pointed by the macroblock in another frame in the local area of ​​the inference target frame By. Hereinafter, this collection of six pieces of information will be referred to as motion vector information.

[0100] The control unit 101 checks whether the reference / referenced frames in the generated motion vector information exist in the training data candidates, and if not, updates the motion vector information to indicate whether or not there is a motion vector.

[0101] If the frame to be inferred is a B picture and the block has two motion vectors, the referenced frame that is closer in time to the frame to be inferred is used. If the difference in temporal distance from the frame to be inferred is the same, the motion vector and referenced frame information that are closer in spatial distance indicated by the motion vector are used. If these are also the same, either can be used.

[0102] In S1803, for a block area Bynb whose motion vector information indicates "absence" of a motion vector, the control unit 101 searches for a block that references the block area Bynb from among the training data candidates. Note that the method for obtaining the motion vector and reference frame information required to determine whether a block references the block area Bynb has been described in S1802 and is therefore omitted here.

[0103] If a block that references block area Bynb is found, the motion vector information for that block area Bynb is updated. The frame range to be searched is within three frames before and after the frame containing block area Bynb. The macroblock range to be searched is within MaxVmvR for each level defined by MPEG-4 AVC. MaxVmvR is derived from the SPS of video b. The frame range and macroblock range to be searched are not limited to this example.

[0104] The inference unit 452 acquires, from the frame set UB, a reference-target or reference-source local area UBXnb for each block area Bynb for which the motion vector information indicates a motion vector, and stores the local area UBXnb in the RAM 103. The inference unit 452 acquires, from the frame set UA, a local area UAXnb that corresponds to the same coordinate position as the local area UBXnb acquired from the motion vector information for each block area Bynb recorded in the RAM 103, and stores the local area UAXnb in the RAM 103.

[0105] In S1804, the control unit 101 determines whether or not there is a motion vector in the motion vector information for all block regions Bynb, and if there is, executes S1805. If there is no motion vector, executes S1807.

[0106] In S1805, the inference unit 452 combines the local regions UBXnb recorded in the RAM 103 based on the coordinate position information of each block region Bynb to generate a combined local region UBXn, and stores the combined local region UBXn in the RAM 103.

[0107] The inference unit 452 combines the local area UAXnb corresponding to the same coordinate position of the local area UBXnb recorded in the RAM 103 based on the coordinate position information of each block area Bynb to generate a combined local area UAXn and store it in the RAM 103.

[0108] The learning unit 451 uses the combined local area UAXn and the combined local area UBXn to generate a learning model Mn using the learning model generation function shown in Fig. 8. The combined local area UBXn is training data corresponding to the same coordinate position as the combined local area UAXn of the paired frame. The learning unit 451 reads out this training data from the RAM 103, inputs it to the learning model generation function, and stores the generated learning model Mn in the RAM 103.

[0109] In S1806, the inference unit 452 performs inference on the local region Byn of the frame By using the learning model Mn generated in S1805, and generates the local region Cyn1705 of the high-resolution frame.

[0110] First, the inference unit 452 reads out the learning model Mn stored in RAM 103 in S1805. Next, the inference unit 452 inputs the local region Byn of the frame By stored in RAM 103 into the CNN of the learning model Mn, and generates "high-frequency components expected in the local region Byn when the inference target frame By is enlarged to resolution XA." The inference unit 452 generates a local region Cyn1705 by adding the generated high-frequency components to "a local region Byn linearly enlarged based on the ratio between resolution XB and resolution XA," and records this in RAM 103. Note that the processing from inferring high-frequency components performed on the local region Byn to generating a high-resolution image is the same as the inference process shown in FIG. 8.

[0111] In S1807, the control unit 101 determines whether the presence or absence of a motion vector in the motion vector information is "present" for each block region Bynb included in the local region Byn, and if "present", executes S1808. If not, executes S1811.

[0112] In S1808, the learning unit 451 generates a learning model Mnb using the block region Bynb and the local region UBXnb, using the learning model generation function shown in FIG.

[0113] In S1808, the inference unit 452 uses the local regions UBXnb and UAXnb recorded in RAM 103 to generate a learning model Mnb for inferring the block region Bynb using the learning model generation function shown in FIG. 8. The combined local region UBXnb is training data corresponding to the same coordinate position as the combined local region UAXnb in the paired frame. The learning unit 451 reads this training data from RAM 103 and inputs it to the learning model generation function, and stores the generated learning model Mnb in RAM 103.

[0114] In S1809, the inference unit 452 performs inference on the block region Bynb using the learning model Mnb to generate a local region Cynb of a high-resolution frame. First, the inference unit 452 reads the learning model Mnb stored in RAM 103 in S1808. Next, the inference unit 452 inputs the block region Bynb stored in RAM 103 to the CNN of the learning model Mnb to generate "high-frequency components expected in the local region Bynb when the inference target frame By is enlarged to resolution XA." The inference unit 452 generates a local region Cynb by adding the generated high-frequency components to "a local region Bynb linearly enlarged based on the ratio between resolution XB and resolution XA," and records this in RAM 103. Note that the processing from inferring high-frequency components performed on the block region Bynb to generating a high-resolution image is the same as the inference process shown in FIG. 8.

[0115] In S1810, the control unit 101 linearly enlarges a block area Bynb, for which the presence or absence of a motion vector in the motion vector information is "absent," based on the ratio between the resolutions XA and XB, to obtain a local area Cynb of the high-definition frame Cy1706, and stores the result in the RAM 103. Note that the method of enlargement is not limited to linear enlargement, as long as it can be performed based on the ratio between the resolutions XA and XB.

[0116] In S1811, the control unit 101 determines whether the above processing has been completed for all block areas Bynb. If the control unit 101 determines that the processing has not been completed, the control unit 101 proceeds to S1807, where it performs processing on the incomplete block areas Bynb. If the control unit 101 determines that the processing has been completed, it reads out the local areas Cynb stored in the RAM 103 in S1809 and S1810, combines them based on the coordinate position information of the corresponding block areas Bynb, generates local area Cyn1705, and stores it in the RAM 103.

[0117] As described above, according to the eighth embodiment, learning is performed using motion vectors that refer to / are referenced by areas with high similarity to the inference area, so it is possible to generate higher-resolution images even for videos in which the subject moves rapidly.

[0118] Ninth Embodiment (Improving super-resolution performance by using multiple training data weighted by similarity) In the ninth embodiment, a method for solving another problem in the sixth embodiment that the seventh and eighth embodiments aim to solve will be described. The problem in the sixth embodiment is that, because the learning data for super-resolution processing of a local region of the inference target frame By is selected from a local region corresponding to the same coordinate position as the local region, super-resolution performance decreases when the subject moves rapidly, for example.

[0119] The ninth embodiment will be described below, focusing only on the differences from the sixth embodiment.

[0120] (High-definition video generation processing) The ninth embodiment differs from the sixth embodiment only in the processes of S1605 and S1606 in the flowchart of the high-definition moving image generation process shown in Fig. 16. Therefore, only the processes of S1605 and S1606 in the ninth embodiment will be described.

[0121] In S1605, the control unit 101 selects local areas (UAn5 and UBn5) corresponding to the same coordinate position as the local area Byn in the inference target frame By, and stores them in the RAM 103. In addition, eight areas adjacent to UBn5 and of the same size as UBn5 are stored in the RAM 103. Similarly, eight areas adjacent to UAn5 and of the same size as UAn5 are saved in the RAM 103. An example of area selection for frames included in the frame group UB is shown in FIG. 19. Note that in this embodiment, the area at the same position coordinate as the local area Byn and its eight adjacent areas are selected for the inference target area, but the means and number of area selections are not limited to this.

[0122] Next, the control unit 101 uses SSIM provided as similarity evaluation means to perform a similarity evaluation with the local region Byn of the inference target frame By for each of UBn1 to UBn9, obtains a similarity evaluation value, generates learning count information for each of UBn1 to UBn9 (determines the number of times), and stores this in RAM 103. The learning count information includes "information specifying one of UBn1 to UBn9," "similarity evaluation value with the local region Byn," and "number of times of learning." If the similarity evaluation value with the local region Byn in the learning count information is less than a threshold predetermined in the system, the control unit 101 updates the number of times of learning in the learning information to 0. For areas where the similarity evaluation value is equal to or greater than the threshold, the control unit 101 determines the number of times of learning based on the ratio of the similarity evaluation values ​​between areas where the similarity evaluation value is equal to or greater than the threshold, and updates the learning information. For example, if the similarity evaluation values ​​of UBn4, UBn5, and UBn6 are equal to or greater than the threshold, the ratio is 2:5:3, and the total number of learning times is specified as 1000, the number of learning times in the learning information for UBn4 to UBn6 will be 200, 500, and 300, respectively. Note that in the method for determining the number of learning times in this embodiment, the number of learning times is linearly allocated to areas having similarity evaluation values ​​exceeding the threshold, but the method is not limited to this.

[0123] In S1606, the learning unit 451 uses the local area (any of UBn1 to UBn9) indicated by the learning information and any of the local areas (UAn1 to UBn9) in the frame group UA corresponding to the local area (UBn1 to UBn9) as training data, and performs learning using the learning model generation function shown in FIG. 8 for the number of learning times indicated by the learning information to generate a learning model Mn. The generated learning model Mn is stored in the RAM 103.

[0124] The processing from S1607 onwards is the same as in the sixth embodiment, and therefore a description thereof will be omitted.

[0125] As described above, according to the ninth embodiment, multiple areas that have a high similarity to the inference area are used for learning according to their similarity to the inference area, so that it is possible to generate higher-resolution images even for videos in which the subject moves vigorously.

[0126] As described above, according to the sixth to ninth embodiments, by determining a local region from an image to be refined and narrowing down the amount of information used for training the learning model, and further selecting a local region of the training data that is highly correlated with the local region determined from the image to be refined and using it for training the learning model, it is possible to more accurately infer the high-frequency components of the image to be refined and obtain a highly accurate high-resolution image. In other words, the accuracy of video super-resolution technology that refines videos is improved.

[0127] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0128] The present invention is not limited to the above-described embodiments, and various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to apprise the public of the scope of the present invention. [Explanation of symbols]

[0129] 101: control unit, 102: ROM, 103: RAM, 104: decoding unit, 105: learning and inference unit, 106: recording unit, 107: bus, 211: analysis unit, 212: decoded video generation unit, 413: candidate acquisition unit, 414: teacher data extraction unit, 451: learning unit, 452: inference unit

Claims

1. 1. An image processing device that uses a first image group to increase the resolution of images of a second image group having fewer high-frequency components in corresponding frames than the first image group, an acquisition means for acquiring, as candidates for training data, a plurality of pairs of a first image selected from the first image group and a third image having fewer high frequency components than the first image and having a shooting time that differs from the shooting time of the first image by less than a predetermined threshold; a selection means for selecting a pair of training data to be used for learning from the plurality of pairs of training data candidates based on an image to be refined selected from the second image group; a learning model generation means for generating a learning model using the pair of training data selected by the selection means; an inference means for inferring high frequency components of the image to be refined using the generated learning model; an image generating means for generating a high-resolution image based on the image to be refined and the high-frequency components inferred by the inference means; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the acquisition means acquires, as the third image, an image that has the greatest similarity to the first image from the second image group.

3. 3. The image processing apparatus according to claim 2, wherein the acquisition means determines a similarity between an image obtained by reducing the first image to a resolution of the second image group and an image of the second image group.

4. 2. The image processing apparatus according to claim 1, wherein the acquiring means acquires, as the third image, an image obtained by reducing the resolution of the first image by reducing the size of the first image.

5. 5. The image processing apparatus according to claim 4, wherein the third image is an image obtained by reducing the resolution of the first image to that of the second image group.

6. The image processing device according to any one of claims 1 to 5, characterized in that the selection means selects candidate teacher data including images whose shooting time difference from the shooting time of the image to be enhanced in resolution is smaller than a predetermined threshold as teacher data to be used for the learning.

7. The image processing device according to any one of claims 1 to 5, characterized in that the selection means selects candidate training data including an image whose similarity to the image to be refined is greater than a predetermined threshold as training data to be used for the learning.

8. 8. The image processing device according to claim 1, wherein the inference means controls updating of parameters by error backpropagation in the learning based on training data used in the learning and the image to be refined.

9. 9. The image processing device according to claim 8, wherein the inference means determines coefficients based on the training data used in the learning and the image to be refined, and controls the amount of parameter update by the error backpropagation based on the coefficients.

10. 9. The image processing device according to claim 8, wherein the inference means determines coefficients based on training data used in the learning and the image to be refined, and controls the number of iterations of updating the parameters by the error backpropagation based on the coefficients.

11. 11. The image processing apparatus according to claim 9, wherein the inference means determines the coefficient based on a difference between the photographing time of an image of the training data used in the learning and the photographing time of the image to be subjected to high definition enhancement.

12. 11. The image processing apparatus according to claim 9, wherein the inference means determines the coefficients based on a similarity between an image of training data used in the learning and the image to be refined.

13. The method further includes a local region determination means for cutting out one or more local regions to be inferred from the image to be refined, and a local region selection means for selecting a pair of local regions to be used for learning from a pair of teacher data obtained by the selection means, based on the local image cut out by the local region determination means. the learning model generation means generates a local region learning model using the pair of teacher data obtained by the local region selection means; the inference means infers local high-frequency components of the extracted image to be refined using the local region learning model; 13. The image processing device according to claim 1, wherein the image generating means generates a high-definition image of the local region using the local high-frequency component and a local image of the image to be high-definition-enhanced, and combines them.

14. 14. The image processing device according to claim 13, wherein the local region selection means selects a local region corresponding to the same coordinate position as the local region cut out by the local region determination means, in the pair of training data obtained by the selection means.

15. 15. The image processing apparatus according to claim 14, wherein the image generating means generates a high-definition image by combining the high-definition images of the local regions based on information about coordinate positions.

16. 14. The image processing device according to claim 13, wherein the local region selection means further comprises evaluation means for evaluating a degree of similarity of an image, and using the evaluation means, selects a local region having a high degree of similarity to the local region determined by the local region determination means from an image included in the pair of training data obtained by the selection means.

17. 14. The image processing device according to claim 13, wherein the local region selection means selects a local region using a motion vector of the inference target region determined by the local region determination means, or a motion vector referencing the inference target region, or both.

18. 14. The image processing device according to claim 13, wherein the learning model generation means further comprises a number of times determination means for determining the number of times of learning for each of the teacher data, and the learning model generation means determines the number of times of learning for each of the plurality of pairs of local regions for learning obtained by the local region selection means using the number of times determination means, and performs learning the determined number of times.

19. 19. The image processing device according to claim 18, wherein the number of times determination means further comprises evaluation means for evaluating a similarity of an image, and using the evaluation means, obtains a similarity between the inference target region determined by the local region determination means and each of the images included in the plurality of pairs of local regions for learning obtained by the local region selection means, and determines the number of times of learning based on the similarity.

20. 20. The image processing device according to claim 1, wherein the first image group and the second image group are two image groups obtained by performing different image processing on a single image captured by a single image sensor included in a single imaging device.

21. 20. The image processing apparatus according to claim 1, wherein the first image group and the second image group are image groups captured by two image sensors, respectively.

22. 22. The image processing apparatus according to claim 1, wherein a frame rate of the first group of images is lower than a frame rate of the second group of images.

23. 1. An image processing method for increasing the resolution of images of a second image group, which has fewer high-frequency components in corresponding frames than the first image group, using a first image group, comprising: an acquisition step of acquiring, as candidates for training data, a plurality of pairs of a first image selected from the first image group and a third image having fewer high frequency components than the first image and having a shooting time that differs from the shooting time of the first image by less than a predetermined threshold; a selection step of selecting a pair of training data to be used for learning from the plurality of pairs of training data candidates based on an image to be refined selected from the second image group; a learning model generation step of generating a learning model using the pair of training data selected in the selection step; an inference step of inferring high-frequency components of the image to be refined using the generated learning model; an image generation step of generating a high-resolution image based on the image to be refined and the high-frequency components inferred in the inference step; An image processing method comprising:

24. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 22.

Images