Image processing device, control method and program

Abstract

To provide an image processing device, a control method and a program that can perform super-resolution processing on an inference-objective image using a learned model with certain inference precision without lowering the processing efficiency of learning processing.SOLUTION: An image processing device 100 is configured to: acquire a moving image A of frame rate FA and resolution XA and a moving image B of frame rate FB (>FA) and resolution XB (<XA) generated by capturing one subject simultaneously; gather an image group UA from the moving image A based upon an image By each time the image By as an object image to be increased in definition is acquired from the moving image B when a learned model is used to increase the resolution of the moving image B up to the resolution XA, and select, as the learned model, one of a learned model M generated through learning based upon the image group UA as teacher data and a learned model MG generated in advance through learning based upon a general teacher image group K as teacher data according to a propriety determination result as teacher data of the image group UA.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, a control method, and a program for improving the resolution of a group of images by utilizing machine learning. [Background technology]

[0002] Super-resolution technology using machine learning is a technology that generates high-definition images by inferring high-frequency components that cannot be compensated for by linear interpolation of pixel values ​​when enlarging an image and performing resolution conversion. In super-resolution technology, a learning process is first performed on a learning model using an image group G and a degraded image obtained by degrading each image of the image group G by an arbitrary method as training data. Specifically, the learning model is trained based on the difference in pixel values ​​between the original image and the degraded image, and a trained model is generated by updating the super-resolution processing parameters held by the learning model. When an image H lacking high-frequency components is input to the trained model thus generated, the high-frequency components are acquired by inference using the trained model. A high-definition image can be generated by superimposing the high-frequency components acquired by inference on the image H.

[0003] Generally, when a product or service using a trained model is provided, the developer collects training data and generates a trained model, and the generated trained model is provided to the user. Therefore, the content of the image input by the user is unknown at the time of the training process. Therefore, the developer prepares a large number of diverse images with no bias in image patterns as training data so that inference can be performed with uniform accuracy for all inference target images, and generates a trained model by repeatedly performing training processes.

[0004] However, when a wide variety of images without bias in image patterns are used as training data, only a small amount of training data has a high similarity to the inference target image specified by the user. When a trained model is generated using such training data, the results of training images with low similarity to the inference target image are reflected in the inference process. For this reason, the inference result is limited to an improvement in resolution due to the emphasis on the edges of the subject, and it is difficult to accurately infer high-frequency components such as fine patterns of the subject, and the inference accuracy cannot be said to be high.

[0005] To solve these problems, a device has been proposed that collects images suitable for learning for each image to be inferred, performs learning, and generates a trained model.

[0006] For example, Patent Document 1 describes an apparatus that performs learning using each frame of a high-resolution, low-frame-rate video A and a low-resolution, high-frame-rate video B recorded of the same scene, and executes super-resolution processing on video B using the generated trained model. In this apparatus, the training data is limited to frames of the same video as the frame to be inferred, and the image similarity is high, so that the trained model can be expected to have high accuracy. As a result, by inputting video B into the trained model, video C having the resolution of video A and the frame rate of video B can be generated with high accuracy.

[0007] However, in a device such as that described in Patent Document 1 that "collects a teacher image for each image to be inferred, executes learning, and generates a trained model," there is a possibility that the collected teacher image is not suitable for learning, or that the collection of the teacher image fails. Taking Patent Document 1 as an example, when a frame of a video at the moment a flash goes off is the subject of inference, it is considered that the similarity between the frames before and after the flash is low, and it is difficult to collect an image (frame) suitable as a teacher image from videos A and B. In such a case, the result of learning an image with low similarity to the image to be inferred is reflected in the inference process, resulting in a problem that the accuracy of inference is significantly reduced.

[0008] On the other hand, if the collected teacher images are not suitable for learning or if collection of teacher images fails, it is possible to prevent a significant decrease in inference accuracy by using a pre-prepared trained model.

[0009] Patent document 2 describes an apparatus that compares shooting environment information of images used in the learning of multiple pre-prepared trained models with shooting environment information of an image to be inferred, and determines which trained model to use based on the degree of similarity.

[0010] In this way, the device of Patent Document 2 is able to switch between appropriate trained models depending on the image to be inferred. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Patent Publication No. 2023-57860 [Patent Document 2] JP 2019-87229 A Summary of the Invention [Problem to be solved by the invention]

[0012] However, it is difficult to improve the above problem by combining Patent Documents 1 and 2, comparing a "prepared trained model" with a "trained model generated by training using teacher images collected after the inference target is determined", and using a trained model that is expected to have high inference accuracy. This is because the device of Patent Document 2 uses the comparison results of shooting environment information associated with each trained model to estimate the inference accuracy, and the estimation of the inference accuracy is insufficient.

[0013] In addition, the apparatus disclosed in Patent Document 2 is configured on the premise of determining a learned model to be used by comparing after the learning of each learning model. Therefore, when the apparatus of Patent Document 2 is used in combination with an apparatus that "collects and learns teacher images after determining an inference target" as in Patent Document 1, it is necessary to execute the learning process of a large number of learning models that are not used, resulting in poor processing efficiency.

[0014] Therefore, an object of the present invention is to provide an image processing apparatus, a control method, and a program that can perform super-resolution processing of an inference target image using a learned model with a certain inference accuracy without reducing the processing efficiency of the learning process.

Means for Solving the Problems

[0015] In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus that super-resolves a first image group having a frame rate FA (<FB) to a resolution XA (>XB) by inferring a second image group having a frame rate FB and a resolution XB among first and second image groups that are each a group of frames of two videos generated by simultaneously photographing the same subject at different resolutions and frame rates using a learned model, the apparatus comprising: a learning means for learning a learning model using a teacher image group having the resolution XA to generate the learned model; a collecting means for collecting an image group from the first image group based on the super-resolution target image every time one super-resolution target image is obtained from the second image group; a first selection means for generating a first learned model by the learning means using the collected image group as the teacher image group and selecting the generated first learned model as the learned model when it is determined that the collected image group is suitable as the teacher image group; and a second selection means for selecting a second learned model previously generated by the learning means using an image group collected in advance as the teacher image group as the learned model when it is determined that the collected image group is not suitable as the teacher image group.

Effects of the Invention

[0016] According to the present invention, super-resolution processing of an image to be inferred using a trained model can be performed with a certain level of inference accuracy without reducing the processing efficiency of the learning process. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a hardware configuration of an image processing apparatus according to a first embodiment. [Diagram 2] 2 is a diagram illustrating functional blocks of a process of expanding compressed video data by a control unit in FIG. 1. FIG. [Diagram 3] 2 is a diagram for explaining functional blocks of a teacher data candidate acquisition process by a control unit and functional blocks of a high-definition video generation process by a learning and inference unit in FIG. 1. FIG. [Figure 4] 3 is a diagram showing an example of a frame configuration of an input moving image according to the first embodiment; FIG. [Diagram 5] 1 is a flowchart of a teacher data candidate acquisition process according to the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a data configuration of a candidate database according to the first embodiment; [Figure 7] 4 is a flowchart of a high definition video generation process according to the first embodiment. [Figure 8] FIG. 4 is a diagram illustrating a trained model generation function of the training unit in FIG. 3. [Figure 9] 10 is a flowchart of a high definition video generation process according to a second embodiment. [Figure 10] 13 is a flowchart of a high definition moving image generating process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, each embodiment of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in each embodiment, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0019] It should be noted that hereinafter, the learning model means a machine learning model that is prepared in advance and is unlearned or in the process of being learned, and the learned model means a machine learning model after learning.

[0020] <First Embodiment> (Overview of the Image Processing Apparatus) The image processing apparatus 100 according to the first embodiment uses, as input videos, two videos (videos A and B in FIG. 4 described later) that are simultaneously captured by the same imaging device (not shown). The relationship between the resolution XA / frame rate FA of video A and the resolution XB / frame rate FB of video B is "XA > XB and FA < FB". The image processing apparatus 100 has a function (high-definition video generation function) of generating a learned model by learning using the frames of videos A and B, and generating a video C with a resolution of XA / frame rate FB from video B by inference using this learned model. Also, as will be described in detail later, the resolution of various types of teacher data (teacher image group) used for this learning is resolution XA.

[0021] (Description of the Configuration of the Image Processing Apparatus) FIG. 1 is a block diagram showing the hardware configuration of the image processing apparatus 100.

[0022] In FIG. 1, the image processing apparatus 100 includes a control unit 101, a ROM 102, a RAM 103, a decoding unit 104, a learning and inference unit 105, a recording unit 106, and a bus 107.

[0023] The control unit 101 is a calculation device such as a CPU, and realizes various functions by expanding and executing a program stored in the ROM 102 in a working area of ​​the RAM 103. The control unit 101 can function as each of the functional blocks of an analysis unit 201 and a decoded moving image generation unit 202, which will be described later in Fig. 2, a candidate acquisition unit 301, and a teacher data extraction unit 302, which will be described later in Fig. 3, for example.

[0024] The ROM 102 stores a control program executed by the control unit 101. The RAM 103 is used as a work memory for the control unit 101 to execute the program, a temporary storage area for various data, and the like.

[0025] The decoding unit 104 decodes moving image or image data compressed in an encoding format defined by the Moving Picture Experts Group (hereinafter abbreviated as MPEG) into uncompressed data.

[0026] The learning and inference unit 105 has a functional block (learning unit 303 described later in FIG. 3) that generates / updates a learned model by machine learning a learning model using teacher data. The learning and inference unit 105 also has a functional block (inference unit 304 described later in FIG. 3) that analyzes an input image using the learned model and infers high-frequency components to generate a high-definition image of the input image. In this embodiment, a CNN model for super-resolution processing based on a convolutional neural network (hereinafter abbreviated as CNN) is used as the learning model. This model includes enlargement of an input image by linear interpolation, generation of high-frequency components to be added to the enlarged image, and additive synthesis of the two.

[0027] The recording unit 106 is composed of a recording medium, such as a hard disk drive (HDD) or a memory card, which is detachably connected to the image processing device 100, and a recording medium control device that controls the recording medium. The recording medium control device, in accordance with commands from the control unit 101, controls initialization of the recording medium, data transfer between the recording medium and the RAM 103 for reading and writing data, and the like.

[0028] The bus 107 is an information communication path that connects the various functions of the image processing device 100. The control unit 101, the ROM 102, the RAM 103, the decoding unit 104, the learning and inference unit 105, and the recording unit 106 are connected via the bus 107 so as to be able to communicate with each other.

[0029] It should be noted that the hardware blocks described in this embodiment and the functional blocks executed therein do not necessarily have to be configured as described above. For example, two or more of the control unit 101, the decoding unit 104, and the learning and inference unit 105 may be realized by one piece of hardware. Furthermore, the function of one functional block or the functions of multiple functional blocks may be executed by the cooperative operation of several pieces of hardware. Furthermore, each functional block may be realized by a CPU executing a computer program deployed on a memory, or may be realized by dedicated hardware. Furthermore, a configuration may be adopted in which some of the functional blocks exist on a cloud server and transfer data of the processing result by communication.

[0030] (Video data recorded on a recording medium and its decryption and extraction method) FIG. 2 is a diagram illustrating functional blocks that perform decompression processing of compressed video data by the control unit 101 (analysis unit 201 and decoded video generation unit 202).

[0031] The recording unit 106 stores moving images a and b obtained by compressing moving images A and B used in high-definition moving image generation processing. Here, a moving image refers to one or more pieces of image data that are continuous in time. The moving images a and b in this embodiment are a group of images simultaneously captured by an imaging device (not shown) having an image sensor and compressed by the MPEG format. The moving images a and b may be generated by applying thinning and reduction processing to images captured by a single image sensor, or may be generated by capturing the same subject with image sensors having different resolutions and frame rates. In the following, the moving images a and b are two image groups obtained by performing different image processing on one image captured by one image sensor of one imaging device. The moving image data of each of the moving images a and b is compressed by the MPEG format, multiplexed with information on the shooting time (hereinafter simply referred to as time information), and stored as a file in MP4 format. Note that a format other than the above may be used as long as the image data and the corresponding time information can be acquired as a pair from the recording unit 106.

[0032] The analysis unit 201 has a function of parsing the video data of the videos a and b recorded in the recording unit 106 (in this example, MP4 format files) and calculating the storage positions within each file of the included compressed image data and the time information registered as metadata. In the MP4 format, position information indicating the recording position within the file of each frame data and time information is recorded in the Moov portion. The analysis unit 201 expands the Moov portion of the video a from the recording unit 106 into the RAM 103 and parses it, generating a table Pa having the frame numbers in the video a, the position information of the frame data, and the position information of the shooting time. The analysis unit 201 also parses the Moov portion of the video b in the same manner, generating a table Pb having the frame numbers in the video b, the position information of the frame data, and the position information of the shooting time. The tables Pa and Pb are held in the RAM 103.

[0033] In order to use the moving images a and b in the high definition moving image generating process, it is necessary to perform a process of converting the moving images a and b into uncompressed formats. As shown in FIG. 2, the decoded moving image generating unit 202 of the control unit 101 decodes the moving images a and b to generate moving images A and B, and records them in the recording unit 106. More specifically, the decoded moving image generating unit 202 refers to the table Pa and table Pb held in the RAM 103, and sequentially inputs the frame data of the moving images a and b stored in the recording unit 106 to the decoding unit 104. The decoded moving image generating unit 202 multiplexes the uncompressed frame data output by the decoding unit 104 with time information acquired by referring to the table Pa and table Pb, and records the multiplexed data in the recording unit 106. In this way, the moving image A is obtained by decoding the moving image a, and the moving image B is obtained by decoding the moving image b. The decoded moving image generating unit 202 also generates a table PA having frame numbers in the moving image A, position information of the frame data, and position information of the shooting time, and records the table PA in the RAM 103. Similarly, the decoded moving image generator 202 generates a table PB having frame numbers in moving image B, position information of frame data, and position information of shooting time, and records the table PB in the RAM 103 .

[0034] An example of the frame configuration of video A and video B, which are input video according to the first embodiment, is shown in Fig. 4. In Fig. 4, the total number of frames of video A is n, and the total number of frames of video B is m. Frame pairs indicated by dashed lines (A1 and B2, A2 and B5, A3 and B8, etc.) are pairs of frames having the same time information or the closest time information, and indicate that the images of these paired frames can be considered to have been captured at the same timing.

[0035] Next, a process for generating a high-definition image according to the present embodiment will be described. This process is roughly divided into three parts: a general-purpose trained model generation process, a training data candidate acquisition process, and a high-definition video generation process.

[0036] FIG. 3 is a diagram for explaining functional blocks of the teacher data candidate acquisition process by the control unit 101 and functional blocks of the high definition moving image generation process by the learning and inference unit 105 in FIG.

[0037] As described in FIG. 2, the video A and video B are stored in the recording unit 106, and the table PA and table PB are stored in the RAM 103. The "general-purpose trained model generation process" is performed by the learning unit 303, the "teacher data candidate acquisition process" is performed by the candidate acquisition unit 301, and the "high-definition video generation process" is performed by the teacher data extraction unit 302, the learning unit 303, and the inference unit 304. The learning unit 303 of the learning and inference unit generates a trained model MG (second trained model), which is a general-purpose trained model, using the general-purpose teacher image group K recorded in the recording unit 106. The candidate acquisition unit 301 extracts a pair of frames that are candidates for teacher data from the frame group (first image group) of the video A and the frame group (second image group) of the video B as teacher data candidates, and generates a teacher data candidate DB. The frame By, which is an image to be refined, is acquired from the frame group of the image B. The teacher data extraction unit 302 extracts teacher data further suitable for learning from the teacher data candidates registered in the teacher data candidate DB to generate a trained model suitable for inferring high frequency components of the frame By, and generates the teacher data DB. The learning unit 303 of the learning and inference unit 105 uses the teacher data DB to generate a trained model M (first trained model) for the frame By. The inference unit 304 inputs the frame By to be high-resolution-enhanced to the trained model M generated by the learning unit 303, and performs high-resolution processing of the frame By. The "general-purpose trained model generation process", "teacher data candidate acquisition process", and "high-resolution video generation process" will be described in more detail below.

[0038] (General-purpose trained model generation process) In the general-purpose trained model generation process, the learning unit 303 of the learning and inference unit 105 generates a trained model MG. A group of images collected without bias in image patterns is pre-recorded in the recording unit 106. This group of images is called a general-purpose teacher image group K (not shown).

[0039] 8 is a diagram showing a model generation function of the learning unit 303 (learning means). The model generation function has a learning process and an inference process, and the inference process can be further classified into a feature extraction process using a filter including a CNN and a reconstruction process.

[0040] First, in the feature extraction process, the learning unit 303 randomly extracts one image (assuming it is image K1) from the image group K, and reduces it using a bicubic method or the like to generate a reduced image K1s. The generated K1s is input to a CNN, and convolution feature extraction is performed to generate a large number of feature maps using the CNN.

[0041] Next, in the reconstruction step, the learning unit 303 performs a transposed convolution reconstruction in which all feature maps are upsampled by deconvolution to generate predicted high-frequency components. In the reconstruction step, the learning unit 303 further performs image reconstruction by adding an enlarged image K1c obtained by enlarging the image K1s by a bicubic method or the like to the predicted high-frequency components, thereby generating a predicted high-definition image K1'. In the learning step, the learning unit 303 compares the predicted high-definition image K1' generated in the above-mentioned inference step with the image K1, and performs minute tuning of the trained model MG by an error backpropagation method using the difference between them. The learning unit 303 repeats the training of the CNN using images randomly extracted from the image group K a predetermined number of times as described above, thereby constructing a trained model MG that can demonstrate a certain level of inference accuracy for images of various patterns. The learning unit 303 stores the trained model MG generated by the trained model generation function in the RAM 103.

[0042] (Processing to obtain candidate teacher data) In the teacher data candidate acquisition process, the control unit 101 (candidate acquisition unit 301) generates a teacher data candidate DB. In the first embodiment, the candidate acquisition unit 301 acquires pairs of video A frames and video B frames having the same shooting time from among the frames of video A and video B as teacher data candidates. Specifically, all frame pairs having a common shooting time in video A and video B (frame pairs linked by dashed lines in FIG. 4) are acquired as teacher data candidates. Before the learning process described below is performed, the candidate acquisition unit 301 investigates which frames can be used as teacher data, constructs a teacher data candidate DB, and registers the investigation results.

[0043] FIG. 6 shows an example of the data configuration of the teacher data candidate DB. In the teacher data candidate DB, frame numbers in each video file of a frame group TA that can be used as teacher data among the frame groups of video A and a frame group TB that can be used as teacher data among the frame groups of video B are registered. Here, pairs of frames (pairs of frame numbers) that have the same shooting time are associated and registered by a unique index I in the teacher data candidate DB. For example, in the video A and video B shown in FIG. 4, the frame pairs A1 and B2, A2 and B5, and A3 and B8 (omitted below) are combinations of frames that were shot at the same time. The teacher data candidate DB shown in FIG. 6 shows how these pairs are recorded by frame numbers and given a unique index I. In this way, the acquired teacher data candidates are managed by the teacher data candidate DB.

[0044] The above-described processing for acquiring candidate teacher data will be further described in detail with reference to the flowchart of FIG.

[0045] In step S501, the candidate acquisition unit 301 selects one frame of the video A and acquires time information corresponding to the selected frame from table PA. In this embodiment, the frames are selected in order from the beginning of the video A recorded in the recording unit 106. More specifically, the candidate acquisition unit 301 selects one frame in order from the beginning of the video A recorded in the recording unit 106. Hereinafter, the selected frame will be referred to as frame Ax. The candidate acquisition unit 301 reads out the time information corresponding to frame Ax from the recording unit 106 by referring to table PA recorded in RAM 103, and transfers it to RAM 103.

[0046] In step S502, the candidate acquisition unit 301 compares the time information of frame Ax read in step S501 with the time information of each frame of the video B. Specifically, the candidate acquisition unit 301 refers to table PB, sequentially acquires the time information of each frame of the video B from the recording unit 106, and compares it with the time information of frame Ax.

[0047] In step S503, the candidate acquisition unit 301 acquires a frame of the moving image B having a shooting time that matches the time information of the frame Ax, and designates this as the frame Bx.

[0048] In step S504, the candidate acquisition unit 301 gives a unique index Ix in the teacher data candidate DB to the combination of the above frame Ax and frame Bx, and registers it in the teacher data candidate DB. Specifically, the candidate acquisition unit 301 issues a unique index Ix to the combination of frame Ax and frame Bx, and registers the index Ix, the frame number of frame Ax in video A, and the frame number of frame Bx in video B in the teacher data candidate DB.

[0049] In step S505, the candidate acquisition unit 301 determines whether the above-mentioned processes in steps S501 to S504 have been completed for all frames of the video A. If the candidate acquisition unit 301 determines that the processes have been completed (YES in step S505), this process ends. On the other hand, if the candidate acquisition unit 301 determines that the processes have not been completed (NO in step S505), the process returns to step S501, and the above-mentioned processes are performed for the next frame of the video A.

[0050] Through the above process, a training data candidate DB is generated.

[0051] In this embodiment, the pair of frames to be registered in the teacher data candidate DB is determined by comparing the shooting times in step S502, but this is not limited to this. For example, the frame Ax may be reduced to a resolution XB, and a similarity determination may be performed using an index indicating the similarity between the frame Ax and the image of each frame of the video B, and the pair of frames to be registered in the teacher data candidate DB may be selected using the determination result. In this case, the candidate acquisition unit 301 has a similarity determination function that compares two or more image data and determines the similarity. Note that, for example, a structural similarity index (SSIM) may be used as an index indicating the similarity of the images. In addition, in acquiring the index indicating the similarity, the image of the frame Ax is reduced to a resolution XB, but this is not limited to this. The image of the frame Ax may not be reduced, and the resolution after reduction may be other than XB.

[0052] (High-definition video generation processing) Next, the high definition moving image generation process performed by the control unit 101 (teacher data extraction unit 302) and the learning and inference unit 105 (learning unit 303, inference unit 304) will be described.

[0053] First, an overview of the high definition video generation process will be described with reference to Fig. 3 (details will be described later with reference to Fig. 7).

[0054] The teacher data extraction unit 302 selects teacher data suitable for learning the "trained model for the inference target frame By" from the teacher data candidate DB, and generates the teacher data DB. If the teacher data pairs registered in the generated teacher data DB meet a predetermined number of registrations, the learning unit 303 generates a trained model M using the extracted teacher data. Then, the inference unit 304 infers high-frequency components of the inference target frame By using the trained model M, performs high-definition processing, and obtains a high-definition frame (image) Cy of the inference target frame By. However, if the teacher data pairs registered in the generated teacher data DB do not meet a predetermined number of registrations, the subsequent learning process is not performed, and the high-frequency components of the inference target frame By are inferred using the trained model MG generated in advance and stored in the RAM to obtain Cy. Note that before the high-definition video generation process begins, the control unit 101 generates a video C on the recording unit 106. At the start of the generation of the high-definition video, the video C does not have frame data and is empty. The inference unit 304 records the generated frames Cy in the video C sequentially.

[0055] The above-mentioned high definition moving image generating process will be described in more detail below with reference to the flowchart of FIG.

[0056] In step S701, the teacher data extraction unit 302 reads out one frame from the video B as a frame to be refined. In this embodiment, the teacher data extraction unit 302 reads out one frame at a time in sequence from the beginning of the video B recorded in the recording unit 106. Hereinafter, the frame read out in step S701 is referred to as frame By. More specifically, the teacher data extraction unit 302 reads out the frame data and time information of frame By from the recording unit 106 with reference to table PB, and transfers them to RAM 103.

[0057] In step S702, the teacher data extraction unit 302 (collection means) first extracts frames whose difference in shooting time with the frame By is shorter than a threshold value previously determined by the system from among the teacher data candidates TB registered in the teacher data candidate DB. Next, the teacher data extraction unit 302 registers the extracted frames in the teacher data DB. For example, the one-frame display period of the video A (the display period of one frame at the frame rate XA) may be used as the threshold value. The structure of the teacher data DB is the same as that of the teacher data candidate DB (FIG. 6). Specifically, first, the teacher data extraction unit 302 refers to the position information of the table PB to acquire time information of each of the frames TB registered in the teacher data candidate DB. The teacher data extraction unit 302 compares each of the acquired time information with the shooting time of the frame By, extracts frames whose difference between the two is shorter than the threshold value from the frame group TB, and registers them in the teacher data DB on the RAM 103. Hereinafter, the frame group of the video B registered in the teacher data DB by the above process is referred to as UB. In this embodiment, when constructing the teacher data DB, a frame group whose shooting time with respect to the frame By is shorter than a threshold is extracted from the teacher data candidate DB, but this is not limited to this. The frame group UB may be extracted using an index representing the similarity with the frame By. For example, the teacher data extraction unit 302 may use SSIM to extract a frame group whose similarity index with the frame By is higher than a threshold previously set in the system from the frame group TB and register it as the frame group UB.

[0058] In step S703, the teacher data extraction unit 302 registers in the teacher data DB the frames of the frame group TA that are paired with each frame of the frame group UB in the teacher data candidate DB. Specifically, the teacher data extraction unit 302 refers to the teacher data candidate DB on the RAM 103, and registers in the teacher data DB the frames of the frame group TA that are associated with each frame of the frame group UB by index I. At this time, the combination of the two associated frames is not changed, and a unique index J is assigned to each combination in the teacher data DB. Hereinafter, the frame group of the video A registered in the teacher data DB is referred to as a frame group UA.

[0059] In step S704, the control unit 101 (first and second selection means) judges whether the number of teacher data pairs registered in the teacher data DB (i.e., the number of images in the frame group UA) exceeds a predetermined value. If the control unit 101 judges that the number of teacher data pairs exceeds the predetermined value (YES in step S704), it judges that the frame group UA is suitable as a teacher image group, and proceeds to step S705. On the other hand, if the control unit 101 judges that the number of teacher data pairs does not exceed the predetermined value (NO in step S704), it judges that the frame group UA is not suitable as a teacher image group, and proceeds to step S709.

[0060] In step S705, the learning unit 303 performs learning using the teacher data (frame group UA and frame group UB) registered in the teacher data DB by the trained model generation function as shown in FIG. 8, and generates a trained model M. The contents of the generation process are the same as those in the general-purpose trained model generation process, and therefore will be omitted. The generated trained model M is recorded in the RAM 103.

[0061] As described above, the learning unit 303 refers to the teacher data DB and tables PA and PB, reads out the frame data of the frame pairs registered as teacher data from the recording unit 106, and inputs the frame data to the trained model generation function described above. The learning unit 303 stores the trained model M generated by the trained model generation function in the RAM 103.

[0062] In step S706, the inference unit 304 generates a high-resolution frame Cy from the frame By by inference using the trained model M generated in step S705. Specifically, first, the inference unit 304 reads out the trained model M stored in the RAM 103. Next, the inference unit 304 inputs the frame data (image) of the frame By stored in the RAM 103 in step S701 to the trained model M. As a result, the inference result output from the trained model M, that is, "high-frequency components expected when the image of the frame By is enlarged to a resolution XA", is acquired. The inference unit 304 generates an image of the high-resolution frame Cy with a resolution XA by adding the acquired high-frequency components to "an image obtained by linearly enlarging the image of the frame By to a resolution XA", and records this in the RAM 103. Note that the above-mentioned processing from the inference of the high-frequency components performed for the frame By to the generation of the high-resolution image is the same processing as the inference process described above with reference to FIG. 8. Then, the process proceeds to step S707.

[0063] On the other hand, in step S709, the inference unit 304 generates a high-resolution frame Cy from the frame By by inference using the trained model MG generated in the general-purpose trained model generation process. Specifically, first, the inference unit 304 reads out the trained model MG stored in the RAM 103. Next, the inference unit 304 inputs the frame data (image) of the frame By stored in the RAM 103 in step S701 to the trained model MG. As a result, the inference result output from the trained model MG, that is, "high-frequency components expected when the image of the frame By is enlarged to a resolution XA", is acquired. The inference unit 304 generates an image of the high-resolution frame Cy with a resolution XA by adding the acquired high-frequency components to "an image obtained by linearly enlarging the image of the frame By to a resolution XA", and records this in the RAM 103. Note that the above-mentioned processing from the inference of the high-frequency components performed for the frame By to the generation of the high-resolution image is the same processing as the inference process described above with reference to FIG. 8. Then, proceed to step S707.

[0064] In step S707, the inference unit 304 appends the frame data of the high-definition frame Cy recorded in the RAM 103 to the end of the high-definition video C on the recording unit 106. In addition, the inference unit 304 duplicates the time information of the frame By, multiplexes it as the shooting time of the high-definition frame Cy, and records it in the video C.

[0065] In step S708, the control unit 101 determines whether or not the above process has been completed for all frames of the video B. If the control unit 101 determines that the process has not been completed (NO in step S708), the process returns to step S701, the teacher data extraction unit 302 selects the next frame of the video B as frame By, and the above process is repeated. On the other hand, if the control unit 101 determines that the process has been completed (YES in step S708), this process ends.

[0066] As described above, when the high definition moving image generating process is completed, the high definition moving image C with the resolution XA and the frame rate FB is recorded in the recording unit 106 in an uncompressed format.

[0067] In the above, each functional block has been described as being realized only by the control unit 101 or only by the learning and inference unit 105, but this is not limited thereto. For example, each functional block may be realized by cooperation between the control unit 101 and the learning and inference unit 105. For example, the function of the inference unit 304 may be realized by the control unit 101 and the learning and inference unit 105, and the process of recording the high-definition frame Cy and the shooting time in the video C on the recording unit 106 may be executed by the control unit 101.

[0068] In addition, in this embodiment, the teacher data candidate acquisition process is performed before the learning process for the entire video and the high-definition video generation process are performed, but it may be performed in parallel with the execution of the high-definition video generation process. In addition, in this embodiment, a new trained model M is generated for each frame to be inferred in step S705, and the previously generated model is discarded, but this is not limited to this. For example, a trained model M' that has been trained in advance externally may be loaded, and additional learning may be performed on the loaded trained model M' using the frame group UA and the frame group UB in step S705. In addition, when a new trained model M is generated in this embodiment, the initial value (trained model) of the trained model MG may be set to the trained model MG. This makes it possible to shorten the learning time.

[0069] As described above, according to the first embodiment, the trained model M trained using an image group similar to the image to be refined among the image groups captured in the same shooting period is used, so that the image can be refined with high accuracy. In addition, a pair of images from the two image groups captured at the same time is used as training data, so that more accurate learning is possible.

[0070] According to the first embodiment, if the number of pairs of teacher data registered in the teacher data DB does not exceed a predetermined value (NO in step S704), it is determined that the frame group UA is not suitable as a teacher image group. In this case, inference is performed using the trained model MG without generating a trained model M, so that the learning process can be performed with a certain level of inference accuracy without reducing the processing efficiency.

[0071] Second Embodiment In the first embodiment, the determination of whether to generate a trained model M by learning using training data pairs (step S704 in FIG. 7) was made based on "whether the number of training data pairs registered in the training data DB exceeds a predetermined value." In contrast, in the present embodiment, this determination is made based on "whether the average value of the similarity between the frame By and each training data pair registered in the training data DB exceeds a predetermined value."

[0072] That is, in this embodiment, except for the difference in the content of the high definition video generation process, the basic hardware configuration and software configuration of the image processing device according to this embodiment are the same as those of the first embodiment. Therefore, the same configurations and steps as those of the first embodiment are given the same numbers, and duplicated explanations are omitted.

[0073] Hereinafter, the high definition moving image generating process (FIG. 9), which is a difference between this embodiment and the first embodiment, will be described.

[0074] (High-definition video generation processing) FIG. 9 is a flowchart of high definition moving image generation processing according to the second embodiment.

[0075] The process of Fig. 9 is the same as the process of Fig. 7b, except that step S904 is executed instead of step S704 in Fig. 7. Therefore, only the process of step S904 will be described here.

[0076] In step S904, the control unit 101 sequentially reads out frames of the frame group UB registered in the teacher data DB, and executes a similarity calculation with the frame By read out in step S701 using SSIM. Next, the control unit 101 calculates the average value of the similarities calculated for all frames of the frame group UB, and determines whether the calculated average value exceeds a predetermined value. If the control unit 101 determines that the average value exceeds the predetermined value (YES in step S904), the process proceeds to step S705. On the other hand, if the control unit 101 determines that the average value does not exceed the predetermined value (NO in step S904), the process proceeds to step S909.

[0077] As described above, according to the second embodiment, for example, if frame By is a frame at the moment the flash goes off, the similarity between frame By and frame group UB is low, and the average value calculated in step S904 does not exceed a predetermined value (NO in step S904). In such a case, even if the number of teacher data pairs registered in the teacher data DB exceeds a predetermined value (YES in step S704), the trained model M is not generated by learning using the teacher data pairs, and inference is performed using the trained model MG. This allows the learning process to be performed with a certain level of inference accuracy without reducing the processing efficiency.

[0078] <Third embodiment> In the first embodiment, when the trained model M is not generated by learning using the teacher data pair (NO in step S704 in FIG. 7), the high-resolution frame Cy is generated using the trained model MG that was generated in advance by the general-purpose trained model generation process. On the other hand, in the present embodiment, two trained models MG1 and MG2 (third and fourth trained models) trained using different teacher image groups are generated in advance in the general-purpose trained model generation process. Thereafter, when the trained model M is not generated by learning using the frame group UA and the frame group UB (teacher data pair), a model more suitable as a model for inferring the high-resolution frame Cy from the frame By is selected from the trained models MG1 and MG2.

[0079] That is, in this embodiment, except for the differences in the contents of the general-purpose trained model generation process and the high-definition video generation process, the basic hardware configuration and software configuration of the image processing device according to this embodiment are the same as those of the first embodiment. Therefore, the same configurations and steps as those of the first embodiment are denoted by the same numbers, and duplicated explanations will be omitted.

[0080] Below, we will explain the general-purpose trained model generation process and high-definition video generation process (Figure 10), which are the parts of this embodiment that differ from the first embodiment.

[0081] (General-purpose trained model generation process) In the general-purpose trained model generation process according to the third embodiment, the learning unit 303 of the learning and inference unit 105 generates general-purpose trained models MG1 and MG2. Two image groups collected to have different image patterns are recorded in the recording unit 106. These image groups are called general-purpose teacher image groups K1 and K2. The learning unit 303 constructs a trained model MG1 by learning using the general-purpose teacher image group K1 (one of the two different image groups) as a teacher image using the trained model generation function (FIG. 8). Similarly, the learning unit 303 constructs a trained model MG2 by learning using the general-purpose teacher image group K2 (the other of the two different image groups) as a teacher image. After that, the learning unit 303 stores the trained model MG1 in the RAM 103 together with additional information that corresponds to the position information on the recording unit 106 of the general-purpose teacher image group K1 used in the learning. Similarly, the learning unit 303 stores the trained model MG2 in the RAM 103 together with additional information that associates the trained model MG2 with the position information on the recording unit 106 of the general-purpose teacher image group K2 used in the training.

[0082] (High-definition video generation processing) FIG. 10 is a flowchart of high definition moving image generation processing according to the third embodiment.

[0083] The process in Fig. 10 is the same as the process in Fig. 7b, except that steps S1009 and S1010 are executed instead of step S709, so only the processes in steps S1009 and S1010 will be described here.

[0084] In step S1009, the control unit 101 first refers to the additional information recorded in the RAM 103 together with the trained model MG1, and acquires the position on the recording unit 106 of the general-purpose teacher image group K1 associated with the trained model MG1. The control unit 101 reads images belonging to the general-purpose teacher image group K1 one by one into the RAM 103, and executes a similarity calculation using SSIM between the frame By on the RAM 103. After the calculation of the similarity for all images of the general-purpose teacher image group K1 is completed, the average value Av1 is calculated and stored in the RAM 103. The same process is also performed for the general-purpose teacher image group K2 associated with the trained model MG2, and an average value Av2 of the similarity between each image belonging to the general-purpose teacher image group K2 and the frame By is acquired. Thereafter, the control unit 101 compares the average value Av1 with the average value Av2, and selects the trained model with the higher average value from the trained models MG1 and MG2.

[0085] In step S1010, the inference unit 304 generates a high-resolution frame Cy from the frame By by inference using the trained model MG selected in step S1009.

[0086] As described above, according to the third embodiment, when the trained model M is not generated (NO in step S704), a model trained using training data having a higher similarity to the frame By, which is the frame to be refined, is selected from the trained models MG1 and MG2. After that, a high-resolution frame Cy is generated from the frame By by inference using the selected model.

[0087] The trained model MG used in the first embodiment is a model generated in advance using training data collected when the contents of the frame By are unknown. For this reason, the training data uses a wide variety of images without bias in image patterns, and the training data with a high similarity to the frame By is very small, so the inference accuracy of the trained model MG is not high. On the other hand, the trained models MG1 and MG2 used in this embodiment are similar to the trained model MG in that they are models generated in advance using training data collected when the contents of the frame By are unknown. However, of the trained models MG1 and MG2, the model generated using training data with a higher similarity to the frame By is selected in step S1009. For this reason, it is expected that the inference result of the high-resolution frame Cy obtained in step S1010 of the third embodiment will be more accurate than the result obtained in step S709 of the first embodiment.

[0088] The configuration of this embodiment can also be applied to the second embodiment.

[0089] (Other embodiments) In this embodiment, a program for implementing one or more functions may be provided to a computer of the system or device via a network or storage medium, and the system controller of the system or device may read and execute the program. The system controller may have one or more processors or circuits, and may include multiple separate system controllers or a network of multiple separate processors or circuits to read and execute the executable instructions.

[0090] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0091] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0092] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) An image processing apparatus that super-resolves a second image group having a frame rate FB and a resolution XB to a resolution XA (>XB) of a first image group having a frame rate FA (<FB) by inferring the second image group among the first and second image groups generated by simultaneously photographing the same subject at different resolutions and frame rates using a learned model, the learning means for learning a learning model using a teacher image group having the resolution XA to generate the learned model, the collecting means for collecting an image group from the first image group based on the super-resolution target image every time one super-resolution target image is obtained from the second image group, the first selection means for generating a first learned model by the learning means using the collected image group as the teacher image group and selecting the generated first learned model as the learned model when it is determined that the collected image group is suitable as the teacher image group, and the second selection means for selecting a second learned model previously generated by the learning means using an image group collected in advance as the teacher image group as the learned model when it is determined that the collected image group is not suitable as the teacher image group. (Configuration 2) The image processing apparatus according to Configuration 1, characterized in that when the number of images in the collected image group exceeds a predetermined value, it is determined that the collected image group is suitable as the teacher image group. (Configuration 3) The image processing apparatus according to Configuration 2, characterized in that the collected image group is a frame group collected from the first image group and having a difference in shooting time from the super-resolution target image shorter than a threshold value. (Configuration 4) An image processing device as described in Configuration 2, characterized in that from among frames of the second image group that are deemed to have been taken at the same time as the first image group, a frame group having a similarity to the image to be enhanced in definition that is higher than a threshold value is extracted, and the image group that is deemed to have been taken at the same time as the extracted frame group of the first image group is set as the collected image group. (Configuration 5) An image processing device as described in configuration 1, characterized in that when the average similarity between a group of frames from the second image group that is deemed to have been taken at the same time as the collected image group and the image to be refined exceeds a predetermined value, the collected image group is judged to be suitable as the teacher image group. (Configuration 6) The image processing device described in any one of configurations 1 to 5, characterized in that the learning means uses the second trained model as an initial value when generating the first trained model. (Configuration 7) The previously collected image group includes two different image groups, The image processing device according to any one of claims 1 to 6, characterized in that the learning means generates a third trained model using one of the two different image groups as the teacher image group, and generates a fourth trained model using the other of the two different image groups as the teacher image group, and selects the third trained model as the second trained model if one of the two different image groups has a higher average similarity to the high definition target image than the other, and selects the fourth trained model as the second trained model if one of the two different image groups has a higher average similarity to the high definition target image than the other. (Method 1) Among the first and second image groups, which are groups of frames of two videos generated by simultaneously photographing the same object at different resolutions and frame rates, the second image group having a frame rate FB and a resolution XB is inferred using a learned model to increase the resolution to the resolution XA (> XB) of the first image group having a frame rate FA (<FB). A control method for an image processing apparatus, comprising: a learning step of learning a learning model using a teacher image group having the resolution XA to generate the learned model; a collection step of collecting an image group from the first image group based on the super-resolution target image every time one super-resolution target image is obtained from the second image group; a first selection step of, when it is determined that the collected image group is suitable as the teacher image group, generating a first learned model in the learning step using the collected image group as the teacher image group and selecting the generated first learned model as the learned model; and a second selection step of, when it is determined that the collected image group is not suitable as the teacher image group, selecting a second learned model previously generated in the learning step using a previously collected image group as the teacher image group as the learned model. (Program 1) A program for causing a computer to function as each means of the image processing apparatus according to any one of Configurations 1 to 7.

Explanation of Signs

[0093] 100 Image processing apparatus 101 Control unit 102 ROM 103 RAM 104 Decoding unit 105 Learning and inference unit 106 Recording unit 107 Bus 201 Analysis unit 202 Decoded video generation unit 301 Candidate acquisition unit 302 Teacher data extraction unit 303 Learning unit 304 Inference unit

Claims

1. (Basic configuration) An image processing device that, among first and second image groups (A, B) which are respective frame groups of two videos (A, B) generated by simultaneously shooting the same subject at different resolutions and frame rates, increases the resolution of the second image group (B) having a frame rate FB and a resolution XB to the resolution XA (>XB) of the first image group (A) having a frame rate FA (<FB) by inference using a trained model, A learning means (303) for learning a learning model using a group of teacher images having the resolution XA to generate the learned model; a collection means (302) for collecting an image group (UA) from the first image group (TA) based on an image to be refined (By) obtained from the second image group (B); When it is determined that the collected image group (UA) is suitable as the teacher image group (Y in S704), a first selection means generates a first trained model (M) by the learning means using the collected image group (UA) as the teacher image group, and selects the generated first trained model (M) as the trained model; A second selection means for selecting, as the trained model, a second trained model (MG) generated in advance by the learning means using a previously collected image group (K) as the trained image group when it is determined that the collected image group (UA) is not suitable as the trained image group (N in S704); An image processing device comprising:

2. (Decision on whether to carry out learning) The image processing device according to claim 1, characterized in that if the number of images (number of images registered in D2) in the collected image group (UA) exceeds a predetermined value (S704), the collected image group (UA) is judged to be suitable as the teacher image group.

3. 3. The image processing device according to claim 2, wherein the collected image group (UA) is a frame group collected from the first image group (TA) and the difference between the shooting time of the image to be enhanced and the shooting time of the image to be enhanced is shorter than a threshold value.

4. extracting a frame group (UB) having a similarity to the high definition target image higher than a threshold value from a frame group (TB) of the second image group (B) that is deemed to have been taken at the same timing as each of the first image group (TA); 3. The image processing apparatus according to claim 2, wherein an image group considered to have been taken at the same timing as the extracted frame group (UB) from the first image group (TA) is defined as the collected image group (UA).

5. The image processing device described in claim 1, characterized in that if the average similarity between a frame group (UB) of the second image group (B) that is deemed to have been taken at the same time as the collected image group (UA) and the image to be refined (By) exceeds a predetermined value (S904), the collected image group (UA) is judged to be suitable as the teacher image group.

6. (Initial value before learning the learning model: to shorten the learning time) The image processing device according to claim 1 , wherein the learning means uses the second trained model (MG) as an initial value when generating the first trained model (M).

7. (Model selection method when there are multiple candidate learning models to be stored in advance) The pre-collected image group (K) includes two different image groups; The learning means generates a third trained model (MG1) using one of the two different image groups as the teacher image group, and generates a fourth trained model (MG2) using the other of the two different image groups as the teacher image group; If the average similarity between the high definition target image and one of the two different image groups is higher than the other, the third trained model (MG1) is selected as the second trained model (MG); The image processing device according to claim 1, characterized in that, when the average similarity to the high-definition target image is higher for one of the two different image groups than for the other, the fourth trained model (MG2) is selected as the second trained model (MG).

8. A control method for an image processing device for increasing the resolution of a second image group having a frame rate FB and a resolution XB by inference using a trained model, the second image group being one of first and second image groups, each of which is a frame group of two moving images that are simultaneously shot at different resolutions and frame rates, to a resolution XA (>XB) of the first image group having a frame rate FA (<FB), the method comprising: A learning step of learning a learning model using a teacher image group having the resolution XA to generate the learned model; a collection step of collecting an image group from the first image group based on an image to be refined each time an image to be refined is obtained from the second image group; When it is determined that the collected image group is suitable as the teacher image group, a first trained model is generated in the learning step by using the collected image group as the teacher image group, and the generated first trained model is selected as the trained model; and A control method characterized by having a second selection step of selecting, if it is determined that the collected image group is not suitable as the teacher image group, a second trained model that was generated in advance in the learning step using a previously collected image group as the teacher image group, as the trained model.

9. 2. A program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images