Super-resolution processing system, image processing device, inference device, control method, and program

Abstract

The present invention provides a super-resolution processing system, image processing device, inference device, control method, and program that can reduce the data size of data required for playback of high-resolution, high-frame-rate videos while maintaining the accuracy of super-resolution processing performance. [Solution] An image processing device that analyzes an input image and generates a trained model for inferring high-frequency components using machine learning, uses a first resolution XA video A and a resolution XB video (

Description

Technical Field

[0001] The present invention relates to a super-resolution processing system, an image processing apparatus, an inference apparatus, a control method, and a program, and particularly relates to a super-resolution processing system, an image processing apparatus, an inference apparatus, a control method, and a program for enhancing the resolution of an image group using machine learning.

Background Art

[0002] Conventionally, when enlarging an image and performing resolution conversion, there is a known super-resolution technique for generating a high-definition image by inferring high-frequency components that cannot be compensated by linear interpolation processing of pixel values using machine learning. In the super-resolution technique, first, machine learning is performed using an image group G and a degraded image obtained by degrading each image in the image group G by an arbitrary method as teacher data, and a learned model is generated. The learned model is generated by learning the difference between the pixel values of the current image and the degraded image and updating the super-resolution processing parameters it holds. When an image H lacking high-frequency components is input to the learned model generated in this way, high-frequency components are obtained by inference using the learned model. By superimposing the high-frequency components obtained by inference on the image H, a high-definition image can be generated. When performing super-resolution processing on a video, a high-definition video can be generated by inputting each frame one by one into the learned model.

[0003] Generally, when products or services using a learned model are provided, the learning process using the collected teacher data is performed by developers. Therefore, techniques such as preparing a large amount of various images with no bias in image patterns as teacher data and repeatedly learning while assuming all inference targets, or learning only images whose inference targets, shooting locations, and shooting conditions are similar as teacher data have been proposed. However, none of them can be said to have sufficient inference accuracy for high-frequency components.

[0004] Therefore, in Patent Document 1, in a first video group and a second video group in which the corresponding frames have fewer high-frequency components than the first video group, an image included in the first video group and its corresponding image are selected from the second video group, and a trained model is generated from these images. Then, a high-resolution image is generated by inferring the high-frequency components of the inference target image in the second video group using the generated trained model.

[0005] Patent Document 1 describes a method in which a high-resolution, low-frame-rate video A and a low-resolution, high-frame-rate video B are simultaneously captured using the same imaging device. A trained model is generated using frames from each video at the same capture time, and the frames of video B are inferred using the generated trained model. This makes it possible to generate a high-resolution, high-frame-rate video C. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-127577 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the technology disclosed in Patent Document 1 has the problem that the video obtained by super-resolution processing has a large data size due to its high resolution and high frame rate, which can strain storage.

[0008] Therefore, the object of the present invention is to provide a super-resolution processing system, image processing device, inference device, control method, and program that can reduce the data size of data required for playback of high-resolution, high-frame-rate videos while maintaining the accuracy of super-resolution processing performance. [Means for solving the problem]

[0009] To solve the above problems, the image processing apparatus according to claim 1 of the present invention is an image processing apparatus that analyzes an input image and generates a trained model for inferring high-frequency components by machine learning, and is characterized by comprising: acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; selection means for selecting at least one of the first group of images as training data for each image in the second group of images; learning means for performing machine learning using the training data selected by the selection means and generating a trained model corresponding to each image in the second group of images; and storage means for associating and storing each image in the second group of images with the corresponding trained model generated by the learning means.

[0010] To solve the above problems, the inference device according to claim 8 of the present invention is an inference device that exchanges data with the image processing device, and is characterized by comprising: a first acquisition means for acquiring an image to be refinished from the second group of images stored in the storage means; a second acquisition means for acquiring a trained model associated with the image acquired by the first acquisition means from the storage means; an inference means for inputting the image to be refinished into the trained model acquired by the second acquisition means and inferring the high-frequency components of the image to be refinished; and a generation means for generating an image in which the image to be refinished has been refinished based on the image to be refinished and the high-frequency components inferred by the inference means.

[0011] To solve the above problems, the super-resolution processing system according to claim 9 of the present invention is a super-resolution processing system comprising an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components by machine learning, wherein the image processing device comprises: acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; selection means for selecting at least one of the first group of images as training data for each image in the second group of images; learning means for performing machine learning using the training data selected by the selection means and generating a trained model corresponding to each image in the second group of images; and storage means for associating and storing each image in the second group of images with the corresponding trained model generated by the learning means. [Effects of the Invention]

[0012] According to the present invention, it is possible to reduce the data size of the data required for playback of high-resolution, high-frame-rate videos while maintaining the accuracy of super-resolution processing performance. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the hardware configuration of the super-resolution processing system according to the first embodiment. [Figure 2] This is a block diagram showing the functional configuration of the super-resolution processing system according to the first embodiment. [Figure 3] This is a flowchart of the training data selection process according to the first embodiment. [Figure 4] This block diagram shows the hardware configuration of the super-resolution processing system according to the second embodiment. [Figure 5] This block diagram shows the hardware configuration of the super-resolution processing system according to the third embodiment. [Modes for carrying out the invention]

[0014] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined.

[0015] (First Embodiment) In the super-resolution processing system 1a of the first embodiment, videos A and B taken during the same period by the same imaging device are previously held in a recording medium 300 described later. Here, the resolution XA (first resolution) of video A is high resolution, and the resolution XB (second resolution) of video B is low resolution (XA > XB). Also, the frame rate of video A is less than or equal to the frame rate of video B.

[0016] FIG. 1 is a block diagram showing the hardware configuration of the super-resolution processing system 1a according to the first embodiment.

[0017] As shown in FIG. 1, the super-resolution processing system 1a includes a recording medium 300, an image processing device 100 that exchanges (transmits and receives) data via the recording medium 300, and an inference device 200.

[0018] The image processing device 100 includes a control unit 101, a ROM 102, a RAM 103, a learning unit 104, and a recording unit 105.

[0019] The control unit 101 is an arithmetic device such as a CPU, and realizes various functions by expanding and executing a program stored in the ROM 102 or dedicated hardware not shown into the RAM 103. The control unit 101 has an analysis unit 111 and a selection unit 112, which will be described later with reference to FIG. 2, as functional blocks.

[0020] The ROM 102 stores programs executed by the control unit 101. The RAM 103 is used as a work memory for the control unit 101 to execute programs and a temporary storage area for various data.

[0021] The learning unit 104 has, as functional blocks, a learning processing unit 141 and a storage unit 142, which will be described later with reference to FIG. 2, for inputting teacher data to generate and store a learned model.

[0022] The recording unit 105 controls the recording medium 300. Specifically, the recording unit 105 controls, in accordance with an instruction from the control unit 101, initialization of the recording medium 300, data transfer between the recording medium 300 and the RAM 103 for reading and writing data, and the like.

[0023] The bus 106 is an information communication path connecting each function, and the control unit 101, the ROM 102, the RAM 103, the learning unit 104, and the recording unit 105 are connected to each other via the bus 106 so as to be communicable.

[0024] The inference device 200 includes a control unit 201, a ROM 202, a RAM 203, an inference unit 204, a recording unit 205, and a bus 206. The control unit 201, the ROM 202, the RAM 203, the recording unit 205, and the bus 206 have the same functions as the control unit 101, the ROM 102, the RAM 103, the recording unit 105, and the bus 106 in the image processing device 100.

[0025] The inference unit 204 analyzes an input image using the learned model generated by the learning in the learning unit 104, infers high-frequency components, and generates a high-definition image of the input image based on the input image to be analyzed and the inferred high-frequency components. For this purpose, the inference unit 204 has, as functional blocks, an inference processing unit 241 and an acquisition unit 242, which will be described later with reference to FIG. 2. In the present embodiment, a learned model is generated using a CNN model for super-resolution processing based on a convolutional neural network (CNN).

[0026] The recording medium 300 is a recording medium such as a hard disk drive (HDD) or a memory card, or a cloud server, which is detachably connected to the image processing device 100 and the inference device.

[0027] It should be noted that the hardware blocks and functional blocks executed within them described in this embodiment do not necessarily have to be configured as described above. Furthermore, some of the functional blocks may reside on a cloud server (not shown), and the processing result data may be transferred via communication.

[0028] Next, we will describe the processing method used in this embodiment to reduce data size while maintaining high-resolution super-resolution processing performance.

[0029] Figure 2 is a block diagram showing the functional configuration of the super-resolution processing system 1a according to the first embodiment. The operation of the functional blocks in image processing and playback using the image processing device 100, inference device 200, and recording medium 300 will be explained below using Figure 2.

[0030] The image processing device 100 includes an analysis unit 111 and a selection unit 112 in the control unit 101, and a learning processing unit 141 and a storage unit 142 in the learning unit 104.

[0031] The analysis unit 111 (acquisition means) acquires video A and video B stored in the recording medium 300 via the recording unit 105, analyzes each image frame of the acquired videos A and B, and stores the frame number, position information of the frame data, and shooting time in RAM 103. The selection unit 112 (selection means) receives one image frame Bn of video B as input, and first selects training data suitable for generating a trained model of image frame Bn from the candidate training data of image A in RAM 103. Next, the selection unit 112 stores the selected candidate training data together with image frame Bn in RAM 103.

[0032] Here, the training data selection process according to the first embodiment, in which the selection unit 112 selects training data suitable for generating a trained model, will be explained using the flowchart in Figure 3.

[0033] This process is executed by the control unit 101 loading the program stored in the ROM 102 into the RAM 103 and reading it.

[0034] In step S301, the image frame Bn (target image) of video B is read from the recording medium 300 via the recording unit 105. In step S302, an image frame of video A is obtained from the RAM 103 in which the difference between image frame Bn and the shooting time is less than a predetermined threshold T, and the process proceeds to step S303. In step S302, it is sufficient to obtain an image frame from video A in which the difference between image frame Bn and the shooting time is less than a predetermined threshold T; a single image frame may be obtained, or a group of image frames may be obtained.

[0035] In step S303, it is determined whether one or more image frames of video A smaller than the threshold T have been acquired. If one or more image frames of video A smaller than the threshold T have been acquired (YES in step S303), the process proceeds to step S304, where it is checked whether there is an image frame among the acquired image frames of video A that is captured at the same time as image frame Bn. If such an image frame exists (YES in step S304), the process proceeds to step S305, where the image frame of video A captured at the same time as image frame Bn is stored in RAM 103 along with image frame Bn as training data. On the other hand, if no image frame of the same time exists in step S304 (NO in step S304), the process proceeds to step S306, where the group of image frames of video A acquired in step S303 (one or more) is stored in RAM 103 along with image frame Bn as training data.

[0036] On the other hand, if no image frames smaller than the threshold T are acquired (NO in step S303), the process proceeds to step S307. In step S307, the image frame with the smallest difference in capture time from image frame Bn among the image frame group of video A is stored in RAM 103 along with image frame Bn as training data. Note that the training data obtained in steps S306 and S307 may consist of multiple image frames.

[0037] The image frame Bn obtained by the selection unit 112 and the image frames of video A selected as suitable training data may be stored together in RAM 103 for all frames of video B. Alternatively, the analysis unit 111 and the selection unit 112 may be treated as a single functional block.

[0038] Next, in the learning unit 104, the learning processing unit 141 first obtains image frame Bn of video B and training data suitable for image frame Bn selected by the selection unit 112 from the RAM 103. Then, the learning processing unit 141 (learning means) performs machine learning using the acquired training data and generates a trained model corresponding to image frame Bn. The storage unit 142 adds information linking this generated trained model with image frame Bn and saves it to the recording medium 300 (storage means) via the recording unit 105.

[0039] After generating a trained model for all frames of video B, video A is deleted from the recording medium 300. The recording medium 300 that stores video B and the trained model may be a different recording medium from the one that stores video A and video B, in which case video A does not need to be deleted.

[0040] Here, the following examples of addition means for adding information that links the trained model stored in the storage unit 142 with the image frame Bn are provided: Addition means 1 to 4. Addition means 1 adds the identification information of image frame Bn as metadata to the generated trained model. Addition means 2 adds the identification information of the trained model as metadata to image frame Bn. Addition means 3 adds the generated trained model itself to image frame Bn. Addition means 4 maintains a management table separate from image frame Bn and the generated trained model, and manages the information that links image frame Bn and the trained model in that management table. Note that other means may also be used as addition means.

[0041] Next, the inference device 200 will be described using Figure 2.

[0042] The inference device 200 includes an inference unit 204 with an inference processing unit 241 and an acquisition unit 242, and a RAM 203 with a group of trained models.

[0043] The acquisition unit 242 (first acquisition means) acquires image frames Bn of the video B, which is the image to be high-resolution, from the recording medium 300 via the recording unit 205.

[0044] Here, the acquisition means (second acquisition means) for acquiring the trained model associated with the image frame Bn of video B by the acquisition unit 242 is exemplified by acquisition means 1 to 4, which are described below.

[0045] When the above additional means 1 is used, the acquisition means 1 is used. In the acquisition means 1, the acquisition unit 242 first imports the trained model group from the recording medium 300 to the RAM 203 via the recording unit 205. The acquisition unit 242 analyzes the trained model group 231 in the RAM 203 and acquires a trained model having identification information for image frame Bn.

[0046] When the above-mentioned additional means 2 is used, the acquisition means 2 is used. In the acquisition means 2, the acquisition unit 242 analyzes the image frame Bn, identifies the trained model associated with the image frame Bn from the trained model identification information contained in the image frame Bn, and acquires the trained model from the recording medium 300 via the recording unit 205.

[0047] When the above-mentioned addition means 3 is used, the acquisition means 3 is used. In the acquisition means 3, the acquisition unit 242 analyzes the image frame Bn, extracts the trained model attached to the image frame Bn, and acquires it as a trained model.

[0048] When the above-mentioned additional means 4 is used, the acquisition means 4 is used. In the acquisition means 3, the acquisition unit 242 acquires identification information of the trained model associated with the image frame Bn from the management table and acquires the trained model from the recording medium 300 via the recording unit 205.

[0049] Next, in the inference unit 204, the inference processing unit 241 performs super-resolution processing using the image frame Bn of video B and the learned model acquired by the acquisition unit 242. Specifically, the inference unit 204 (inference means) first inputs the image frame Bn into the learned model associated with the image frame Bn and infers its high-frequency components. Next, the inference unit 204 (generation means) generates a super-resolved image of the image frame Bn based on the image frame Bn and its inferred high-frequency components. It is played and displayed on a display device (not shown). If there is no problem with storage space, all frames of the super-resolved video B may be collectively stored in the recording medium 300.

[0050] As described above, according to the present embodiment, in order to obtain a video obtained by super-resolving video B (resolution XA, frame rate FB), only video B and the learned model of each frame (for each image) of video B need to be stored in the recording medium 300. That is, there is no need to store video A in the recording medium 300. This makes it possible to reduce the data size of the recording medium 300.

[0051] Here, as the teacher data for each image frame of the video B (first image group) to be super-resolved, the image frames (second image group) of the video A that were shot by the same imaging device during the same period as the video B and stored in the recording medium 300 were used, but it is not limited to this. That is, each first image group having a resolution XB (<XA) only needs to have a certain correlation with at least one of the second image groups having a resolution XA.

[0052] For example, the image frames of video A shot by an imaging device may be used as the first image group, and the image frames of video B obtained by resizing from that video A may be used as the second image group. At this time, the relationship between the resolution XA of video A and the resolution XB of video B is "XA>XB". In this case, video B holds information on the shooting time of the image that is the resizing source as metadata.

[0053] Alternatively, still images of resolutions XA and XB may be acquired multiple times simultaneously using the same imaging device. From the acquired still image groups, the still images with resolution XA may be used as the first image group, and the still images with resolution XB may be used as the second image group.

[0054] (Second embodiment) Next, a second embodiment will be described. In this embodiment, the same numbering will be used for components identical to those in the first embodiment, and redundant explanations will be omitted.

[0055] In the super-resolution processing system 1a according to the first embodiment, the image processing device 100 and the inference device 200 exchange data via the recording medium 300, thereby reducing the data size of the recording medium 300. In contrast, in the super-resolution processing system 1b according to this embodiment, as shown in Figure 4, the image processing device 100 and the inference device 200 exchange data directly without going through the recording medium 300, thereby reducing the data size of the recording medium 300.

[0056] As shown in Figure 4, in this embodiment, the image processing device 100 and the inference device 200 each have communication units 107 and 207 (first communication unit and second communication unit) instead of recording units 105 and 205 (Figure 1). Furthermore, the recording medium 300 is not connected to the recording units 105 and 205, but is connected to the RAM 103.

[0057] Therefore, in this embodiment, the process shown in Figure 3 is basically performed in the same way as in the first embodiment, but the image frame Bn from the recording medium 300 in step S301 is read via the RAM 103 instead of the recording unit 105.

[0058] Furthermore, the trained model generated by the learning unit 104 is transmitted along with video B from the image processing device 100 to the inference device 200 via the communication unit 107 and a communication path such as Wi-Fi. The inference device 200 receives video B and the trained model from the image processing device 100 via the communication unit 207 and stores them in the RAM 203 (storage means).

[0059] As a result, the RAM 203 of the inference device 200 only needs to store video B and the trained models for each frame of video B, thus reducing the data size. The communication path is not limited to Wi-Fi; it may also be a wireless communication path using other communication methods such as Bluetooth (registered trademark), or a wired communication path.

[0060] In this embodiment, after generating a trained model for all frames of video B, video A is deleted from the recording medium 300. However, in this embodiment, since the trained model for video B is not stored on the recording medium 300, it is not necessarily required to delete video A from the recording medium 300.

[0061] (Third embodiment) Next, a third embodiment will be described. In this embodiment, the same numbering will be used for components identical to those in the first embodiment, and redundant explanations will be omitted.

[0062] The super-resolution processing system 1a according to the first embodiment had an image processing device 100 having a learning unit 104 and an inference device 200 having an inference unit 204. In contrast, the super-resolution processing system 1c according to this embodiment consists of an image processing device 100 having a learning unit 104 and an inference unit 204, as shown in Figure 5. In this case, the recording medium 300 and the RAM 103 (storage means) of the image processing device 100 only need to store video B and the trained models for each frame of video B, thus reducing the data size. That is, in this embodiment as in the first embodiment, video A is captured by the same imaging device during the same period as video B and stored in the recording medium 300, but after a trained model is generated for all frames of video B, video A is deleted from the recording medium 300.

[0063] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of this embodiment to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and operate the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0064] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0065] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An image processing apparatus for analyzing an input image and generating a trained model for inferring high-frequency components by machine learning, comprising: acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; selection means for selecting at least one of the first group of images as training data for each image in the second group of images; learning means for performing machine learning using the training data selected by the selection means and generating a trained model corresponding to each image in the second group of images; and storage means for associating and storing each image in the second group of images with the corresponding trained model generated by the learning means. (Configuration 2) The image processing apparatus according to Configuration 1, characterized in that the selection means selects an image from the first image group that was taken at the same time for each image in the second image group. (Configuration 3) The image processing apparatus according to Configuration 1, characterized in that the selection means selects from the first image group an image in which the difference in shooting time is smaller than a predetermined threshold for each image in the second image group. (Configuration 4) The image processing apparatus according to Configuration 3, wherein the selection means, if there is no image in the first image group whose difference in shooting time from the target image in the second image group is less than the predetermined threshold, selects at least one image from the first image group that has the smallest difference in shooting time from the target image. (Configuration 5) An image processing apparatus according to any one of Configurations 1 to 4, characterized in that the first image group and the second image group are image frames of a video captured at the same time by the same imaging device, and the frame rate of the first image group is less than or equal to the frame rate of the second image group. (Configuration 6) The image processing apparatus according to any one of Configurations 1 to 4, characterized in that the first image group and the second image group are image groups of the first resolution and the second resolution, which were captured simultaneously by the same imaging device. (Configuration 7) The image processing apparatus according to any one of Configurations 1 to 4, wherein the second group of images is a group of images obtained by resizing each of the first group of images, and the apparatus holds information on the time of capture of the original image from the first group of images as metadata. (Configuration 8) An inference device that exchanges data with an image processing device described in any one of Configurations 1 to 7, comprising: a first acquisition means for acquiring an image to be refinished from the second group of images stored in the storage means; a second acquisition means for acquiring a trained model associated with the image acquired by the first acquisition means from the storage means; an inference means for inputting the image to be refinished into the trained model acquired by the second acquisition means and inferring the high-frequency components of the image to be refinished; and a generation means for generating an image in which the image to be refinished has been refinished based on the image to be refinished and the high-frequency components inferred by the inference means. (Configuration 9) A super-resolution processing system comprising an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components by machine learning, wherein the image processing device comprises: acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; selection means for selecting at least one of the first group of images as training data for each image in the second group of images; learning means for performing machine learning using the training data selected by the selection means and generating a trained model corresponding to each image in the second group of images; and storage means for associating and storing each image in the second group of images with the corresponding trained model generated by the learning means. (Configuration 10) The super-resolution processing system according to Configuration 9, further comprising an inference device that exchanges data with the image processing device, wherein the inference device includes: a first acquisition means for acquiring a high-resolution target image from the second group of images stored by the storage means; a second acquisition means for acquiring a trained model associated with the image acquired by the first acquisition means from the storage means; an inference means for inputting the high-resolution target image into the trained model acquired by the second acquisition means and inferring the high-frequency components of the high-resolution target image; and a generation means for generating a high-resolution image of the high-resolution target image based on the high-resolution target image and the high-frequency components inferred by the inference means. (Configuration 11) The super-resolution processing system according to Configuration 10, characterized in that the storage means is a recording medium capable of transmitting and receiving data between the image processing device and the inference device, respectively. (Configuration 12) The super-resolution processing system according to Configuration 10, wherein the storage means is a recording medium provided by the inference device, the image processing device further comprises a first communication unit, and the inference device further comprises a second communication unit, and the images of the second image group and the corresponding trained models are stored in the storage means via the first communication unit and the second communication unit. (Configuration 13) The super-resolution processing system according to Configuration 9, wherein the storage means is a recording medium provided by the image processing device, and the image processing device further comprises: a first acquisition means for acquiring an image to be refinished from the second group of images stored by the storage means; a second acquisition means for acquiring a trained model associated with the image acquired by the first acquisition means from the storage means; an inference means for inputting the image to be refinished into the trained model acquired by the second acquisition means and inferring the high-frequency components of the image to be refinished; and a generation means for generating an image in which the image to be refinished has been refinished based on the image to be refinished and the high-frequency components inferred by the inference means. (Method 1) A control method for an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components by machine learning, comprising: an acquisition step of acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; a selection step of selecting at least one of the first group of images as training data for each image in the second group of images; a learning step of performing machine learning using the training data selected in the selection step to generate a trained model corresponding to each image in the second group of images; and a storage step of associating each image in the second group of images with the corresponding trained model generated in the learning step and saving them. (Method 2) A control method for an inference device that exchanges data with an image processing device described in any one of configurations 1 to 7, comprising: a first acquisition step of acquiring an image to be refinished from the second group of images saved in the saving step; a second acquisition step of acquiring a trained model associated with the image acquired in the first acquisition step from the saving step; an inference step of inputting the image to be refinished into the trained model acquired in the second acquisition step and inferring the high-frequency components of the image to be refinished; and a generation step of generating an image in which the image to be refinished has been refinished based on the image to be refinished and the high-frequency components inferred in the inference step. (Method 3) A control method for a super-resolution processing system comprising an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components by machine learning, the control method comprising: an acquisition step of acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images; a selection step of selecting at least one of the first group of images as training data for each image in the second group of images; a learning step of performing machine learning using the training data selected in the selection step to generate a trained model corresponding to each image in the second group of images; and a storage step of associating each image in the second group of images with the corresponding trained model generated in the learning step and saving them. (Program 1) A program for causing a computer to function as one of the means of the image processing apparatus described in any one of Configurations 1 to 7. (Program 2) A program for causing the computer to function as one of the means of the inference device described in Configuration 8. (Program 3) A program for causing a computer to function as one of the means of the super-resolution processing system described in any one of configurations 9 to 13. [Explanation of symbols]

[0066] 100 Image Processing Devices 200 Reasoning device 101,201 Control Unit 111 Analysis Department 112 Selection Section 102,202 ROM 103,203 RAM 104 Learning Department 141 Learning Processing Unit 142 Preservation Department 204 Reasoning section 241 Inference Processing Unit 242 Acquisition Department 105,205 Records Section 106,206 buses 231 Pre-trained models (group of pre-trained models) 300 recording media

Claims

1. An image processing device that analyzes an input image and generates a trained model for inferring high-frequency components using machine learning, An acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images, A selection means for selecting at least one of the first image group as training data for each image in the second image group, A learning means that performs machine learning using the training data selected by the selection means and generates a trained model corresponding to each image in the second image group, A storage means for storing each image in the second group of images linked to the corresponding trained model generated by the learning means, An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, characterized in that the selection means selects an image from the first image group that was taken at the same time for each image in the second image group.

3. The image processing apparatus according to claim 1, characterized in that the selection means selects from the first image group an image for each image in the second image group in which the difference in shooting time is smaller than a predetermined threshold.

4. The image processing apparatus according to claim 3, wherein the selection means, if there is no image in the first image group whose difference in shooting time with the target image in the second image group is less than the predetermined threshold, selects at least one image from the first image group that has the smallest difference in shooting time with the target image.

5. The image processing apparatus according to claim 1, characterized in that the first image group and the second image group are image frames of a video captured simultaneously by the same imaging device, and the frame rate of the first image group is less than or equal to the frame rate of the second image group.

6. The image processing apparatus according to claim 1, characterized in that the first image group and the second image group are image groups of the first resolution and the second resolution, respectively, which were captured simultaneously by the same imaging device.

7. The image processing apparatus according to claim 1, wherein the second group of images is a group of images obtained by resizing each of the first group of images, and the image processing apparatus is characterized in that it stores information about the time of capture of the original image from the first group of images as metadata.

8. An inference device that exchanges data with an image processing device described in claim 1, A first acquisition means for acquiring the image to be re-resolution from the second group of images stored in the storage means, A second acquisition means retrieves a trained model associated with an image acquired by the first acquisition means from the storage means, An inference means inputs the high-resolution target image into the trained model acquired by the second acquisition means and infers the high-frequency components of the high-resolution target image, A generation means that generates a high-resolution image of the image to be refinished based on the high-resolution target image and the high-frequency components inferred by the inference means, An inference device characterized by having the following features.

9. A super-resolution processing system comprising an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components using machine learning, The aforementioned image processing device is An acquisition means for acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images, A selection means for selecting at least one of the first image group as training data for each image in the second image group, A learning means that performs machine learning using the training data selected by the selection means and generates a trained model corresponding to each image in the second image group, A super-resolution processing system characterized by having a storage means for storing each image in the second group of images in association with the corresponding trained model generated by the learning means.

10. The system further comprises an inference device that exchanges data with the aforementioned image processing device, The inference device is A first acquisition means for acquiring the image to be re-resolution from the second group of images stored by the storage means, A second acquisition means retrieves a trained model associated with an image acquired by the first acquisition means from the storage means, An inference means inputs the high-resolution target image into the trained model acquired by the second acquisition means and infers the high-frequency components of the high-resolution target image, A generation means that generates a high-resolution image of the image to be refinished based on the high-resolution target image and the high-frequency components inferred by the inference means, The super-resolution processing system according to claim 9, characterized by having the following features.

11. The super-resolution processing system according to claim 10, characterized in that the storage means is a recording medium that enables the image processing device and the inference device to send and receive data, respectively.

12. The storage means is a recording medium provided by the inference device, The image processing apparatus further comprises a first communication unit, and the inference apparatus further comprises a second communication unit. The super-resolution processing system according to claim 10, characterized in that the images of the second group of images and the corresponding trained models are stored in the storage means via the first communication unit and the second communication unit.

13. The storage means is a recording medium provided by the image processing device, The aforementioned image processing device is A first acquisition means for acquiring the image to be re-resolution from the second group of images stored by the storage means, A second acquisition means retrieves a trained model associated with an image acquired by the first acquisition means from the storage means, An inference means inputs the high-resolution target image into the trained model acquired by the second acquisition means and infers the high-frequency components of the high-resolution target image, A generation means that generates a high-resolution image of the image to be refinished based on the high-resolution target image and the high-frequency components inferred by the inference means, The super-resolution processing system according to claim 9, further comprising the following:

14. A control method for an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components using machine learning, An acquisition step of acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images, A selection step in which at least one of the first image group is selected as training data for each image in the second image group, A learning step involves performing machine learning using the training data selected in the selection step to generate a trained model corresponding to each image in the second image group, A saving step which involves associating each image in the second image group with the corresponding trained model generated in the training step and saving them together, A control method characterized by having the following features.

15. A control method for an inference device that exchanges data with an image processing device described in claim 1, A first acquisition step involves acquiring the image to be re-resolution from the second group of images saved in the saving step, A second acquisition step involves acquiring the trained model associated with the image acquired in the first acquisition step from the storage step, An inference step in which the trained model acquired in the second acquisition step is input to the image to be refinished, and the high-frequency components of the image to be refinished are inferred, A generation step that generates a high-resolution image of the target image based on the high-resolution image and the high-frequency components inferred in the inference step, A control method characterized by having the following features.

16. A control method for a super-resolution processing system comprising an image processing device that analyzes an input image and generates a trained model for inferring high-frequency components using machine learning, In the aforementioned image processing apparatus, An acquisition step of acquiring a first group of images having a first resolution and a second group of images having a second resolution lower than the first resolution and having a certain correlation with the first group of images, A selection step in which at least one of the first image group is selected as training data for each image in the second image group, A learning step involves performing machine learning using the training data selected in the selection step to generate a trained model corresponding to each image in the second image group, A control method characterized by comprising a saving step of associating each image in the second group of images with the corresponding trained model generated in the learning step and saving them together.

17. A program for causing a computer to function as each means of the image processing apparatus described in claim 1.

18. A program for causing a computer to function as each means of the inference apparatus described in claim 8.

19. A program for causing a computer to function as each means of the super-resolution processing system described in claim 9.

Images