DATA PROCESSING APPARATUS, CONTROL METHOD FOR DATA PROCESSING APPARATUS, AND PROGRAM

Abstract

To achieve deterioration restoration processing for an image series having time series represented by a video image in a more suitable mode.SOLUTION: A restoration error calculation unit 1003 calculates a restoration error related to restoration of a high-quality image 103 based on the high-quality image 103 restored from a degraded image 102 obtained by degrading a pre-degradation image 101 and at least either pre-degradation image 101 or degraded image 102. A score calculation unit 1007 calculates, based on the restoration error and a feature amount of a predetermined type extracted from the degraded image 102, a score according to the type of the feature amount. A sampling unit 1008 extracts the pre-deterioration image 101 to be used for creating a data set from a series of pre-deterioration images 101 based on the score.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a data processing device, a control method for a data processing device, and a program. [Background technology]

[0002] Examples of image processing applied to a target image include noise reduction processing that reduces noise that appears in the input image, high-resolution processing that increases the resolution of the input image, and image quality improvement processing such as blur removal processing that removes blur caused by the movement of a subject or focus. In recent years, methods have been proposed that realize these image processing processes using learning models (hereinafter simply referred to as models) constructed based on machine learning. Many of the model-based image quality improvement processing technologies generate degraded images by modeling the process of image quality degradation and simulating the degradation that may become apparent in the pre-degraded image. The degraded image generated in this way is used as input data, and a model is trained by supervised learning using the pre-degraded image as GT (Ground Truth), and the resulting model is then applied to image quality improvement processing.

[0003] Generally, creating GTs for supervised learning in image recognition tasks such as image classification and object detection tends to be costly because users add GTs based on the recognition results for the input image to the image as annotations. On the other hand, creating GTs for image quality improvement tasks can be done annotation-free by using a degradation process model, making it relatively easy to increase the amount of data. However, when training data is randomly selected from available still image and video sources, samples with high learning effectiveness are not necessarily extracted. To address this issue, a method known as HEM (Hard Example Mining) has been proposed as a method for obtaining training data with high learning effectiveness. Non-Patent Document 1 proposes a HEM method for object detection. Non-Patent Document 2 proposes a HEM method for super-resolution. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Abhinav Shrivastava, Abhinav Gupta, Ross Girshick, Training Region-based Object Detectors with Online Hard Example Mining,CVPR2016(2016) [Non-patent document 2] Wang, Shizun, et al. “SamplingAug: On the Importance of Patch Sampling Augmentation for Single Image Super-Resolution.” arXiv preprint arXiv:2111.15185 (2021). Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, the method disclosed in Non-Patent Document 1 assumes a specific structure in which a feature map is estimated for the entire input image and final inference is performed using a portion of that image, so there are limitations regarding the network structure of the target neural network. Furthermore, the method disclosed in Non-Patent Document 2 requires inference of an image quality improvement model for a series of images, which results in excessive computational costs. Furthermore, when dealing with a degradation process that takes into account the optical characteristics of the image capture system, the computational costs required for the simulation are required for each series of images. Furthermore, when considering application to moving images, it is not always possible to express the level of difficulty derived from the movement of moving objects.

[0006] In view of the above problems, an object of the present invention is to provide a more suitable method for restoring degradation of a time-series image sequence, such as a moving image sequence. [Means for solving the problem]

[0007] The data processing device according to the present invention is characterized by comprising: an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and on at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to the type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from the reference image series based on the score calculated by the score calculation means. [Effects of the Invention]

[0008] According to the present invention, it is possible to realize a degradation restoration process for a time-series image sequence, such as a moving image, in a more suitable manner. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 10 is a diagram illustrating an example of a noise reduction process. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 3] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Figure 4] 10 is a flowchart showing an example of processing of the image processing system. [Figure 5] FIG. 10 is a diagram illustrating an example of a process related to extraction of training examples. [Figure 6] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system. [Figure 7] 10 is a flowchart showing an example of processing of the image processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] First Embodiment An image processing system according to a first embodiment of the present disclosure will be described below. For convenience, in this embodiment, various explanations will be given focusing on a case where a noise reduction process (NR) is applied, in which a degraded image degraded by noise is input and an image with reduced noise is generated.

[0012] An example of noise reduction processing will be outlined with reference to FIG. Reference numeral 101 schematically indicates an image before degradation such as noise becomes apparent (hereinafter also referred to as an undegraded image). In the example shown in Fig. 1, the undegraded image 101 is a moving image, and is a frame sequence including a plurality of frame images (still images). In this disclosure, a series of still images (for example, frame images) having a time sequence, such as the frame sequence (for example, moving images) exemplified as the pre-degraded image 101, is also referred to as an image sequence for convenience. Reference numeral 102 schematically indicates an image in which degradation such as noise is apparent (hereinafter also referred to as a degraded image). In the example shown in Fig. 1, the degraded image 102 is also a moving image, and includes frame images in which degradation is apparent for each frame image of the undegraded image 101. Reference numeral 103 schematically indicates an image (hereinafter also referred to as a high-quality image) generated by applying noise reduction (NR) to the degraded image 102. In the example shown in Fig. 1, the high-quality image 103 is also a moving image, and includes frame images generated as a result of applying noise reduction to each frame of the degraded image 102. In this embodiment, noise is mainly assumed to occur in the process of converting photons detected by an image sensor into a digital signal. Noise sources include photon shot noise, readout noise, dark current noise, and quantization error, and these noises are assumed to be modeled. Therefore, a degraded image (e.g., degraded image 102) in which noise is apparent corresponding to an undegraded image (e.g., undegraded image 101) can be artificially generated based on the noise model exemplified above.

[0013] An example of the hardware configuration of information processing devices (for example, a data processing device 1000, a learning device 2000, and an image processing device 3000, which will be described separately below) that make up the image processing system according to this embodiment will be described with reference to Figure 2. Note that, for convenience, the configuration of the data processing device 1000 will be described in the example shown in Figure 2, but a substantially similar configuration can also be applied to the learning device 2000 and the image processing device 3000.

[0014] The data processing device 1000 includes a CPU (Central Processing Unit) 201, a memory 202, an input unit 203, a storage unit 204, an output unit 205, and a communication unit 206. These components of the data processing device 1000 are connected via a bus so as to be able to send and receive information to and from each other.

[0015] The CPU 201 controls the overall operation of the data processing device 1000 and implements functions provided by the data processing device 1000 by loading various programs stored in the storage unit 204 or the like into the memory 202 and executing them. The memory 202 is also used as a storage area for temporarily storing various data, such as a work area for the CPU 201. The memory 202 can be implemented, for example, by a random access memory (RAM). The storage unit 204 is a storage area for storing various programs and various data. The storage unit 204 can be implemented, for example, by an auxiliary storage device represented by a read only memory (ROM) or a hard disk drive (HDD).

[0016] The input unit 203 serves as an input interface that accepts instructions from a user and can be realized by an input device such as a mouse, a keyboard, or a touch panel. The output unit 205 serves as an output interface for presenting various types of information to the user. The configuration of the output unit 205 may be changed as appropriate depending on the information output method. For example, the output unit 205 may be realized by a display device such as a display, and may display the information to be presented as an image in a predetermined display area. As another example, the output unit 205 may be realized by an audio output device such as a speaker, and may output the information to be presented as sound, such as voice or electronic sound. The communication unit 206 serves as a communication interface for connecting the data processing device 1000 to a network such as the Internet or a LAN (Local Area Network), etc. The configuration of the communication unit 206 may be changed as appropriate depending on the type of network to be connected and the communication method to be applied.

[0017] In addition, when the program stored in the storage unit 204 is expanded into the memory 202 and executed by the CPU 201, the functional configuration described below with reference to Figures 3 and 6, etc., and the processing described below with reference to Figures 4 and 7, etc., are realized.

[0018] An example of the functional configuration of various information processing devices that make up the image processing system according to this embodiment will be described with reference to FIG. 3(a) is a diagram showing an example of the functional configuration of a data processing device 1000 according to this embodiment. The data processing device 1000 includes a degradation unit 1001, an image quality improvement unit 1002, a restoration error calculation unit 1003, a selection criterion feature extraction unit 1004, a selection unit 1005, a feature extraction unit 1006, a score calculation unit 1007, and a sampling unit 1008. The data processing device 1000 also includes an image sequence set 1009 and a training dataset 1010 as storage devices. 3(b) is a diagram showing an example of the functional configuration of a learning device 2000 according to this embodiment. The learning device 2000 includes a learning data acquisition unit 2001, a degradation unit 2002, an image quality improvement unit 2003, a loss calculation unit 2004, an update unit 2005, and a display unit 2007. The learning device 2000 also includes a learning dataset 2006 as a storage device. 3C is a diagram showing an example of the functional configuration of the image processing device 3000 according to this embodiment during runtime processing. The image processing device 3000 includes an image acquisition unit 3001, an image quality improvement unit 3002, and a display unit 3003. Details of the components of the data processing device 1000, learning device 2000, and image processing device 3000 listed above will be described separately together with the processes described below with reference to FIG.

[0019] An example of processing by the image processing system according to this embodiment will be described below with reference to Fig. 4. First, an example of processing by the data processing device 1000 according to this embodiment will be described with reference to Fig. 4(a), focusing on processing related to the creation of a dataset used for model training.

[0020] In S1001, the selection unit 1005 uses at least a part of a series of image sequences stored in the image sequence set 1009 to select the feature extraction unit 1006 and the score calculation unit 1007 to be used for model training. The details of the processing of S1001 will be described below with reference to FIG. 4(b). It is assumed that the image sequence set stores a plurality of image sequences each consisting of a plurality of temporally consecutive frame images (in other words, a series of still images having a time sequence). For convenience, in this embodiment, each image sequence is assumed to have an arbitrary frame length and to be composed of a single scene. However, this does not limit the configuration of the image sequence to be processed by the image processing system according to this embodiment. In other words, the image sequence to be processed may include an image sequence with the same frame length that is unrelated to the subject scene. Hereinafter, this image sequence will also be referred to as a scene, as appropriate. A portion of the image sequence included in the image sequence set 1009 is applied as the image sequence (scene) to be processed in S1001.

[0021] In S1101, the degradation unit 1001 acquires a partial image sequence from the image sequence stored in the image sequence set 1009, and executes a process of degrading the image sequence through simulation. In this embodiment, noise is added to the acquired image sequence as the degradation. In the process of S1101, the input image sequence corresponds to the image sequence example shown as undegraded images 101 in the example shown in FIG. 1, and the image sequence corresponds to an example of a "reference image sequence." In addition, the output image sequence corresponds to the image sequence shown as degraded images 102 in the example shown in FIG. 1, and the image sequence corresponds to an example of a "degraded image sequence."

[0022] In S1102, the image quality improvement unit 1002 generates a high-quality image by performing image quality improvement processing on the sequence of degraded images created in S1101. As described above, in this embodiment, noise reduction processing (NR) is applied as the image quality improvement processing. Note that it is desirable that the noise reduction processing applied in S1102 has characteristics closer to those of the noise reduction processing applied by the image processing device 3000 at runtime. However, it is not always possible to use noise reduction processing similar to the processing at runtime. Therefore, in such cases, an appropriate alternative processing (for example, rule-based noise reduction processing that does not use machine learning) may be applied. In this embodiment, an NR model implemented by a neural network is used for runtime processing. Therefore, if an NR model trained using some training data set is available, it may be used for the image quality improvement processing in S1102. The series of high quality images output as a result of the noise reduction process corresponds to the series of images shown as high quality images 103 in the example shown in FIG. 1, and this series of images corresponds to an example of a "series of restored images."

[0023] In S1103, the restoration error calculation unit 1003 calculates the restoration error that occurs between the pre-degraded image and the restored image when the degradation that is apparent in the image is restored from the pre-degraded image series, the degraded image series, the high-quality image series, etc. The restoration error is an error that serves as an index of the learning effect. In this embodiment, a quantitative image quality index such as PSNR (Peak Signal to Noise Ratio) or SSIM (Structural SIMilarity) is used as the restoration error. The PSNR is calculated using the undegraded image and the image with improved image quality, and the average value over time is used. For example, the PSNR is calculated based on the following relational expression:

[0024]

number

[0025] In the above equation, PSNR() represents a function for calculating PSNR, and a commonly known formula is applied. Also, N represents the length of the image sequence, n represents the time index of the image sequence, In(NR) represents the nth high-quality image, and In(GT) represents the nth undegraded image.

[0026] Furthermore, by focusing on the extent of improvement in image quality of a high-quality image obtained by inputting a degraded image, the following ΔPSNR may be used as the restoration error, with the PSNR of the degraded image as the reference.

[0027]

number

[0028] In this case, since the relative improvement in PSNR from the degraded image is evaluated as ΔPSNR, this is effective when creating learning data that is effective in improving the improvement.

[0029] Alternatively, an evaluation index that focuses on the movement of a moving object may be used. As an example of such an evaluation index, the inter-frame difference PSNR expressed by the following relational expression may be used.

[0030]

number

[0031] In the above formula, PSNRmask() is a function for calculating PSNR when areas of the GT inter-frame difference where the value is close to zero (i.e., areas with no movement) are masked and excluded from the calculation. This allows the PSNR for the inter-frame difference image to be calculated only in areas where the inter-frame difference value is finite, and as a result, the reproducibility of the image quality of moving objects is evaluated. It should be noted that ΔPSNR may also be calculated for this inter-frame differential PSNR.

[0032] In addition to PSNR, evaluation values ​​such as SSIM, L2 distance, and L1 distance, or a combination of these evaluation values, may be applied as the restoration error and used as an index of the learning effect.

[0033] In S1104, the selection criterion feature extraction unit 1004 extracts selection criterion features. The selection criterion features correspond to a plurality of feature amounts used as criteria when the selection unit 1005 selects feature amounts indicating the characteristics of the images used to create the dataset in S1002. As a specific example, the movement of moving objects in a moving image is considered to contribute more to the learning effect when improving the image quality of the image, and optical flow, which represents the movement of moving objects as a vector field, may be calculated as one of the selection criteria features. As another example, since textured regions tend to be more difficult to reduce noise in than non-textured regions within an image, a feature representing the presence or absence of texture may be used as one of the selection criteria features. As an example of such a feature, an edge detection result by a Sobel filter or the Canny method, or a spatial frequency analysis result by a two-dimensional FFT may be calculated as one of the selection criterion features. As illustrated above, the selection reference features to be extracted by the selection reference feature extractor 1004 include at least feature amounts according to the spatiotemporal frequency characteristics of a time-series image sequence such as a moving image. It is preferable that the feature amount used as one of the selection reference features is sufficiently lighter (lower calculation cost) than when image quality improvement processing is executed.

[0034] In S1105, the selection unit 1005 selects, from a plurality of candidates, the feature extraction unit 1006 and the score calculation unit 1007 to be applied to the creation of the dataset. The score calculation unit 1007 is a component that scores the feature amount extracted by the feature extraction unit 1006 based on a predetermined criterion. The selection unit 1005 determines a score calculation method corresponding to each of the multiple selection criteria features acquired in S1104, and based on the scores calculated by each score calculation method, selects the feature extraction unit 1006 and score calculation unit 1007 to be applied to creating the dataset.

[0035] Here, an example of a method for calculating a score according to a feature amount will be described. For example, in optical flow, the movement of a subject in two adjacent frames, the previous frame and the current frame, is expressed as a vector field on a two-dimensional plane defined by the vertical (Y-axis) and vertical (X-axis) directions of the image plane. In such a case, for example, to express the magnitude of the movement of a moving object, the sum of the magnitudes of the vectors may be used as the score for the current frame. Alternatively, the score for a scene may be calculated by averaging the scores for each frame in the time direction across the entire image sequence. As another example, when edge detection results using a Sobel filter or the Canny algorithm are used as features, the sum of edges within a frame may be used as the frame score. Alternatively, the average of the scores for each frame may be used as the scene score. As another example, when spatial frequency is used as a feature, the frame score may be a value obtained by integrating high-frequency components in the time direction, or the score of a scene may be an average value of the scores of each frame.

[0036] In this way, the selection unit 1005 calculates the correlation coefficient between the scene score calculated for each selection criterion feature and the restoration error for each scene calculated in S1103, and selects the feature extraction unit 1006 and the score calculation unit 1007 based on the magnitude of the correlation. Among the learning effect indicators, PSNR (including ΔPSNR and inter-frame differential PSNR) and SSIM indicate that the higher the values, the closer the image quality to GT is reproduced. Therefore, the higher the PSNR and SSIM values, the easier the scene is to restore, and the lower the learning effect can be considered. In addition, in the case of the L1 distance and the L2 distance, the higher the value, the greater the distance from the GT. Therefore, the higher the value of the L1 distance or the L2 distance, the more difficult the scene is to restore, and the higher the learning effect can be considered. Due to these characteristics, among the correlation coefficients exemplified above, PSNR and SSIM show a negative correlation with the scene score calculated from the selection criteria features. Furthermore, L1 distance and L2 distance show a positive correlation with the scene score calculated from the selection criteria features. Regardless of which of the correlation coefficients exemplified above is applied, the selection unit 1005 simply selects the feature extraction unit 1006 and score calculation unit 1007 with the higher correlation based on the magnitude (absolute value) of the correlation.

[0037] The details of the process of S1001 shown in Fig. 4(a) have been described above with reference to Fig. 4(b). As a result, the feature extraction unit 1006 and score calculation unit 1007 are selected and applied to creating a dataset using a part of the image sequences stored in the image sequence set 1009.

[0038] Here, reference is made again to FIG. 4(a). In S1002, the feature extraction unit 1006 extracts features from a series of images stored in the image sequence set 1009. The features to be extracted at this time are those according to the feature extraction unit 1006 (feature extraction method) selected in S1001. As described above, examples of features to be extracted include optical flow, edge detection results using a Sobel filter or the Canny method, spatial frequency, etc. Of course, these features are merely examples, and the type of feature to be applied is not limited as long as it can be extracted from an image sequence and its correlation with the restoration error for each scene can be evaluated.

[0039] In S1003, the score calculation unit 1007 calculates two types of scores, a frame score and a scene score, for each of the series of images from which feature amounts have been extracted in S1002. The score calculation method used here depends on the calculation method selected by the score calculation unit 1007 in S1001. For example, when optical flow is used as a feature, the score calculation unit 1007 calculates the frame score by summing up the absolute values ​​of the vectors of the optical flow of each frame acquired in S1002. The score calculation unit 1007 also calculates the average value of the frame scores for a scene as the scene score. Even when a feature other than optical flow is selected, a score for the frame and a score for the scene are calculated using a score calculation method that corresponds to the type of the target feature.

[0040] In S1008, the sampling unit 1008 extracts training examples (in other words, a series of images used to create a dataset) based on the scores calculated in S1007. Here, as an example of a method for extracting training examples, a method for extracting training examples in two stages will be described below. In the first stage, the sampling unit 1008 determines in advance the total number of samples of the image sequence to be extracted, and normalizes the score for each scene so that the score is expressed as a value (e.g., a value between 0.0 and 1.0) indicating a relative proportion to the maximum possible value of the score. Then, the sampling unit 1008 determines the number of samples of the image sequence to be obtained from each scene probabilistically, using the normalized value of the score for each scene as a probability. In the second stage, the sampling unit 1008 performs normalization on the scores for the frames of each scene in the same way as in the first stage. Then, the sampling unit 1008 uses the normalized scores for the frames of each scene as probabilities to probabilistically determine the positions from which to extract frame images in each scene. The training examples that are finally extracted (sampled) as described above are image sequences of a predetermined frame length (for example, 10 frames, 20 frames) that is shorter than the frame length of the target image sequence.

[0041] Here, an example of the first stage of processing of the processing related to the extraction of training examples by the sampling unit 1008 will be described in more detail with reference to FIG. 5(a). Reference numerals 501, 502, 503, and 504 denote scenes (image sequences) stored in the image sequence set 1009, with the scores of each scene being 0.8, 0.9, 0.4, and 0.7, respectively. The sampling unit 1008 uses the normalized values ​​of these scores as probabilities to determine the number of samples of the image sequence to be extracted for each scene probabilistically. In the example shown in FIG. 5(a), the sampling unit 1008 sets the total number of samples to 7 and determines the number of samples of the image sequence to be extracted from scenes 501, 502, 503, and 504 as 2 samples, 2 samples, 1 sample, and 2 samples, respectively. Then, the sampling unit 1008 extracts a sequence of images as samples from the scene based on the number of samples determined for each scene and the position within the scene. For example, in the example shown in Figure 5(a), the sampling unit 1008 extracts a sequence of images of a predetermined frame length as samples from the positions indicated by the dashed rectangles with reference numerals 505 and 506 for scene 501. The sampling unit 1008 also extracts a sequence of images of a predetermined frame length as samples from the positions indicated by the dashed rectangles with reference numerals 507 and 508 for scene 502.

[0042] Next, with reference to FIG. 5(b), an example of the second stage of processing related to the extraction of training examples by the sampling unit 1008 will be described in more detail. Reference numerals 501, 505, and 506 in FIG. 5(b) each represent the same objects as those in FIG. 5(a). Reference numeral 507 represents a graph in which scores calculated for each frame of the scene 501 are plotted in chronological order. The horizontal axis of the graph 507 represents time, and the vertical axis represents the score. As shown in FIG. 5(b), the positions of the image sequence to be extracted as samples are determined probabilistically using the normalized values ​​of the frame scores as described above as probabilities. For example, in the example shown in FIG. 5(b), the normalized values ​​(probabilities) of the frame scores at the positions indicated by reference numerals 505 and 506 (the center positions of the range indicated by the dashed rectangle) are higher than those at other positions.

[0043] Note that the above-mentioned method (sampling method) for extracting a sequence of images from each scene is merely an example, and the method is not particularly limited as long as it is possible to extract a sequence of images from the scene that has a higher correlation with the scene score calculated from the selection criteria features. As a specific example, after the above-described first stage processing is performed, the positions from which an image sequence is extracted from each scene in the second stage processing may be determined by uniform distribution. As another example, the first-stage processing may be omitted and the second-stage processing may be applied across a series of scenes. Specifically, the scores for frames across a series of images may be normalized as described above, and the normalized score values ​​may be used as probabilities to determine positions within a scene from which to extract a series of images as samples, up to a predetermined total number of samples.

[0044] In addition, while the methods exemplified above all involve determining the number of frames of an image sequence to be extracted from a target scene in advance, the number of frames may not be predetermined but may be determined during a series of processes. As a specific example, a method may be applied in which an image sequence is probabilistically extracted from each scene, and when the accumulated total score reaches a predetermined threshold, extraction of the image sequence from that scene is discontinued.

[0045] Furthermore, in the above-described method, there may be cases where at least a portion of the positions (extraction ranges) at which an image sequence is extracted from a target scene overlap, as illustrated in Fig. 5(a) by the reference numerals 507 and 508. In such cases, non-maximum-suppression (NMS) may be applied to the overlapping extraction positions (extraction ranges) to extract a training example (image sequence) with a higher score.

[0046] As described above, a series of training examples (image series) extracted by executing the series of processes shown in FIG. 4(a) are stored in a training dataset 1010. An example of the processing of the data processing device 1000 according to this embodiment has been described above with reference to Figures 4(a) and 4(b), focusing on the processing related to the creation of a dataset used for model training.

[0047] Next, with reference to FIG. 4(c), an example of processing by the learning device 2000 according to this embodiment will be described, focusing particularly on processing related to model learning. In S2001, the learning device 2000 performs various settings related to model learning. In this embodiment, an NR model is applied as a neural network to the image quality improvement unit 2003, and the stochastic gradient descent method is applied to the learning of the NR model. In addition, in the processing of S2001, a mini-batch size (the number of data constituting a mini-batch), a learning coefficient, solver parameters for the stochastic gradient descent method, etc., which will be described later, are set.

[0048] Loop L2001 is a loop related to iterations of the stochastic gradient descent method. A preset value may be applied to the maximum value N of variable n (variable indicating the number of loops) related to the termination condition of the series of processes indicated by loop L2001. As another example, instead of the number of loops, a condition related to loss, which will be described separately later, may be applied as the termination condition of the series of processes indicated by loop L2001. As a specific example, when the loss calculated in S2005 becomes equal to or less than a threshold, control may be applied to terminate the series of processes indicated by loop L2001.

[0049] In S2002, the training data acquisition unit 2001 acquires data to be used for model training from the training dataset 2006. The training dataset 2006 stores an image sequence of a predetermined frame length created by the series of processes described with reference to FIG. 4(a). The image sequence stored in the training dataset 2006 corresponds to the image sequence with extremely little noise (pre-degraded image sequence) shown as pre-degraded images 101 in FIG. 1. In S2002, the training data acquisition unit 2001 acquires image sequences in the number equal to the mini-batch size set in S1001.

[0050] In S2003, the degradation unit 2002 imparts degradation to the pre-degraded image sequence acquired in S2002. In this embodiment, the degradation unit 2002 imparts artificial noise as degradation to the pre-degraded image sequence. As described above, noise is modeled. Therefore, the degradation unit 2002 generates noise from a noise model and then imparts the noise to the pre-degraded image sequence to generate a degraded image sequence in which the noise is apparent.

[0051] In S2004, the image quality improvement unit 2003 performs restoration processing on the degraded image sequence generated in S2003 to restore degradation such as noise that has become apparent in the image sequence as a result of image quality improvement processing. The NR model that performs noise reduction processing is a neural network, and in the processing of S2004, forward propagation processing is performed, with the degraded image sequence as input and the image quality improved image sequence as output. The NR model performs noise reduction processing on moving images, and neural networks that perform image processing on moving images typically fall into the following three types. The first is a sliding window method with a multi-frame input / multi-frame output structure. The second is a recurrent method with a 1-frame input / 1-frame output structure with an internal recurrent structure. The third is a hybrid method of the two, with a recurrent structure with multi-frame input / multi-frame output. In either case, the processing in S2004 uses an image sequence of a specified frame length that makes up a mini-batch, and the appropriate forward propagation processing is performed using one of the three types above.

[0052] In S2005, the loss calculation unit 2004 calculates the loss. In this case, the loss is an L1 loss or an L2 loss, where the undegraded image sequence is used as the GT and the image-enhanced image sequence is used as the prediction. Alternatively, regularization such as total variation regularization may be used. When there are multiple losses, the loss calculation unit 2004 integrates the multiple losses with appropriate weights and determines the result as the final loss.

[0053] In S2006, the update unit 2005 updates the parameters of the image quality improvement unit 2003. In this embodiment, the update unit 2005 determines the amount of update for the parameters of each layer of the NR model by the backpropagation learning method based on the loss calculated in S2005, and updates the parameters of the NR model.

[0054] In S2007, the display unit 2007 displays information related to the learning. As a specific example, it is assumed that scores of a sequence of undegraded images are stored in the learning dataset 2006. In S2007, the display unit 2007 associates each of the sequence of undegraded images, the sequence of degraded images, and the sequence of high-quality images with their scores and displays them in a predetermined display area. In addition, the display unit 2007 may also display scores other than the loss and the selected feature for reference. 5(c) shows an example of information displayed by the display unit 2007. Reference numeral 508 schematically indicates a display area displayed in a part of the display screen. In the display area 508, an undegraded image 509, a degraded image 510, and an improved image quality 511 are displayed. In addition, the display area 508 is provided with a display area 512 in which scores associated with the undegraded image 509, the degraded image 510, and the improved image quality 511, respectively, scores other than the selected feature amount, losses, etc. are displayed.

[0055] 4(c), an example of the processing of the learning device 2000 according to this embodiment has been described, focusing particularly on the processing related to model learning. Through the above-described series of processing, parameters of the image quality improvement unit 3002 to be used at runtime are learned.

[0056] Next, an example of runtime processing of the image processing device 3000 according to this embodiment will be described with reference to FIG. Loop L3001 is a loop related to time in noise reduction processing of moving images. The maximum value T of the time variable t related to the termination condition of the series of processing steps indicated by loop L2001 is set to the frame length of the moving image on which noise reduction processing is performed if the frame length can be set in advance. In cases where the frame length is not set in advance, such as with video streamed in real time, the loop is executed until a separately set termination condition for the processing (for example, a condition such as the end of playback of the streaming video) is met.

[0057] In S3001, the image acquisition unit 3001 acquires an image to be processed. The image acquisition unit 3001 acquires images (for example, still images such as frame images) of a number of frames that conform to the NR model method for performing noise reduction processing, and supplies them to the image quality improvement unit 3002 at the subsequent stage. For example, if the NR model is a sliding window method or a hybrid method with multiple frame inputs, the image acquisition unit 3001 acquires a series of images for multiple frames that match the number of input frames of the model. As another example, in the case of a recurrent method with one frame input, the image acquisition unit 3001 acquires one frame of images. The frame rate of the images acquired in the process of S3001 is assumed to be slower than the throughput of the NR model. The images acquired in the process of S3001 are target data for noise reduction processing and are a degraded image (series).

[0058] In S3002, the image quality improvement unit 3002 performs noise reduction processing on the degraded image (series) acquired in S3001 to create an improved image (series). As mentioned above, the NR model applied to the noise reduction processing can be either a multiple frame input / multiple frame output or a single frame input / single frame output as an input / output format depending on the method. Therefore, the image quality improvement unit 3002 controls so that noise reduction processing is performed for the number of frames appropriate for the target method.

[0059] In S3003, the display unit 3003 displays the high quality image created in S3002 in a predetermined display area. When the NR model is a sliding window model, multiple frames are output in one inference process. In such a case, the display unit 3003 may buffer and display the frames so that each frame is switched at equal intervals. Furthermore, the display unit 3003 controls the frame rate for displaying the high quality image so that it is the same as the frame rate of the image acquired in S3001.

[0060] Above, with reference to Figure 4(d), we have explained an example of runtime processing of the image processing device 3000 according to this embodiment, focusing on noise reduction processing using the image quality improvement unit 2003 trained by the learning device 2000.

[0061] Note that the configurations of the data processing device 1000, the learning device 2000, and the image processing device 3000 described with reference to FIG. 3 are merely examples, and the configuration of each device is not limited as long as the functions of the components of each device can be realized. For example, at least one of the data processing device 1000, the learning device 2000, and the image processing device 3000 may be realized by multiple devices or services working together. As a specific example, among the components of the data processing device 1000, the functions of the degradation unit 1001, the image quality improvement unit 1002, and the restoration error calculation unit 1003 may be realized by an external device different from the data processing device 1000. As another example, two or more of the data processing device 1000, the learning device 2000, and the image processing device 3000 may be realized by a single device. As a specific example, the functions of the data processing device 1000 and the learning device 2000 may be realized by a single device.

[0062] As described above, the image processing system according to this embodiment collects training examples based on the movement of moving objects, which affects the difficulty of restoration processing such as improving the image quality of moving images, making it possible to construct a training dataset with high learning efficiency. Therefore, in restoration processing for a series of images such as moving images, it is possible to learn good restoration processing regardless of the movement of moving objects, and it is possible to execute restoration processing such as improving the image quality using the trained model.

[0063] <Modification of the first embodiment> In the first embodiment described above, an example of a case where noise reduction processing is applied was described as an example of degradation restoration processing applied to the image quality improvement section, but this does not limit the scope of application of this embodiment. For example, super-resolution processing may be applied as a degradation restoration process. Even when super-resolution is applied to moving images, alignment becomes important in situations where there is movement of moving objects, and therefore the movement of moving objects is thought to affect the difficulty of learning (the greater the movement, the more difficult it is). Therefore, by applying the technology according to this embodiment and constructing a learning dataset based on the movement of moving objects, it is expected that the effect of improving image restoration accuracy in super-resolution of moving images can be achieved.

[0064] <Second embodiment> A second embodiment of the present disclosure will be described below. In the first embodiment described above, multiple types of feature amounts were extracted as selection reference features, and feature amounts and score calculation methods used to create a dataset were selected based on the correlation between the scores of each feature amount and the reconstruction error. On the other hand, in order to obtain a score calculation unit suitable for sampling a dataset with a high learning effect, it is also possible to use a model trained to directly predict reconstruction errors from feature amounts, without relying on a selection method. In this embodiment, an example of a method using such a model will be described.

[0065] 6(a) is a diagram showing an example of the functional configuration of a data processing device according to this embodiment. The data processing device 4000 includes a degradation unit 4001, an image quality improvement unit 4002, a restoration error calculation unit 4003, a feature extraction unit 4004, a score prediction model training unit 4005, and a sampling unit 4007. The data processing device 4000 also includes an image sequence set 4008 and a training dataset 4009 as storage devices. The details of the components of the data processing device 4000 will be described separately below with reference to FIGS. 7(a) and 7(b) along with the processing to be described later.

[0066] 7(a), an example of processing of the data processing device 4000 according to this embodiment will be described, focusing on processing related to creation of a dataset used for model training. Note that the following description will mainly focus on parts that are different from the first embodiment, and detailed description of parts that are substantially the same as those in the first embodiment will be omitted.

[0067] In S4001, the score prediction model learning unit 4005 uses at least a portion of the image series stored in the image series set 4008 to learn a score prediction model that predicts a score from features extracted from the image series. The details of the processing in S4001 will be described below with reference to FIG. 7(b).

[0068] In S4101, the degradation unit 4001 acquires a part of a series of images from the series of images stored in the image sequence set 4008, and executes a process of degrading the part of the series of images. Note that this process is substantially the same as the process of S1101 shown in Fig. 4(b), and therefore a detailed description thereof will be omitted. Similarly, the processes of S4102 and 4103 are substantially the same as the processes of S1102 and S1103 shown in Fig. 4(b). In this case, the image quality improvement unit 4002 and the restoration error calculation unit 4003 correspond to the image quality improvement unit 1002 and the restoration error calculation unit 1003 shown in Fig. 3(a). As a result, an enhanced image is generated from a sequence of degraded images, and restoration errors are calculated from the sequence of undegraded images, the sequence of degraded images, the sequence of enhanced image images, etc.

[0069] In S4104, the feature extraction unit 4004 extracts feature amounts (selection reference features) from the target image series, similar to the process in S1104 shown in FIG. 4(b). In the first embodiment, optical flow between adjacent frames was calculated and edge detection was performed in each frame, and a scalar score for the frame was calculated by summing the calculated features within the frame. In the process of S4104, the feature extraction unit 4004 calculates a multidimensional score for the frame by calculating the sum of the calculated feature amounts, as well as the average, variance, histogramming, etc. The feature extraction unit 4004 also calculates a multidimensional score from multiple types of feature amounts, such as optical flow and edge detection. The feature extraction unit 4004 concatenates the calculated multidimensional scores to obtain a multidimensional feature amount for each frame.

[0070] In S4105, the score prediction model learning unit 4005 learns a score prediction model. Specifically, the score prediction model learning unit 4005 learns a model that predicts a recovery error (objective variable), which is an index of the learning effect, based on features (explanatory variables). The multidimensional features extracted in S4104 are applied as the features, and an index such as PSNR calculated in S4103 is applied as the recovery error. The model to be applied may be a regression model such as polynomial multiple regression, or a neural network such as a multi-layer perceptron. In either case, the model should be optimized so that the difference between the output of the prediction model and the objective variable is minimized.

[0071] The details of the process of S4001 shown in Fig. 7(a) have been described above with reference to Fig. 7(b). Note that the process of S4001 is executed using a portion of the data stored in the image sequence set 4008. In contrast, the subsequent processes are executed on the entire series of data stored in the image sequence set 4008.

[0072] Here, reference is made again to FIG. In S4002, the feature extraction unit 4004 extracts multidimensional features for each frame of a series of images stored in the image sequence set 4008. Note that the content of the process related to the extraction of multidimensional features from the image sequence is substantially similar to the process of S4104 shown in Fig. 7(b), and therefore a detailed description thereof will be omitted.

[0073] In S4003, the score calculation unit 4006 calculates a score for each of the image sequences stored in the image sequence set 4008. Specifically, the score calculation unit 4006 inputs the multidimensional features for each frame extracted in S4002 into the score prediction model trained in S4105, and obtains a score for each frame as the output of the score prediction model. The above score is calculated for each frame, but depending on the processing of the sampling unit 4007 at the subsequent stage, the scores may be averaged over multiple frames in the image sequence and used as the score for the scene.

[0074] In S4004, the sampling unit 4007 samples training examples from a series of image sequences stored in the image sequence set 4008. Note that the processing of S4004 is substantially the same as the processing of S1004 shown in FIG. 4(a) except for the score used. Specifically, the score used in the processing of S4004 is the predicted value of the learning effect index output by the prediction model in S4003. Apart from this difference in score, the processing of S4004 is substantially the same as the processing of S1004, and therefore a detailed description thereof will be omitted.

[0075] An example of the functional configuration and processing of the data processing device 4000 according to this embodiment has been described above with reference to FIGS. 6(a), 7(a), and 7(b). The details of model learning and runtime processing are substantially the same as those in the first embodiment, so detailed explanations will be omitted.

[0076] As described above, the image processing system according to this embodiment makes it possible to predict a learning effect index using lightweight features such as optical flow and edge detection, and to construct a training dataset with high learning efficiency. This makes it possible to learn good degradation restoration processing regardless of the movement of moving objects in degradation restoration processing for image sequences such as moving images, and to use the trained model to execute degradation restoration processing such as image quality improvement processing.

[0077] <Third embodiment> A third embodiment of the present disclosure will be described below. In the first and second embodiments described above, an example was described in which a training dataset is created prior to training of an NR model that performs noise reduction processing. In such a case, if the image quality improvement unit used in the data processing device and the image quality improvement unit used in the training device differ significantly during the training process, it may be desirable to recreate the training dataset. In light of this situation, in this embodiment, an example is described in which a score prediction model is learned during the learning of an NR model, and the sampling of learning examples is performed dynamically using a score prediction model that is updated as needed.

[0078] An example of the functional configuration of the learning device according to this embodiment will be described with reference to Fig. 6(b). The learning device 5000 includes a learning data acquisition unit 5001, a feature extraction unit 5002, a score calculation unit 5003, a filtering unit 5004, a degradation unit 5005, an image quality improvement unit 5006, a loss calculation unit 5007, an update unit 5008, a recovery error calculation unit 5009, and a score prediction model learning unit 5010. The learning device 5000 also includes a learning dataset 5011 as a storage device. The details of the components of the learning device 5000 will be described separately below with reference to FIG. 7(c) along with the processing to be described later.

[0079] 7(c), an example of processing by the learning device 5000 according to this embodiment will be described, focusing particularly on processing related to model learning. Note that the following description will mainly focus on differences from the first and second embodiments, and detailed descriptions of portions that are substantially the same as those of the first and second embodiments will be omitted.

[0080] In S5001, the learning device 5000 performs various settings related to model learning. In this embodiment, a score prediction model is used in S5004, and initial values ​​of the score prediction model are set in S5001. Although the initial values ​​are preferably trained parameters, it is possible to train the target model even if they are random initial values. In this embodiment, two types of batches with different sizes are used. One is a mini-batch of the stochastic gradient descent method, and the other is a batch larger in size than the mini-batch. In the process of S5005, the image sequence is filtered based on the score (i.e., a partial image sequence whose score satisfies a predetermined condition is extracted from a series of images that serve as a parameter). In this filtering, the batch before filtering corresponds to a batch larger in size than the above-mentioned mini-batch, and the batch after filtering corresponds to the above-mentioned mini-batch. Therefore, for convenience, the size of the batch (mini-batch) after filtering (the size indicating the number of data items) will be referred to as the mini-batch size, and the size of the batch before filtering will be referred to as the pre-filtering size. Other setting items are substantially the same as those described in conjunction with the processing of S2001 in FIG. 4(c) in the first embodiment, and therefore detailed description thereof will be omitted.

[0081] Loop L5001 is a loop related to iterations of the stochastic gradient descent method. Loop L5001 is substantially similar to loop L2001 shown in FIG. 4(c), so detailed description thereof will be omitted.

[0082] In S5002, the training data acquisition unit 5001 acquires an image sequence stored in a training dataset 5011. The training dataset 5011 stores an image sequence aligned to an appropriate frame length for training an NR model that improves the image quality of moving images. The image sequence stored in the training dataset 5011 corresponds to an image sequence with extremely little noise (pre-degraded image sequence) shown as pre-degraded images 101 in FIG. 1. In S5002, the training data acquisition unit 5001 acquires, from the training dataset 5011, an image sequence the number of which corresponds to the pre-filtering size set in S5001.

[0083] In S5003, the feature extraction unit 5002 extracts features (multidimensional features) from the image series acquired in S5002. Note that the processing in S5003 is substantially the same as the processing in S4002 shown in Fig. 7(a) in the second embodiment, except that the length of the target image series is a predetermined uniform length, and therefore a detailed description thereof will be omitted. In S5004, the score calculation unit 5003 predicts a score for each frame from the multidimensional feature amounts of the image sequence extracted in S5003. The processing of S5004 is also substantially similar to the processing of S4003 shown in Fig. 7(a) in the second embodiment, except that the sequence length of the target image sequence is a predetermined uniform length, and therefore a detailed description thereof will be omitted.

[0084] In S5005, the filtering unit 5004 performs filtering based on the scores of the image sequences calculated in S5004, so that a number of image sequences equal to the mini-batch size are extracted from the number of image sequences equal to the pre-filtering size obtained in S5002. For example, since a score is predicted for each frame of the image sequence in S5004, the filtering unit 5004 may extract a number of image sequences equal to the mini-batch size in descending order of scene score, obtained by summing or averaging the scores for each frame across all frames. As another example, the filtering unit 5004 may normalize the scene score so that it is expressed as a value indicating a relative ratio to the maximum possible value of the score, and then randomly extract a number of image sequences equal to the mini-batch size, using the normalized value as a probability. In this manner, filtering is performed so that a sequence of images equal in number to the mini-batch size is extracted from the sequence of images acquired in S5002.

[0085] The processes of S5006, S5007, S5008, and S5009 are substantially the same as the processes of S2003, S2004, S2005, and S2006 shown in Fig. 4(c), and therefore detailed description thereof will be omitted. In this case, the degradation unit 5005, image quality improvement unit 5006, loss calculation unit 5007, and update unit 5008 correspond to the degradation unit 2002, image quality improvement unit 2003, loss calculation unit 2004, and update unit 2005 shown in Fig. 3(b).

[0086] In B5001, the score prediction model training unit 5010 determines whether to perform score prediction learning, which will be described later in S5010, based on a predetermined condition. As a specific example, the score prediction model training unit 5010 may determine to perform score prediction learning when a fixed period of one or more epochs has elapsed, and proceed to S5010. As another example, the score prediction model training unit 5010 may determine to perform score prediction learning when the calculated loss has dropped by a fixed amount from the initial or previous loss value, and proceed to S5010. In particular, when the latter determination is applied, learning of a score prediction model that adaptively follows changes in the model during the training process is realized. Note that, if the score prediction model training unit 5010 determines not to perform score prediction learning, it skips the processing of S5010.

[0087] The processing of S5010 is substantially similar to the processing of S4001 shown in Fig. 7(a), and therefore a detailed description thereof will be omitted. In this case, the degradation unit 5005, the image quality improvement unit 5006, the restoration error calculation unit 5009, and the score prediction model training unit 5010 correspond to the degradation unit 4001, the image quality improvement unit 4002, the restoration error calculation unit 4003, and the score prediction model training unit 4005 shown in Fig. 6(a). When the processing of S5010 is performed, a part of an image sequence of a predetermined frame length stored in a training dataset 5011 is used as the image sequence.

[0088] The model (for example, the NR model) learned in the above manner is used in the processing at runtime. Note that the processing at runtime is substantially the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0089] As described above, the image processing system according to this embodiment makes it possible to extract training examples with high learning effectiveness that are adapted to a model (for example, an NR model) that is updated during the learning process.

[0090] <Modification of the third embodiment> In the third embodiment described above, an example of filtering an image sequence based on the score was described. On the other hand, in the method described in the third embodiment, when filtering an image sequence, feature extraction and score prediction are performed on an image sequence with a number equal to or greater than the mini-batch size. Therefore, depending on the number of image sequences that serve as parameters when filtering an image sequence, the computational cost may be higher. To reduce this computational cost, the image sequence may not be filtered, and the scores of the image sequence may be used as weights when calculating the loss. Applying such control is expected to reduce the impact of training examples with low learning effectiveness on learning and increase the impact of training examples with high learning effectiveness.

[0091] In the third embodiment described above, the feature values ​​for the image sequence are calculated in the process of S5003 in the loop of the series of processes related to learning shown in Fig. 7(c). However, since the feature values ​​are independent of the NR model and the score prediction model, they can also be calculated in advance. Therefore, multidimensional features may be calculated in advance for a series of images stored in the training dataset 5011 and stored in association with the image series. By applying such control, it becomes possible to omit the process of extracting features shown as the process of S5003 in FIG. 7(c), which is expected to reduce the computational cost during model training. In this method, prediction is performed using a score prediction model in the process of S5004 shown in FIG. 7(c), but even if the scale of the prediction model is sufficiently small, the effect can be expected. Therefore, even in a form in which filtering is performed, applying such control is expected to suppress increases in computational cost.

[0092] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0093] The disclosure of this embodiment also includes the following configurations, methods, and programs. (Configuration 1) A data processing device characterized by comprising: an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to the type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from the reference image series based on the score calculated by the score calculation means. (Configuration 2) A data processing device according to Configuration 1, comprising: a selection reference feature extraction means for extracting a plurality of types of image features as a plurality of selection reference features from the restored image sequence; and a selection means for selecting, based on the restoration error calculated by the error calculation means and the plurality of selection reference features extracted by the selection reference feature extraction means, a feature extraction method to be applied to the calculation of the score by the score calculation means and a method of calculating the score based on the feature from a plurality of types of candidate extraction methods and a plurality of types of candidate calculation methods, wherein the plurality of selection reference features include at least a feature according to the characteristics of the spatiotemporal frequency of the image sequence. (Configuration 3) The data processing device according to configuration 2, wherein the plurality of selection reference features include a feature related to the movement of a subject as a feature corresponding to the spatiotemporal frequency characteristics of the image sequence. (Configuration 4) The data processing device according to configuration 3, wherein the feature amount relating to the movement of the subject is an optical flow. (Configuration 5) The data processing device according to any one of configurations 2 to 4, characterized in that the selection means selects a feature extraction method and a score calculation method that have a higher correlation between the score calculated from each of the plurality of selection reference features and the recovery error. (Configuration 6) The data processing device according to Configuration 1, wherein the score calculation means inputs the feature quantities of the predetermined type extracted from the degraded image sequence into a prediction model trained based on machine learning to predict the restoration error using the feature quantities extracted from the reference image sequence as input, and applies the restoration error output from the prediction model as the score. (Configuration 7) The data processing device according to Configuration 6, characterized in that the score calculation means inputs multidimensional features composed of multiple features including at least features corresponding to the spatiotemporal frequency characteristics of the image series to the prediction model, and applies the multidimensional recovery errors output from the prediction model as multidimensional scores. (Configuration 8) The data processing device according to configuration 7, wherein the prediction model is a regression model in which the multidimensional feature amount is an explanatory variable and the restoration error is a target variable. (Configuration 9) The data processing device according to configuration 7, wherein the prediction model is a neural network trained using the multidimensional feature as an input and the restoration error as a correct answer. (Configuration 10) The data processing device according to any one of configurations 1 to 9, wherein the restoration error is a PSNR or SSIM calculated from the restored image sequence and the degraded image sequence. (Configuration 11) The data processing device according to any one of configurations 1 to 9, characterized in that the restoration error is a PSNR or SSIM of an inter-frame difference calculated from the restored image sequence and the degraded image sequence. (Configuration 12) A data processing device according to any one of configurations 1 to 11, characterized in that the image series extraction means normalizes the score so that it is expressed as a value indicating a relative proportion to a maximum value, and then extracts a reference image series to be used for creating a dataset from the series of reference image series based on the probability indicated by that value. (Configuration 13) An image processing device having an image processing means that has been trained based on machine learning to generate and output a restored image series by performing a process on an image series received as input to restore degradation that has become apparent in the image series, wherein the dataset used for training the image processing means is generated based on a reference image series extracted from a series of the reference image series, based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and on a restoration error related to the restoration of the restored image series calculated based on at least one image series of the reference image series and the degraded image series, and on a score corresponding to the type of feature calculated based on a predetermined type of feature extracted from the degraded image series. (Configuration 14) The image processing device described in Configuration 13, characterized in that the image processing means performs learning based on the loss between the reference image series included in the dataset generated in advance and the restored image series restored from the degraded image series generated by applying degradation to the reference image series. (Configuration 15) The image processing device described in Configuration 14 is characterized in that the degraded image series is generated by degrading at least a predetermined number of reference image series extracted from the reference image series based on the score predicted from multidimensional features composed of multiple features including features corresponding to the spatiotemporal frequency characteristics of the image series extracted from the reference image series, and the image processing means learns based on the loss between the extracted at least a portion of the reference image series and the restored image series restored from the degraded image series generated from the reference image series. (Configuration 16) The image processing device described in Configuration 15, characterized in that the score is obtained by inputting features extracted from the reference image series into a prediction model trained based on machine learning to predict a restoration error related to the restoration of the restored image series based on the restored image series and at least one of the reference image series and the degraded image series, and applying the restoration error output from the prediction model, and the prediction model is trained when predetermined conditions are satisfied during the learning process of the image processing means. (Configuration 17) The image processing device according to Configuration 16, characterized in that the prediction model is trained when the change in the loss becomes equal to or greater than a threshold during the learning process of the image processing means. (Configuration 18) The image processing device described in Configuration 16, characterized in that the predictive model is learned when a predetermined number of iterations related to learning have elapsed during the learning process of the image processing means. (Configuration 19) An image processing device according to any one of configurations 15 to 18, characterized in that the reference image series included in the data set is associated with the multidimensional features extracted from the reference image series. (Method 1) A control method for a data processing device, comprising: 1. A method for controlling a data processing device, comprising: an error calculation step of calculating a restoration error associated with the restoration of a restored image series, based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and on at least one of the reference image series and the degraded image series; a score calculation step of calculating a score according to a type of feature, based on the restoration error calculated in the error calculation step and a predetermined type of feature extracted from the degraded image series; and an image series extraction step of extracting a reference image series to be used for creating a dataset from the reference image series, based on the score calculated in the score calculation step. (Program 1) A program for causing a computer to function as a data processing device, characterized by having an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and on at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to the type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from the reference image series based on the score calculated by the score calculation means. [Explanation of symbols]

[0094] 1000 Data Processing Device 1003 Restoration error calculation unit 1007 Score calculation unit 1008 Sampling section

Claims

1. an error calculation means for calculating a restoration error related to the restoration of a restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation means for calculating a score according to a type of a feature amount based on the restoration error calculated by the error calculation means and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction means for extracting a reference image sequence to be used for creating a dataset from the reference image sequence based on the score calculated by the score calculation means; A data processing device comprising:

2. a selection reference feature extracting means for extracting feature quantities of a plurality of types of images from the restored image sequence as a plurality of selection reference features; a selection means for selecting, based on the recovery error calculated by the error calculation means and the plurality of selection reference features extracted by the selection reference feature extraction means, a feature extraction method to be applied to the calculation of the score by the score calculation means and a score calculation method based on the feature from a plurality of types of candidate extraction methods and a plurality of types of candidate calculation methods that are specified in advance; and The plurality of selection reference features include at least a feature quantity according to the spatiotemporal frequency characteristics of the image sequence.

2. A data processing device according to claim 1.

3. 3. The data processing device according to claim 2, wherein the plurality of selection reference features include a feature quantity relating to the movement of a subject as a feature quantity according to the spatiotemporal frequency characteristics of the image sequence.

4. 4. The data processing device according to claim 3, wherein the feature amount relating to the movement of the subject is an optical flow.

5. 3. The data processing device according to claim 2, wherein the selection means selects a feature extraction method and a score calculation method that have a higher correlation between the score calculated from each of the plurality of selection reference features and the recovery error.

6. 2. The data processing device according to claim 1, wherein the score calculation means inputs the feature quantities of the predetermined type extracted from the degraded image sequence into a prediction model trained based on machine learning so as to predict the restoration error using the feature quantities extracted from the reference image sequence as input, and applies the restoration error output from the prediction model as the score.

7. 7. The data processing device according to claim 6, wherein the score calculation means inputs a multidimensional feature consisting of a plurality of feature amounts including at least a feature amount according to spatiotemporal frequency characteristics of an image sequence to the prediction model, and applies the multidimensional recovery errors output from the prediction model as multidimensional scores.

8. 8. The data processing apparatus according to claim 7, wherein the prediction model is a regression model in which the multidimensional feature amounts are explanatory variables and the restoration error is a response variable.

9. 8. The data processing apparatus according to claim 7, wherein the prediction model is a neural network trained using the multidimensional feature vectors as input and the reconstruction error as a correct answer.

10. 2. The data processing apparatus according to claim 1, wherein the restoration error is PSNR or SSIM calculated from the restored image sequence and the degraded image sequence.

11. 2. The data processing apparatus according to claim 1, wherein the restoration error is a PSNR or SSIM of an inter-frame difference calculated from the restored image sequence and the degraded image sequence.

12. 2. The data processing device according to claim 1, wherein the image sequence extraction means normalizes the scores so that they are expressed as values ​​indicating relative proportions to a maximum value, and then extracts a reference image sequence to be used for creating a data set from the series of reference image sequences based on the probability indicated by the normalized scores.

13. an image processing means that has been trained based on machine learning to generate and output a restored image sequence by performing a process for restoring degradation that has become apparent in an image sequence received as an input, The data set used for training the image processing means is a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence, and a restoration error related to the restoration of the restored image sequence calculated based on at least one of the reference image sequence and the degraded image sequence; A predetermined type of feature amount extracted from the sequence of degraded images; Based on the score according to the type of the feature amount calculated based on generated based on a reference image sequence extracted from the reference image sequence An image processing device comprising:

14. The image processing means the reference image sequence contained in the pre-generated dataset; a restored image sequence restored from the degraded image sequence generated by degrading the reference image sequence; Learning is done based on the loss between The image processing device according to claim 13 ,

15. the degraded image sequence is generated by degrading at least a predetermined number of reference images extracted from the reference image sequence, based on the score predicted from a multidimensional feature amount composed of a plurality of feature amounts including at least a feature amount corresponding to spatiotemporal frequency characteristics of the image sequence extracted from the reference image sequence; The image processing means performs learning based on a loss between the extracted at least part of the reference image sequence and the restored image sequence restored from the degraded image sequence generated from the reference image sequence.

15. The image processing device according to claim 14,

16. the score is determined by inputting feature quantities extracted from the reference image series into a prediction model trained by machine learning so as to predict a restoration error related to restoration of the restored image series based on the restored image series and at least one image series of the reference image series and the degraded image series, and applying the restoration error output from the prediction model; The prediction model is trained when a predetermined condition is satisfied during the training process of the image processing means.

16. The image processing device according to claim 15,

17. 17. The image processing device according to claim 16, wherein the prediction model is trained when the change in the loss becomes equal to or greater than a threshold value during the training process of the image processing means.

18. 17. The image processing device according to claim 16, wherein the prediction model is learned when a predetermined number of iterations related to learning have elapsed in the learning process of the image processing means.

19. The image processing apparatus according to claim 15, wherein the multidimensional feature values ​​extracted from the reference image series included in the data set are associated with the reference image series.

20. A method for controlling a data processing device, comprising: an error calculation step of calculating a restoration error related to the restoration of a restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation step of calculating a score according to a type of a feature amount based on the restoration error calculated in the error calculation step and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction step of extracting a reference image sequence to be used for creating a dataset from the reference image sequence based on the score calculated in the score calculation step; 10. A method for controlling a data processing device, comprising:

21. Computer, an error calculation means for calculating a restoration error related to the restoration of a restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation means for calculating a score according to a type of a feature amount based on the restoration error calculated by the error calculation means and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction means for extracting a reference image sequence to be used for creating a dataset from the reference image sequence based on the score calculated by the score calculation means; A program for causing a data processing device to function, comprising:

Images