Data processing device, method for controlling the same, and program

Abstract

To achieve deterioration restoration processing for an image series having time series represented by a video image in a more suitable mode.SOLUTION: A restoration error calculation unit 1003 calculates a restoration error related to restoration of a high-quality image 103 based on the high-quality image 103 restored from a degraded image 102 obtained by degrading a pre-degradation image 101 and at least either pre-degradation image 101 or degraded image 102. A score calculation unit 1007 calculates, based on the restoration error and a feature amount of a predetermined type extracted from the degraded image 102, a score according to the type of the feature amount. A sampling unit 1008 extracts the pre-deterioration image 101 to be used for creating a data set from a series of pre-deterioration images 101 based on the score.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a data processing device, a control method for a data processing device, and a program. [Background technology]

[0002] Examples of image processing applied to a target image include noise reduction processing for reducing noise manifested in an input image, resolution processing for increasing the resolution of an input image, and image quality improvement processing such as blur removal processing for removing blur caused by the movement of a subject or focus. In recent years, a method has been proposed for implementing these image processing processes using a learning model (hereinafter also simply referred to as a model) constructed based on machine learning. Many of the image quality improvement processing technologies that use models generate degraded images by modeling the process of image quality degradation and simulating the degradation that may become apparent in the image before degradation. The degraded image generated in this way is used as input data, and a model is trained by supervised learning with the image before degradation as GT (Ground Truth), and the resulting model is applied to image quality improvement processing.

[0003] In general, the creation of GTs used in supervised learning in image recognition tasks such as image classification and object detection from an image tends to be very costly because the user adds a GT according to the recognition result for the input image to the image as an annotation. On the other hand, the creation of GTs in image quality improvement tasks can be done annotation-free by using a model of the degradation process, so it is relatively easy to increase the amount of data. However, when learning data is randomly selected from available still image and video image sources, samples with high learning effectiveness are not necessarily extracted. In response to such issues, a method called HEM (Hard Example Mining) has been proposed as a method for acquiring learning data with higher learning effectiveness. Non-Patent Document 1 proposes a HEM method for object detection. In addition, Non-Patent Document 2 proposes a HEM method for super-resolution. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Abhinav Shrivastava, Abhinav Gupta, Ross Girshick, Training Region-based Object Detectors with Online Hard Example Mining,CVPR2016(2016) [Non-Patent Document 2] Wang, Shizun, et al. “SamplingAug: On the Importance of Patch Sampling Augmentation for Single Image Super-Resolution.” arXiv preprint arXiv:2111.15185 (2021). Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, the method disclosed in Non-Patent Document 1 assumes a specific structure in which a feature map is estimated for the entire input image and final inference is performed on a part of it, so there are limitations regarding the network structure of the target neural network. In addition, the method disclosed in Non-Patent Document 2 requires inference of a high-image-quality model for a series of images, which results in excessive computational costs. When dealing with a degradation process that takes into account the optical characteristics of an image capturing system, the computational costs required for the simulation are required for a series of images. In addition, when considering application to moving images, it is not always possible to express the degree of difficulty derived from the movement of a moving object.

[0006] In view of the above problems, an object of the present invention is to provide a more suitable method for realizing degradation restoration processing for a time-sequential image sequence, such as a moving image. [Means for solving the problem]

[0007] The data processing device according to the present invention is characterized in having an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series and on at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to a type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from the reference image series based on the score calculated by the score calculation means. Effect of the Invention

[0008] According to the present invention, it is possible to realize, in a more suitable manner, degradation restoration processing targeted at a time-sequential image sequence, such as a moving image. [Brief description of the drawings]

[0009] [Figure 1]FIG. 13 is a diagram illustrating an example of a noise reduction process. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] FIG. 1 illustrates an example of a functional configuration of an image processing system. [Figure 4] 11 is a flowchart showing an example of processing of the image processing system. [Diagram 5] FIG. 13 is a diagram showing an example of a process related to extraction of training examples. [Figure 6] FIG. 1 illustrates an example of a functional configuration of an image processing system. [Figure 7] 11 is a flowchart showing an example of processing of the image processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] <First embodiment> An image processing system according to a first embodiment of the present disclosure will be described below. For convenience, in this embodiment, various descriptions will be given focusing on a case where a noise reduction process (NR: Noise Reduction) is applied to generate an image with reduced noise by inputting a degraded image degraded by noise.

[0012] An example of noise reduction processing will be outlined with reference to FIG. Reference numeral 101 typically indicates an image before degradation such as noise becomes apparent (hereinafter also referred to as an undegraded image). In the example shown in Fig. 1, the undegraded image 101 is a moving image, and is a frame sequence including a plurality of frame images (still images). In this disclosure, a series of still images (eg, frame images) having a time sequence, such as the frame sequence (eg, video image) exemplified as the undegraded image 101, is also referred to as an image sequence for convenience. Reference numeral 102 typically indicates an image in which degradation such as noise is evident (hereinafter, also referred to as a degraded image). In the example shown in Fig. 1, the degraded image 102 is also a moving image, and includes frame images in which degradation is evident for each frame image of the undegraded image 101. Reference numeral 103 typically indicates an image (hereinafter also referred to as a high-image-quality image) generated by applying noise reduction processing (NR) to the degraded image 102. In the example shown in Fig. 1, the high-image-quality image 103 is also a moving image, and includes frame images generated as a result of applying noise reduction processing to each frame of the degraded image 102. In this embodiment, the noise is mainly assumed to be generated in the process of converting photons detected by an image sensor into a digital signal. Noise sources include photon shot noise, read noise, dark current noise, and quantization error, and these noises are assumed to be modeled. Therefore, it is assumed that a degraded image (e.g., degraded image 102) in which noise is evident corresponding to an undegraded image (e.g., undegraded image 101) can be artificially generated based on the noise model exemplified above.

[0013] An example of the hardware configuration of information processing devices (for example, a data processing device 1000, a learning device 2000, and an image processing device 3000, which will be described later) constituting the image processing system according to this embodiment will be described with reference to Fig. 2. Note that, in the example shown in Fig. 2, for convenience, the configuration of the data processing device 1000 will be described, but substantially the same configuration can be applied to the learning device 2000 and the image processing device 3000.

[0014] The data processing device 1000 includes a CPU (Central Processing Unit) 201, a memory 202, an input unit 203, a storage unit 204, an output unit 205, and a communication unit 206. These components of the data processing device 1000 are connected via a bus so as to be able to transmit and receive information to and from each other.

[0015] The CPU 201 controls the operation of the entire data processing device 1000 and realizes functions provided by the data processing device 1000 by expanding various programs stored in the storage unit 204 or the like in the memory 202 and executing them. The memory 202 is also used as a storage area for temporarily storing various data, such as a working area for the CPU 201. The memory 202 may be realized by, for example, a random access memory (RAM) or the like. The storage unit 204 is a storage area for storing various programs and various data. The storage unit 204 may be realized by, for example, an auxiliary storage device represented by a read only memory (ROM) or a hard disk drive (HDD).

[0016] The input unit 203 serves as an input interface for receiving instructions from a user, and may be realized by an input device such as a mouse, a keyboard, or a touch panel. The output unit 205 serves as an output interface for presenting various information to the user. The configuration of the output unit 205 may be appropriately changed depending on the information output method. For example, the output unit 205 may be realized by a display device such as a display, and may display the information to be presented as an image in a predetermined display area. As another example, the output unit 205 may be realized by an audio output device such as a speaker, and may output the information to be presented as sound such as voice or electronic sound. The communication unit 206 serves as a communication interface for connecting the data processing device 1000 to a network such as the Internet or a LAN (Local Area Network), etc. The configuration of the communication unit 206 may be appropriately changed depending on the type of the network to be connected and the communication method to be applied.

[0017] In addition, the program stored in the storage unit 204 is expanded into the memory 202 and executed by the CPU 201, thereby realizing the functional configuration described later with reference to Figures 3 and 6, etc., and the processing described later with reference to Figures 4 and 7, etc.

[0018] An example of the functional configuration of various information processing devices constituting the image processing system according to this embodiment will be described with reference to FIG. 3(a) is a diagram showing an example of the functional configuration of a data processing device 1000 according to this embodiment. The data processing device 1000 includes a degradation unit 1001, an image quality improvement unit 1002, a restoration error calculation unit 1003, a selection criterion feature extraction unit 1004, a selection unit 1005, a feature extraction unit 1006, a score calculation unit 1007, and a sampling unit 1008. The data processing device 1000 also includes an image sequence set 1009 and a learning dataset 1010 as storage devices. 3(b) is a diagram showing an example of the functional configuration of a learning device 2000 according to this embodiment. The learning device 2000 includes a learning data acquisition unit 2001, a degradation unit 2002, an image quality improvement unit 2003, a loss calculation unit 2004, an update unit 2005, and a display unit 2007. The learning device 2000 also includes a learning dataset 2006 as a storage device. 3C is a diagram showing an example of a functional configuration for runtime processing of the image processing device 3000 according to this embodiment. The image processing device 3000 has an image acquisition unit 3001, an image quality improvement unit 3002, and a display unit 3003. Details of the components of the data processing device 1000, learning device 2000, and image processing device 3000 listed above will be described separately together with the processes described below with reference to FIG.

[0019] An example of processing of the image processing system according to this embodiment will be described below with reference to Fig. 4. First, an example of processing of the data processing device 1000 according to this embodiment will be described with reference to Fig. 4(a), focusing on processing related to the creation of a data set used for model learning.

[0020] In S1001, the selection unit 1005 uses at least a part of a series of image sequences stored in the image sequence set 1009 to select the feature extraction unit 1006 and the score calculation unit 1007 to be used for model training. The details of the processing of S1001 will be described below with reference to FIG. 4(b). Note that the image sequence set stores a plurality of image sequences composed of images of a plurality of frames that are consecutive in time (in other words, a series of still images having a time sequence). In addition, in this embodiment, for convenience, each image sequence has an arbitrary frame length and is composed of a single scene. However, this provision does not limit the configuration of the image sequence to be processed by the image processing system according to this embodiment. In other words, the image sequence to be processed may include an image sequence that is unrelated to the scene of the subject and has the same frame length. Note that, hereinafter, this image sequence will also be referred to as a scene as appropriate. A part of the image sequence included in the image sequence set 1009 is applied as the image sequence (scene) to be processed in S1001.

[0021] In S1101, the degradation unit 1001 acquires a part of an image series among a series of image series stored in the image series set 1009, and executes a process of degrading the image series by simulation. In this embodiment, it is assumed that noise is added to the acquired image series as the degradation. In the process of S1101, the input image series corresponds to the image series example shown as undegraded images 101 in the example shown in FIG. 1, and the image series corresponds to an example of a "reference image series". In addition, the output image series corresponds to the image series shown as degraded images 102 in the example shown in FIG. 1, and the image series corresponds to an example of a "degraded image series".

[0022] In S1102, the image quality improving unit 1002 generates an image quality improved image by performing image quality improvement processing on the degraded image sequence created in S1101. As described above, in this embodiment, it is assumed that noise reduction processing (NR) is applied as the image quality improvement processing. Note that it is desirable that the noise reduction processing applied in S1102 has characteristics closer to those of the noise reduction processing applied by the image processing device 3000 at runtime. However, it is not always possible to use a noise reduction processing similar to the processing at runtime. Therefore, in such a case, an appropriate alternative processing (for example, a rule-based noise reduction processing that does not use machine learning, etc.) may be applied. In this embodiment, an NR model realized by a neural network is used for the runtime process. Therefore, if an NR model trained using a certain training data set is available, it may be used for the image quality improvement process in S1102. The image quality enhanced image series output as a result of the noise reduction process corresponds to the image series shown as image quality enhanced images 103 in the example shown in FIG. 1, and this image series corresponds to an example of a "restored image series".

[0023] In S1103, the restoration error calculation unit 1003 calculates a restoration error that occurs between an undegraded image and a restored image when degradation that is apparent in the image is restored, from a series of undegraded images, a series of degraded images, a series of high-image-quality images, etc. The restoration error is an error that serves as an index of the learning effect. In this embodiment, a quantitative image quality index such as PSNR (Peak Signal to Noise Ratio) or SSIM (Structural SIMilarity) is used as the restoration error. The PSNR is calculated using the undegraded image and the image quality improved image, and the average value with respect to time is used. For example, the PSNR is calculated based on the following relational expression.

[0024]

number

[0025] In the above formula, PSNR() represents a function for calculating PSNR, and a commonly used formula is applied. In addition, N represents the length of the image sequence, n represents the time index of the image sequence, In(NR) represents the nth high quality image, and In(GT) represents the nth undegraded image.

[0026] Furthermore, by focusing on the degree of improvement in image quality of a high-image-quality image obtained by inputting a degraded image, the following ΔPSNR based on the PSNR of the degraded image may be used as the restoration error.

[0027]

number

[0028] In this case, since the relative improvement in PSNR from the degraded image is evaluated as ΔPSNR, this is effective in creating learning data that is effective in improving the improvement.

[0029] Alternatively, an evaluation index that focuses on the movement of a moving object may be used. As an example of such an evaluation index, the inter-frame difference PSNR expressed by the following relational expression may be used.

[0030]

number

[0031] In the above formula, PSNRmask() is a function for calculating PSNR when areas in the GT frame difference where the value is close to zero (i.e. areas with no motion) are masked and excluded from the calculation. This allows the PSNR for the frame difference image to be calculated only in areas where the frame difference value is finite, and as a result, the reproducibility of the image quality of moving objects is evaluated. It should be noted that a ΔPSNR may also be calculated for this inter-frame differential PSNR.

[0032] In addition to PSNR, evaluation values ​​such as SSIM, L2 distance, and L1 distance, or a combination of these evaluation values, may be applied as the restoration error and used as an index of the learning effect.

[0033] In S1104, the selection criterion feature extraction unit 1004 extracts selection criterion features. The selection criterion features correspond to a plurality of feature amounts used as criteria when the selection unit 1005 selects feature amounts indicating the features of the images used to create the data set in S1002. As a specific example, since it is believed that the movement of moving objects in a video image contributes more to the learning effect when improving the image quality of the image, optical flow, which represents the movement of moving objects as a vector field, may be calculated as one of the selection criterion features. As another example, since textured regions tend to be more difficult to reduce noise in than non-textured regions in an image, a feature representing the presence or absence of texture may be used as one of the selection criteria features. As an example of such a feature amount, the result of edge detection using a Sobel filter or the Canny method, or the result of spatial frequency analysis using a two-dimensional FFT may be calculated as one of the selection criterion features. As exemplified above, the selection reference features to be extracted by the selection reference feature extractor 1004 include at least features according to the spatiotemporal frequency characteristics of a time-series image sequence such as a video image. It should be noted that the feature amount used as one of the selection reference features should be one that is sufficiently lighter (has lower calculation cost) than the case where image quality improvement processing is executed.

[0034] In S1105, the selection unit 1005 selects, from a plurality of candidates, the feature extraction unit 1006 and the score calculation unit 1007 to be applied to the creation of the data set. The score calculation unit 1007 is a component that converts the feature amount extracted by the feature extraction unit 1006 into a score based on a predetermined criterion. The selection unit 1005 determines a score calculation method corresponding to each of the multiple selection criteria features acquired in S1104, and selects the feature extraction unit 1006 and score calculation unit 1007 to be applied to creating a dataset based on the scores calculated by each score calculation method.

[0035] Here, an example of a method for calculating a score according to a feature amount will be described. For example, in optical flow, the motion of a subject in two adjacent frames, the previous frame and the current frame, is expressed as a vector field on a two-dimensional plane of the vertical direction (Y axis) and the perpendicular direction (X axis) of the image plane. In such a case, for example, in order to express the magnitude of the motion of a moving object, the sum of the magnitudes of the vectors may be used as the score of the current frame. Also, the score of the scene may be calculated by averaging the scores of each frame in the time direction over the entire image sequence. As another example, when edge detection results using a Sobel filter or the Canny method are applied as features, the sum of edges in a frame may be used as the frame score. Also, the average value of the scores of each frame in a scene may be used as the scene score. As another example, when spatial frequency is used as a feature, the frame score may be a value obtained by integrating high frequency components in the time direction, or the scene score may be an average value of the scores of each frame in a scene.

[0036] In this manner, the selection unit 1005 calculates a correlation coefficient between the scene score calculated for each selection criterion feature and the restoration error for each scene calculated in S1103, and selects the feature extraction unit 1006 and the score calculation unit 1007 based on the magnitude of the correlation. Among the learning effect indices, PSNR (including ΔPSNR and frame difference PSNR) and SSIM indicate that the higher the value, the closer the image quality to GT is reproduced. Therefore, the higher the PSNR or SSIM value, the easier the scene is to restore, and the lower the learning effect can be considered. In addition, in the case of the L1 distance and the L2 distance, the higher the value, the greater the distance from the GT. Therefore, it can be considered that the higher the value of the L1 distance or the L2 distance, the more difficult the scene is to restore, and the higher the learning effect. Due to such characteristics, among the correlation coefficients exemplified above, PSNR and SSIM show a negative correlation with the scene score calculated from the selection criterion features. Also, L1 distance and L2 distance show a positive correlation with the scene score calculated from the selection criterion features. Note that, regardless of which of the correlation coefficients exemplified above is applied, the selection unit 1005 may select the feature extraction unit 1006 and score calculation unit 1007 with a higher correlation based on the magnitude (absolute value) of the correlation.

[0037] The details of the process of S1001 shown in Fig. 4(a) have been described above with reference to Fig. 4(b). As a result, the feature extraction unit 1006 and the score calculation unit 1007, which are applied to creating a dataset using a part of the image sequences stored in the image sequence set 1009, are in a selected state.

[0038] Here, reference is made again to FIG. In S1002, the feature extraction unit 1006 extracts features from a series of image sequences stored in the image sequence set 1009. The features to be extracted at this time correspond to the feature extraction unit 1006 (feature extraction method) selected in S1001. As described above, examples of features to be extracted include optical flow, edge detection results using a Sobel filter or the Canny method, spatial frequency, etc. Of course, these features are merely examples, and the type of feature to be applied is not limited as long as it can be extracted from an image sequence and the correlation with the restoration error for each scene can be evaluated.

[0039] In S1003, the score calculation unit 1007 calculates two types of scores, a frame score and a scene score, for each of the series of images from which feature amounts have been extracted in S1002. The score calculation method used here is the calculation method selected by the score calculation unit 1007 in S1001. For example, when optical flow is applied as a feature, the score calculation unit 1007 calculates the frame score by summing up the absolute values ​​of the vectors of the optical flow of each frame acquired in S1002. The score calculation unit 1007 also calculates the average value of the frame scores for a scene as the scene score. Even if a feature other than optical flow is selected, the frame-related score and the scene-related score are calculated using a score calculation method according to the type of the target feature.

[0040] In S1008, the sampling unit 1008 extracts training examples (in other words, an image sequence used to create a data set) based on the score calculated in S1007. Here, as an example of a method for extracting training examples, a method for extracting training examples in two stages will be described below. In the first stage, the sampling unit 1008 determines in advance the total number of samples of the image sequence to be extracted, and normalizes the score for each scene so that the score is expressed as a value indicating a relative ratio to the maximum value that the score can take (for example, a value between 0.0 and 1.0).The sampling unit 1008 then determines the number of samples of the image sequence to be obtained from each scene probabilistically, using the normalized value of the score for each scene as a probability. Next, in the second stage, the sampling unit 1008 performs normalization on the scores related to the frames of each scene in the same manner as in the first stage. Then, the sampling unit 1008 determines the positions from which to extract frame images in each scene probabilistically, using the normalized values ​​of the scores related to the frames of each scene as probabilities. The training examples finally extracted (sampled) as a result of the above are image sequences of a predetermined frame length (for example, 10 frames, 20 frames) that are shorter than the frame length of the target image sequence.

[0041] Here, an example of the first stage of processing of the processing relating to the extraction of training examples by the sampling unit 1008 will be described in more detail with reference to FIG. 5(a). Reference numerals 501, 502, 503, and 504 denote scenes (image sequences) stored in the image sequence set 1009, and the scores of the scenes are 0.8, 0.9, 0.4, and 0.7, respectively. The sampling unit 1008 determines the number of samples of the image sequence to be extracted as samples for each scene probabilistically, using the normalized values ​​of these scores as described above as probabilities. In the example shown in FIG. 5(a), the sampling unit 1008 sets the total number of samples to 7, and determines the number of samples of the image sequence to be extracted from the scenes 501, 502, 503, and 504 as 2 samples, 2 samples, 1 sample, and 2 samples, respectively. Then, the sampling unit 1008 extracts an image sequence as a sample from the scene based on the number of samples determined for each scene and the position in the scene. For example, in the example shown in Fig. 5(a), the sampling unit 1008 extracts an image sequence of a predetermined frame length as a sample from the positions indicated by the dashed rectangles with reference numbers 505 and 506 for scene 501. The sampling unit 1008 also extracts an image sequence of a predetermined frame length as a sample from the positions indicated by the dashed rectangles with reference numbers 507 and 508 for scene 502.

[0042] Next, referring to FIG. 5(b), an example of the second stage processing of the processing related to the extraction of learning examples by the sampling unit 1008 will be described in more detail. Reference numerals 501, 505, and 506 shown in FIG. 5(b) indicate objects with the same reference numerals in FIG. 5(a). Reference numeral 507 indicates a graph in which the scores calculated for each frame of the scene 501 are plotted in time series. The horizontal axis of the graph 507 indicates time, and the vertical axis indicates the score. As shown in FIG. 5(b), the positions of the image sequence to be extracted as samples are determined probabilistically, with the normalized values ​​of the scores for the frames as described above being set as probabilities. For example, in the example shown in FIG. 5(b), the normalized values ​​(probabilities) of the scores for the frames are higher at the positions indicated by reference numerals 505 and 506 (the center positions of the ranges indicated by the dashed rectangles) than at other positions.

[0043] Note that the above-mentioned method of extracting an image sequence from each scene (sampling method) is merely one example, and the method is not particularly limited as long as it is possible to extract an image sequence from the scene that has a higher correlation with the scene score calculated from the selection criteria features. As a specific example, after the above-mentioned first stage processing is performed, the positions for extracting an image sequence from each scene in the second stage processing may be determined by uniform distribution. As another example, the first stage process may not be performed and the second stage process may be applied across a series of scenes. Specifically, the scores for frames across a series of image sequences may be normalized as described above, and the normalized score values ​​may be used as probabilities to determine positions within the scenes from which the image sequences are extracted as samples for a predetermined total number of samples.

[0044] In addition, although the above-mentioned methods are all methods in which the number of frames of an image sequence to be extracted from a target scene is determined in advance, the number of frames may not be determined in advance but may be determined during a series of processes. As a specific example, a method may be applied in which an image sequence is probabilistically extracted from each scene, and when the accumulated total score number reaches a predetermined threshold, the extraction of the image sequence from that scene is discontinued.

[0045] In addition, in the above-described method, there may be cases where at least a portion of the positions (extraction ranges) at which an image sequence is extracted from a target scene overlap, as illustrated in Fig. 5(a) with reference numerals 507 and 508. In such cases, non-maximum-suppression (NMS) may be applied to a plurality of overlapping extraction positions (extraction ranges) to extract a training example (image sequence) with a higher score.

[0046] In the manner described above, a series of training examples (image sequence) extracted by executing the series of processes shown in FIG. 4(a) is stored in a training dataset 1010. An example of the processing of the data processing device 1000 according to this embodiment has been described above with reference to Figures 4(a) and 4(b), focusing on the processing related to the creation of a dataset used for model training.

[0047] Next, an example of the processing of the learning device 2000 according to this embodiment will be described with reference to FIG. 4(c), focusing in particular on the processing related to model learning. In S2001, the learning device 2000 performs various settings related to model learning. In this embodiment, an NR model is applied as a neural network to the image quality improving unit 2003, and a stochastic gradient descent method is applied to the learning of the NR model. In addition, in the process of S2001, a mini-batch size (the number of data constituting a mini-batch), a learning coefficient, and parameters of a solver of the stochastic gradient descent method, which will be described later, are set.

[0048] The loop L2001 is a loop related to iterations of the stochastic gradient descent method. A preset value may be applied to the maximum value N of the variable n (variable indicating the number of loops) related to the end condition of the series of processes indicated by the loop L2001. As another example, instead of the number of loops, a condition related to loss, which will be described later, may be applied as the end condition of the series of processes indicated by the loop L2001. As a specific example, when the loss calculated in S2005 becomes equal to or less than a threshold, control may be applied to end the series of processes indicated by the loop L2001.

[0049] In S2002, the learning data acquisition unit 2001 acquires data used for model training from the learning dataset 2006. The learning dataset 2006 stores an image sequence of a predetermined frame length created by the series of processes described with reference to FIG. 4(a). The image sequence stored in the learning dataset 2006 corresponds to the image sequence (undegraded image sequence) with extremely low noise shown as the undegraded images 101 in FIG. 1. In S2002, the learning data acquisition unit 2001 acquires image sequences the number of which corresponds to the mini-batch size set in S1001.

[0050] In S2003, the degradation unit 2002 imparts degradation to the pre-degraded image sequence acquired in S2002. In this embodiment, the degradation unit 2002 imparts artificial noise as degradation to the pre-degraded image sequence. As described above, noise is modeled. Therefore, the degradation unit 2002 generates noise from a noise model and imparts the noise to the pre-degraded image sequence to generate a degraded image sequence in which the noise is apparent.

[0051] In S2004, the image quality improvement unit 2003 performs restoration processing on the degraded image sequence generated in S2003 to restore degradation such as noise that has become evident in the image sequence as a result of the image quality improvement processing. The NR model that performs noise reduction processing is a neural network, and in the processing of S2004, forward propagation processing is performed in which the degraded image sequence is input and the image quality improvement image sequence is output. The NR model performs noise reduction processing on moving images, and there are three typical types of neural networks that perform image processing on moving images. The first is a sliding window method with a multiple frame input / multiple frame output structure. The second is a recurrent method with a 1 frame input / 1 frame output structure with an internal recurrent structure. The third is a hybrid method of the two, with a recurrent structure with multiple frame inputs / multiple frame outputs. In any case, the processing in S2004 uses an image sequence of a specified frame length that constitutes a mini-batch, and the appropriate forward propagation processing is performed using one of the three types above.

[0052] In S2005, the loss calculation unit 2004 calculates the loss. In this case, the loss is L1 loss or L2 loss, in which the undegraded image sequence is used as the GT and the image-enhanced image sequence is used as the prediction. Alternatively, regularization such as total variation regularization may be used. When there are multiple losses, the loss calculation unit 2004 integrates the multiple losses with an appropriate weight and uses the result as the final loss.

[0053] In S2006, the update unit 2005 updates the parameters of the image quality improving unit 2003. In this embodiment, the update unit 2005 determines the amount of update for the parameters of each layer of the NR model by the backpropagation learning method based on the loss calculated in S2005, and updates the parameters of the NR model.

[0054] In S2007, the display unit 2007 displays information related to the learning. As a specific example, it is assumed that the scores of the undegraded image sequence are stored in the learning dataset 2006. In S2007, the display unit 2007 displays the scores of the undegraded image sequence, the degraded image sequence, and the image quality improved image sequence in a predetermined display area in association with each other. In addition, the display unit 2007 may also display scores other than the loss and the selected feature for reference. 5(c) shows an example of information displayed by the display unit 2007. Reference numeral 508 diagrammatically indicates a display area displayed in a part of the display screen. In the display area 508, an undegraded image 509, a degraded image 510, and an improved image quality 511 are displayed. In addition, the display area 508 is provided with a display area 512 in which scores associated with the undegraded image 509, the degraded image 510, and the improved image quality 511, respectively, scores other than the selected feature amount, losses, etc. are displayed.

[0055] An example of the processing of the learning device 2000 according to the present embodiment has been described above with reference to Fig. 4(c), focusing in particular on the processing related to model learning. Through the above-described series of processing, parameters of the image quality improvement unit 3002 to be used at runtime are learned.

[0056] Next, an example of a process during runtime of the image processing device 3000 according to this embodiment will be described with reference to FIG. Loop L3001 is a loop related to time in noise reduction processing of a moving image. With regard to the maximum value T of the time variable t related to the end condition of a series of processing indicated by loop L2001, if the frame length of the moving image on which noise reduction processing is performed can be set in advance, the frame length is set. If the frame length is not set in advance, such as in the case of video streamed in real time, the loop is executed until a separately set end condition of the processing (for example, a condition such as the end of playback of the streaming video) is satisfied.

[0057] In S3001, the image acquisition unit 3001 acquires an image to be processed. The image acquisition unit 3001 acquires images (e.g., still images such as frame images) of a number of frames that conform to the NR model method for performing noise reduction processing, and supplies them to the downstream image quality improvement unit 3002. For example, when the NR model is a sliding window method or a hybrid method with multiple frame input, the image acquisition unit 3001 acquires an image sequence of multiple frames that matches the number of input frames of the model. As another example, when the NR model is a recurrent method with one frame input, the image acquisition unit 3001 acquires an image of one frame. The frame rate of the image acquired in the process of S3001 is assumed to be slower than the throughput of the NR model. The image acquired in the process of S3001 is target data for noise reduction processing, and is a degraded image (series).

[0058] In S3002, the image quality improvement unit 3002 creates an image quality improvement image (series) by performing noise reduction processing on the degraded image (series) acquired in S3001. As described above, the NR model applied to the noise reduction processing has input / output forms of multiple frames input / multiple frames output and one frame input / one frame output depending on the method. Therefore, the image quality improvement unit 3002 controls so that noise reduction processing is performed for the number of frames that match the target method.

[0059] In S3003, the display unit 3003 displays the high quality image created in S3002 in a predetermined display area. When the NR model is a sliding window method, multiple frames are output in one inference process. In such a case, the display unit 3003 may buffer and display the frames so that the frames are switched at equal intervals. Furthermore, the display unit 3003 controls the frame rate for displaying the high quality image so that it is the same as the frame rate of the image acquired in S3001.

[0060] Above, with reference to FIG. 4(d), we have focused on noise reduction processing using the image quality improvement unit 2003 trained by the learning device 2000 as an example of runtime processing of the image processing device 3000 according to this embodiment.

[0061] Note that the configurations of the data processing device 1000, the learning device 2000, and the image processing device 3000 described with reference to FIG. 3 are merely examples, and the configurations of each device are not limited as long as the functions of the components of each device can be realized. For example, at least one of the data processing device 1000, the learning device 2000, and the image processing device 3000 may be realized by cooperation of multiple devices or services. As a specific example, among the series of components of the data processing device 1000, the functions of the degradation unit 1001, the image improvement unit 1002, and the restoration error calculation unit 1003 may be realized by an external device different from the data processing device 1000. In addition, as another example, two or more devices among the data processing device 1000, the learning device 2000, and the image processing device 3000 may be realized by one device. As a specific example, each function of the data processing device 1000 and the learning device 2000 may be realized by one device.

[0062] As described above, according to the image processing system of this embodiment, since learning examples are collected based on the movement of moving objects that affect the difficulty of degradation restoration processing such as image quality improvement for moving images, it is possible to construct a learning dataset with high learning efficiency. Therefore, in degradation restoration processing for an image series such as a moving image, it is possible to learn good degradation restoration processing regardless of the movement of moving objects, and it is possible to execute degradation restoration processing such as image quality improvement processing using the learned model.

[0063] <Modification of the first embodiment> In the first embodiment described above, an example of a case where noise reduction processing is applied was described as an example of degradation restoration processing applied to the image quality improvement section, but this does not limit the scope of application of this embodiment. For example, a super-resolution process may be applied as the degradation restoration process. Even when super-resolution is applied to a moving image, alignment is important in a situation where a moving object is moving, so it is considered that the movement of the moving object affects the difficulty of learning (the more movement there is, the more difficult it is). Therefore, by applying the technology according to this embodiment and constructing a learning dataset based on the movement of the moving object, it is expected to have the effect of improving the image restoration accuracy in super-resolution of a moving image.

[0064] <Second embodiment> A second embodiment of the present disclosure will be described below. In the first embodiment described above, a plurality of types of features are extracted as selection reference features, and the features and score calculation method used to create a data set are selected based on the correlation between the score of each feature and the restoration error. On the other hand, in order to obtain a score calculation unit suitable for sampling a data set with a high learning effect, it is also possible to use a model trained to directly predict the restoration error from the features, without relying on the selection method. In this embodiment, an example of a method using such a model will be described.

[0065] 6(a) is a diagram showing an example of the functional configuration of a data processing device according to this embodiment. The data processing device 4000 includes a degradation unit 4001, an image quality improvement unit 4002, a restoration error calculation unit 4003, a feature extraction unit 4004, a score prediction model learning unit 4005, and a sampling unit 4007. The data processing device 4000 also includes an image sequence set 4008 and a learning dataset 4009 as storage devices. Details of the components of the data processing device 4000 described above will be described separately together with the processing described below with reference to FIGS. 7(a) and 7(b).

[0066] 7(a), an example of processing of the data processing device 4000 according to this embodiment will be described with a focus on processing related to the creation of a data set used for model learning. Note that hereinafter, the description will be mainly focused on the parts different from the first embodiment, and detailed description of the parts substantially similar to the first embodiment will be omitted.

[0067] In S4001, the score prediction model learning unit 4005 uses at least a portion of a series of image sequences stored in the image sequence set 4008 to learn a score prediction model that predicts a score from features extracted from the image sequence. The process of S4001 will be described in detail below with reference to FIG.

[0068] In S4101, the degradation unit 4001 acquires a part of an image sequence among a series of image sequences stored in the image sequence set 4008, and executes a process of degrading the image sequence. Note that this process is substantially similar to the process of S1101 shown in Fig. 4(b), and therefore a detailed description thereof will be omitted. Similarly, as the processing of S4102 and 4103, processing substantially similar to the processing of S1102 and S1103 shown in Fig. 4(b) is executed. In this case, the image quality improvement unit 4002 and the restoration error calculation unit 4003 correspond to the image quality improvement unit 1002 and the restoration error calculation unit 1003 shown in Fig. 3(a). In this way, after an image quality improvement image is generated from a sequence of degraded images, restoration errors are calculated from the sequence of undegraded images, the sequence of degraded images, the sequence of enhanced images, etc.

[0069] In S4104, the feature extraction unit 4004 extracts feature amounts (selection criterion features) from the target image sequence, similar to the process in S1104 shown in FIG. In the first embodiment, optical flow between adjacent frames is calculated and edges are detected in each frame, and a scalar score for the frame is calculated by summing the calculated features within the frame. In the process of S4104, the feature extraction unit 4004 calculates a multidimensional score for the frame by calculating the sum of the calculated feature amounts, as well as the average, variance, histogramming, etc. The feature extraction unit 4004 also calculates a multidimensional score from multiple types of feature amounts, such as optical flow and edge detection. The feature extraction unit 4004 connects the calculated multidimensional scores to obtain a multidimensional feature amount for each frame.

[0070] In S4105, the score prediction model learning unit 4005 learns a score prediction model. Specifically, the score prediction model learning unit 4005 learns a model that predicts a recovery error (objective variable) that is an index of a learning effect based on features (explanatory variables). As the features, the multidimensional features extracted in S4104 are applied, and as the recovery error, an index such as PSNR calculated in S4103 is applied. The model to be applied may be a regression model such as polynomial multiple regression, or a neural network such as a multi-layer perceptron. In either case, it is sufficient to optimize the model so that the difference between the output of the prediction model and the objective variable is minimized.

[0071] The process of S4001 shown in Fig. 7(a) has been described in detail above with reference to Fig. 7(b). Note that the process of S4001 is executed using a part of the data stored in the image sequence set 4008. In contrast, the subsequent processes are executed on a series of data stored in the image sequence set 4008.

[0072] Here, reference is made again to FIG. In S4002, the feature extraction unit 4004 extracts multidimensional features for each frame from each of a series of images stored in the image sequence set 4008. Note that the contents of the process relating to the extraction of multidimensional features from an image sequence are substantially similar to the process of S4104 shown in Fig. 7(b), and therefore a detailed description thereof will be omitted.

[0073] In S4003, the score calculation unit 4006 calculates a score for each of a series of image sequences stored in the image sequence set 4008. Specifically, the score calculation unit 4006 inputs the multi-dimensional features for each frame extracted in S4002 into the score prediction model trained in S4105, and obtains a score for each frame as the output of the score prediction model. The above score is calculated for each frame, but depending on the processing of the sampling unit 4007 at the subsequent stage, the scores may be averaged over multiple frames in the image sequence to be used as the score for the scene.

[0074] In S4004, the sampling unit 4007 samples training examples from a series of image sequences stored in the image sequence set 4008. The processing of S4004 is substantially similar to the processing of S1004 shown in FIG. 4(a) except for the difference in the score used. Specifically, the score used in the processing of S4004 is the predicted value of the learning effect index output by the prediction model in S4003. Apart from this difference in score, the processing of S4004 is substantially similar to the processing of S1004, and therefore a detailed description thereof will be omitted.

[0075] An example of the functional configuration and processing of the data processing device 4000 according to this embodiment has been described above with reference to FIGS. 6(a), 7(a), and 7(b). The details of model learning and runtime processing are substantially similar to those in the first embodiment, so detailed description thereof will be omitted.

[0076] As described above, the image processing system according to the present embodiment makes it possible to predict a learning effect index using lightweight features such as optical flow and edge detection, and to construct a learning dataset with high learning efficiency. This makes it possible to learn good degradation restoration processing regardless of the movement of moving objects in degradation restoration processing for a sequence of images such as moving images, and makes it possible to execute degradation restoration processing such as image quality improvement processing using a trained model.

[0077] <Third embodiment> The third embodiment of the present disclosure will be described below. In the above-mentioned first and second embodiments, an example of a case where a learning data set is created prior to learning of an NR model that performs noise reduction processing has been described. In such a case, if the image quality improvement unit used in the data processing device and the image quality improvement unit used in the learning device are significantly different during the learning process, it may be desirable to recreate the learning data set. In consideration of this situation, in this embodiment, an example is described in which a score prediction model is trained during training of an NR model, and sampling of training examples is dynamically performed using a score prediction model that is updated from time to time.

[0078] An example of the functional configuration of the learning device according to this embodiment will be described with reference to Fig. 6(b). The learning device 5000 includes a learning data acquisition unit 5001, a feature extraction unit 5002, a score calculation unit 5003, a filtering unit 5004, a degradation unit 5005, an image quality improvement unit 5006, a loss calculation unit 5007, an update unit 5008, a recovery error calculation unit 5009, and a score prediction model learning unit 5010. The learning device 5000 also includes a learning dataset 5011 as a storage device. The components of the learning device 5000 described above will be described in detail separately together with the processing described later with reference to FIG.

[0079] 7(c), an example of the processing of the learning device 5000 according to this embodiment will be described, focusing in particular on the processing related to model learning. Note that hereinafter, the description will be mainly focused on the parts different from the first and second embodiments, and detailed description of the parts substantially similar to the first and second embodiments will be omitted.

[0080] In S5001, the learning device 5000 performs various settings related to model learning. In this embodiment, a score prediction model is used in S5004, and an initial value of the score prediction model is set in S5001. Although it is desirable for the initial value to be a learned parameter, it is possible to learn the target model even if the initial value is a random value. In addition, in this embodiment, two types of batches with different sizes are used. One is a mini-batch of the stochastic gradient descent method, and the other is a batch with a size larger than the mini-batch. In the process of S5005, the image sequence is filtered based on the score (i.e., a part of the image sequence whose score satisfies a predetermined condition is extracted from the series of image sequences that are the parameter). In this filtering, the batch before filtering corresponds to a batch with a size larger than the above-mentioned mini-batch, and the batch after filtering corresponds to the above-mentioned mini-batch. Therefore, hereinafter, for convenience, the size of the batch (mini-batch) after filtering (the size indicating the number of data) is also referred to as the mini-batch size, and the size of the batch before filtering is also referred to as the pre-filtering size. Other setting items are substantially the same as those described in conjunction with the process of S2001 in FIG. 4(c) in the first embodiment, and therefore detailed description thereof will be omitted.

[0081] The loop L5001 is a loop related to the iteration of the stochastic gradient descent method. The loop L5001 is substantially similar to the loop L2001 shown in FIG. 4(c), and therefore a detailed description thereof will be omitted.

[0082] In S5002, the learning data acquisition unit 5001 acquires an image sequence stored in a learning dataset 5011. The learning dataset 5011 stores an image sequence aligned to an appropriate frame length for learning an NR model that improves the image quality of a moving image. The image sequence stored in the learning dataset 5011 corresponds to an image sequence (undegraded image sequence) with extremely little noise shown as undegraded images 101 in FIG. 1. In S5002, the learning data acquisition unit 5001 acquires an image sequence from the learning dataset 5011, the number of which corresponds to the pre-filtering size set in S5001.

[0083] In S5003, the feature extraction unit 5002 extracts features (multi-dimensional features) from the image sequence acquired in S5002. Note that the process of S5003 is substantially similar to the process of S4002 shown in Fig. 7(a) in the second embodiment, except that the length of the target image sequence is a predetermined uniform length, and therefore a detailed description thereof will be omitted. In S5004, a score calculation unit 5003 predicts a score for each frame from the multidimensional feature amount of the image sequence extracted in S5003. The process of S5004 is also substantially similar to the process of S4003 shown in Fig. 7(a) in the second embodiment, except that the sequence length of the target image sequence is a predetermined uniform length, and therefore a detailed description thereof will be omitted.

[0084] In S5005, the filtering unit 5004 performs filtering based on the scores of the image series calculated in S5004 so that the number of image series corresponding to the mini-batch size is extracted from the number of image series corresponding to the pre-filtering size acquired in S5002. For example, since the score is predicted for each frame of the image series in S5004, the filtering unit 5004 may extract image series corresponding to the mini-batch size in descending order of scene scores obtained by summing or averaging the scores for each frame over all frames. As another example, the filtering unit 5004 may normalize the scene scores so that they are expressed as values ​​indicating a relative ratio to the maximum value that the score can take, and then randomly extract image series corresponding to the mini-batch size using the normalized value as a probability. In this manner, filtering is performed so that an image sequence equal to the mini-batch size is extracted from the series of images acquired in S5002.

[0085] The processes of S5006, S5007, S5008, and S5009 are substantially similar to the processes of S2003, S2004, S2005, and S2006 shown in Fig. 4(c), and therefore detailed explanations will be omitted. In this case, the degradation unit 5005, the image improvement unit 5006, the loss calculation unit 5007, and the update unit 5008 correspond to the degradation unit 2002, the image improvement unit 2003, the loss calculation unit 2004, and the update unit 2005 shown in Fig. 3(b).

[0086] In B5001, the score prediction model learning unit 5010 judges whether or not to perform score prediction learning, which will be described later as S5010, based on a predetermined condition. As a specific example, the score prediction model learning unit 5010 may judge to perform score prediction learning when a certain period of one or more epochs has elapsed, and proceed to the process of S5010. As another example, the score prediction model learning unit 5010 may judge to perform score prediction learning when the calculated loss has dropped by a certain amount or more from the initial or previous loss value, and proceed to the process of S5010. In particular, when the latter judgment is applied, learning of a score prediction model that adaptively follows changes in the model during the learning process is realized. Note that, when the score prediction model learning unit 5010 judges not to perform score prediction learning, it skips the process of S5010.

[0087] The process of S5010 is substantially similar to the process of S4001 shown in Fig. 7(a), and therefore a detailed description thereof will be omitted. In this case, the degradation unit 5005, the image quality improvement unit 5006, the restoration error calculation unit 5009, and the score prediction model learning unit 5010 correspond to the degradation unit 4001, the image quality improvement unit 4002, the restoration error calculation unit 4003, and the score prediction model learning unit 4005 shown in Fig. 6(a). When the process of S5010 is executed, a part of an image sequence of a predetermined frame length stored in the learning dataset 5011 is used as the image sequence.

[0088] The model (for example, the NR model) learned in the above manner is used in the process at runtime. Note that the process at runtime is substantially similar to that in the first embodiment, and therefore a detailed description thereof will be omitted.

[0089] As described above, the image processing system according to this embodiment makes it possible to extract training examples with high learning effectiveness adapted to a model (for example, an NR model) that is updated during the learning process.

[0090] <Modification of the third embodiment> In the above-described third embodiment, an example of a case where an image sequence is filtered based on a score has been described. On the other hand, in the method described in the third embodiment, when filtering an image sequence, feature extraction and score prediction are performed for an image sequence whose number is equal to or larger than the mini-batch size. Therefore, depending on the number of image sequences that serve as a parameter when filtering an image sequence, the calculation cost may become larger. In order to reduce this calculation cost, the image sequence may not be filtered, and the score of the image sequence may be used as a weight when calculating a loss. By applying such control, it is expected that the effect of reducing the influence of learning cases with low learning effect on learning and increasing the influence of learning cases with high learning effect can be achieved.

[0091] In the third embodiment described above, the feature amount for the image sequence is calculated in the process of S5003 in the series of processes related to learning shown in Fig. 7(c). However, since the feature amount does not depend on the NR model or the score prediction model, it is also possible to calculate the feature amount in advance. Therefore, multi-dimensional features may be calculated in advance for a series of images stored in the learning dataset 5011, and stored in association with the image series. By applying such control, it is possible to omit the process related to the extraction of features shown as the process of S5003 in FIG. 7(c), and thus it is expected to have an effect of reducing the calculation cost during model learning. In this method, prediction is performed by the score prediction model in the process of S5004 shown in FIG. 7(c), but the effect can be expected even if the scale of the prediction model is sufficiently small in calculation cost. Therefore, even in a form in which filtering is performed, it is expected to have an effect of suppressing an increase in calculation cost by applying such control.

[0092] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0093] The disclosure of the present embodiment also includes the following configurations, methods, and programs. (Configuration 1) A data processing device comprising: an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series, and on at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to a type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from a series of the reference image series based on the score calculated by the score calculation means. (Configuration 2) A data processing device according to Configuration 1, comprising: a selection reference feature extraction means for extracting a plurality of types of image features as a plurality of selection reference features from the reconstructed image sequence; and a selection means for selecting, based on the reconstructed error calculated by the error calculation means and the plurality of selection reference features extracted by the selection reference feature extraction means, a feature extraction method to be applied to the calculation of the score by the score calculation means, and a method of calculating the score based on the feature, from a plurality of types of candidate extraction methods and a plurality of types of candidate calculation methods, which are specified in advance; wherein the plurality of selection reference features include at least a feature according to the characteristics of the spatiotemporal frequency of the image sequence. (Configuration 3) The data processing device according to configuration 2, wherein the plurality of selection reference features include a feature quantity related to the movement of a subject as a feature quantity according to the characteristics of the spatiotemporal frequency of the image sequence. (Configuration 4) The data processing device according to configuration 3, wherein the feature amount relating to the movement of the subject is an optical flow. (Configuration 5) The data processing device according to any one of configurations 2 to 4, characterized in that the selection means selects a method for extracting the features and a method for calculating the scores that have a higher correlation between the scores calculated from each of the multiple selection criteria features and the recovery error. (Configuration 6) The data processing device according to Configuration 1, wherein the score calculation means inputs the feature quantities of the predetermined type extracted from the degraded image sequence into a prediction model trained based on machine learning to predict the restoration error using the feature quantities extracted from the reference image sequence as input, and applies the restoration error output from the prediction model as the score. (Configuration 7) The data processing device according to configuration 6, wherein the score calculation means inputs a multidimensional feature consisting of a plurality of feature amounts including at least a feature amount corresponding to the spatiotemporal frequency characteristics of an image sequence to the prediction model, and applies the multidimensional recovery error output from the prediction model as a multidimensional score. (Configuration 8) The data processing device according to configuration 7, wherein the prediction model is a regression model in which the multidimensional feature values ​​are explanatory variables and the restoration error is a response variable. (Configuration 9) The data processing device according to configuration 7, wherein the prediction model is a neural network trained using the multidimensional feature as an input and the restoration error as a correct answer. (Configuration 10) The data processing device according to any one of configurations 1 to 9, wherein the restoration error is a PSNR or SSIM calculated from the sequence of restored images and the sequence of degraded images. (Configuration 11) The data processing device according to any one of configurations 1 to 9, characterized in that the restoration error is a PSNR or SSIM of an inter-frame difference calculated from the restored image sequence and the degraded image sequence. (Configuration 12) A data processing device according to any one of configurations 1 to 11, characterized in that the image series extraction means normalizes the score so that it is expressed as a value indicating a relative ratio to a maximum value, and extracts a reference image series to be used for creating a dataset from the series of reference image series based on the probability indicated by the value. (Configuration 13) An image processing device having an image processing means that has been trained based on machine learning to generate and output a restored image series by performing a process on an image series received as input to restore degradation that has become apparent in the image series, wherein a dataset used for training the image processing means is generated based on a reference image series extracted from a series of the reference image series, based on a restored image series restored from a degraded image series in which degradation has been applied to a reference image series, and at least one of the reference image series and the degraded image series, a restoration error related to the restoration of the restored image series calculated based on a predetermined type of feature extracted from the degraded image series, and a score corresponding to the type of feature calculated based on the predetermined type of feature extracted from the degraded image series. (Configuration 14) The image processing device described in Configuration 13, characterized in that the image processing means performs learning based on the loss between the reference image sequence included in the dataset generated in advance and the restored image sequence restored from the degraded image sequence generated by applying degradation to the reference image sequence. (Configuration 15) The image processing device described in Configuration 14, characterized in that the degraded image series is generated by degrading at least a portion of a reference image series extracted from a series of the reference image series so as to be a predetermined number, based on the score predicted from a multidimensional feature consisting of a plurality of feature amounts including at least a feature amount corresponding to the spatio-temporal frequency characteristics of the image series extracted from the reference image series, and the image processing means learns based on the loss between the at least a portion of the extracted reference image series and the restored image series restored from the degraded image series generated from the reference image series. (Configuration 16) The image processing device described in Configuration 15, characterized in that the score is obtained by inputting features extracted from the reference image series into a prediction model trained based on machine learning to predict a restoration error related to the restoration of the restored image series based on the restored image series and at least one of the reference image series and the degraded image series, and the restoration error output from the prediction model is applied, and the prediction model is trained when predetermined conditions are satisfied during the learning process of the image processing means. (Configuration 17) The image processing device according to configuration 16, wherein the predictive model is trained when the change in the loss becomes equal to or greater than a threshold during the learning process of the image processing means. (Configuration 18) The image processing device described in Configuration 16, characterized in that the predictive model is learned when a predetermined number of iterations related to learning have elapsed during the learning process of the image processing means. (Configuration 19) The image processing device according to any one of configurations 15 to 18, characterized in that the multidimensional features extracted from the reference image series included in the data set are associated with the reference image series. (Method 1) A method for controlling a data processing device, comprising the steps of: 1. A method for controlling a data processing device, comprising: an error calculation step of calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series obtained by degrading a reference image series and on at least one of the reference image series and the degraded image series; a score calculation step of calculating a score according to a type of a predetermined type of feature based on the restoration error calculated in the error calculation step and a predetermined type of feature extracted from the degraded image series; and an image series extraction step of extracting a reference image series to be used for creating a dataset from a series of the reference image series based on the score calculated in the score calculation step. (Program 1) A program for causing a computer to function as a data processing device, comprising: an error calculation means for calculating a restoration error associated with the restoration of a restored image series based on a restored image series restored from a degraded image series that has been degraded relative to a reference image series and on at least one of the reference image series and the degraded image series; a score calculation means for calculating a score according to the type of feature based on the restoration error calculated by the error calculation means and a predetermined type of feature extracted from the degraded image series; and an image series extraction means for extracting a reference image series to be used for creating a dataset from a series of the reference image series based on the score calculated by the score calculation means. [Explanation of symbols]

[0094] 1000 Data processing device 1003 Restoration error calculation unit 1007 Score Calculation Unit 1008 Sampling section

Claims

1. an error calculation means for calculating a restoration error related to restoration of the restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation means for calculating a score according to a type of a feature amount based on the restoration error calculated by the error calculation means and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction means for extracting a reference image sequence to be used for creating a data set from a series of the reference image sequences based on the score calculated by the score calculation means; A data processing device comprising:

2. a selection reference feature extracting means for extracting feature quantities of a plurality of types of images as a plurality of selection reference features from the sequence of restored images; a selection means for selecting, based on the recovery error calculated by the error calculation means and the plurality of selection reference features extracted by the selection reference feature extraction means, a feature extraction method to be applied to the calculation of the score by the score calculation means and a score calculation method based on the feature from a plurality of types of candidate extraction methods and a plurality of types of candidate calculation methods defined in advance; having The plurality of selection criteria features include at least a feature quantity according to the spatiotemporal frequency characteristics of the image sequence.

2. A data processing device according to claim 1.

3. 3. The data processing apparatus according to claim 2, wherein the plurality of selection reference features include a feature quantity relating to a movement of a subject as a feature quantity according to a characteristic of a spatiotemporal frequency of the image sequence.

4. 4. The data processing device according to claim 3, wherein the feature amount relating to the movement of the subject is an optical flow.

5. 3. The data processing apparatus according to claim 2, wherein the selection means selects a method for extracting the feature amount and a method for calculating the score that have a higher correlation between the score calculated from each of the plurality of selection reference features and the recovery error.

6. 2. The data processing device according to claim 1, wherein the score calculation means inputs the feature quantities of the predetermined type extracted from the degraded image sequence into a prediction model trained based on machine learning to predict the restoration error using the feature quantities extracted from the reference image sequence as input, and applies the restoration error output from the prediction model as the score.

7. 7. The data processing device according to claim 6, wherein the score calculation means inputs a multidimensional feature amount constituted by a plurality of feature amounts including at least a feature amount according to a characteristic of a spatio-temporal frequency of an image sequence to the prediction model, and applies the multidimensional recovery error output from the prediction model as a multidimensional score.

8. 8. The data processing apparatus according to claim 7, wherein the prediction model is a regression model in which the multidimensional feature amount is an explanatory variable and the restoration error is a response variable.

9. 8. The data processing apparatus according to claim 7, wherein the prediction model is a neural network trained using the multidimensional feature vector as an input and the restoration error as a correct answer value.

10. 2. The data processing apparatus according to claim 1, wherein the restoration error is a PSNR or an SSIM calculated from the sequence of restored images and the sequence of degraded images.

11. 2. The data processing apparatus according to claim 1, wherein the restoration error is a PSNR or SSIM of an inter-frame difference calculated from the sequence of restored images and the sequence of degraded images.

12. 2. The data processing apparatus according to claim 1, wherein the image sequence extraction means normalizes the score so as to be expressed as a value indicating a relative ratio to a maximum value, and extracts a reference image sequence to be used for creating a data set from the series of reference image sequences based on the probability indicated by the value.

13. The image processing device includes an image processing unit that has been trained based on machine learning to perform a process for restoring degradation that has become apparent in an image sequence received as an input, thereby generating and outputting a restored image sequence, The data set used for training the image processing means is a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence, and a restoration error related to the restoration of the restored image sequence calculated based on at least one of the reference image sequence and the degraded image sequence; A feature quantity of a predetermined type extracted from the sequence of degraded images; Based on the score calculated based on the feature type, A sequence of reference images is generated based on the sequence of reference images extracted from the sequence of reference images.

1. An image processing device comprising:

14. The image processing means includes: the reference image sequence contained in the pre-generated dataset; a restored image sequence restored from the degraded image sequence generated by degrading the reference image sequence; Learning is done based on the loss between The image processing device according to claim 13 .

15. the degraded image sequence is generated by degrading at least a predetermined number of reference image sequences extracted from a series of the reference image sequences, based on the score predicted from a multidimensional feature amount composed of a plurality of feature amounts including at least a feature amount corresponding to a spatio-temporal frequency characteristic of the image sequence extracted from the reference image sequence; The image processing means performs learning based on a loss between at least a part of the extracted reference image sequence and the restored image sequence restored from the degraded image sequence generated from the reference image sequence. The image processing device according to claim 14 .

16. the score is determined by inputting feature quantities extracted from the reference image sequence into a prediction model trained based on machine learning so as to predict a restoration error related to restoration of the restored image sequence based on the restored image sequence and at least one image sequence of the reference image sequence and the degraded image sequence, and applying the restoration error output from the prediction model; The prediction model is trained when a predetermined condition is satisfied during the training process of the image processing means.

16. The image processing device according to claim 15.

17. 17. The image processing device according to claim 16, wherein said prediction model is trained when a change in said loss becomes equal to or greater than a threshold value during a training process of said image processing means.

18. 17. The image processing device according to claim 16, wherein the prediction model is learned when a predetermined number of iterations related to learning have elapsed in a learning process of the image processing means.

19. The image processing apparatus according to claim 15, wherein the multidimensional features extracted from the reference image sequence included in the data set are associated with the reference image sequence.

20. A method for controlling a data processing device, comprising: an error calculation step of calculating a restoration error related to restoration of the restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation step of calculating a score according to a type of a feature amount based on the restoration error calculated in the error calculation step and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction step of extracting a reference image sequence to be used for creating a dataset from the reference image sequence based on the score calculated in the score calculation step; 2. A method for controlling a data processing device, comprising:

21. Computer, an error calculation means for calculating a restoration error related to restoration of the restored image sequence based on a restored image sequence restored from a degraded image sequence obtained by degrading a reference image sequence and at least one of the reference image sequence and the degraded image sequence; a score calculation means for calculating a score according to a type of a feature amount based on the restoration error calculated by the error calculation means and a feature amount of a predetermined type extracted from the sequence of degraded images; an image sequence extraction means for extracting a reference image sequence to be used for creating a data set from a series of the reference image sequences based on the score calculated by the score calculation means; A program for causing a data processing device to function, comprising:

Images