Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device includes a mask region acquisition unit, a history complementing unit, and a DNN. The mask region acquisition unit identifies, in an inferred history frame, i.e. an inferred image of a past frame, an image element in which a motion vector used for motion compensation is unknown. The mask region acquisition unit acquires, as a mask region, a region corresponding to the image element. The history complementing unit acquires a complementary history frame in which a portion corresponding to the mask region in the inference history frame after motion compensation is complemented on the basis of an input current frame. The DNN outputs an inferred current frame on the basis of the input current frame and the complementary history frame.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Deep neural network (DNN) processing for video generally uses a recurrent neural network (RNN) structure with high time stability. This is because the RNN structure can correlate the current frame (present information) with the history frames (past information), making it easier to maintain the consistency of the output frame image.

[0003] Special Publication No. 2022-547517

[0004] The history frames, along with the current frame, are used as inputs to the DNN. To ensure consistency with the current frame, the history frames undergo a process called motion compensation, which takes into account the motion between frames. Motion compensation is a process that predicts the data of the current frame from past frames based on the motion between frames. However, for image elements added in post-processing, such as effects, motion compensation cannot be performed because motion vectors cannot be obtained. Data distortion occurs in the history frames after motion compensation, which affects the inference results of the DNN.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that can improve video quality.

[0006] According to the present disclosure, there is provided an information processing device including: a mask region acquisition unit that identifies image elements for which motion vectors used for motion compensation are unknown in an inference history frame, which is an inference image of a past frame, and acquires an area corresponding to the image element as a mask region; a history completion unit that acquires an interpolated history frame in which a portion of the inference history frame after the motion compensation corresponding to the mask region is interpolated based on an input current frame; and a DNN that outputs an inference current frame based on the input current frame and the interpolated history frame. Also according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium storing a program for causing a computer to realize the information processing of the information processing device.

[0007] 1 is a diagram illustrating conventional DNN processing for moving images; FIG. 2 is a diagram illustrating data distortion due to motion compensation; FIG. 3 is a diagram illustrating the occurrence of ghosts due to motion compensation; FIG. 4 is a diagram illustrating an example of a configuration of an information processing device that performs DNN processing for moving images according to the present disclosure; FIG. 5 is a diagram illustrating a method for improving defects caused by motion compensation of image elements; FIG. 6 is a diagram illustrating an example of a process for removing image elements; FIG. 7 is a diagram illustrating an example of a process for removing image elements; FIG. 8 is a diagram illustrating an example of a method for determining a threshold value that serves as a setting standard based on final image quality; FIG. 9 is a diagram illustrating an example of a processing flow for performing inference processing; FIG. 10 is a diagram illustrating an example of a configuration of another information processing device that performs DNN processing for moving images according to the present disclosure; FIG. 11 is a diagram illustrating an example of a process for complementing simply corrected history frames; FIG. 12 is a diagram illustrating an example of a process for classifying inter-frame differences; FIG. 13 is a diagram illustrating an example of a mathematical model used as an analogy judgment criterion; FIG. 14 is a diagram illustrating an example of a method for determining a threshold value for defining the range of an effect area based on final image quality; FIG. 15 is a diagram illustrating an example of a processing flow for performing inference processing; FIG. 16 is a diagram illustrating an example of a hardware configuration of an information processing device.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Background] [1-1. Conventional DNN processing for moving images] [1-2. Data disturbance due to motion compensation] [2. Example 1 of DNN processing for moving images according to the present disclosure] [2-1. System configuration example] [2-2. Image element removal processing] [2-3. Processing flow] [2-4. Effects] [3. Example 2 of DNN processing for moving images according to the present disclosure] [3-1. System configuration example] [3-2. Complement processing of simple correction history frames] [3-3. Processing flow] [3-4. Effects] [4. Hardware configuration example]

[0010] [1. Background] [1-1. Conventional DNN Processing for Video] FIG. 1 is a diagram showing conventional DNN processing for video.

[0011] DNN processing for video generally uses an RNN structure with high temporal stability. The RNN structure can correlate the current frame with history frames, making it easier to maintain image consistency in the output frames. While DNNs have very high inference performance (for example, sharpening effects in the case of super-resolution), they are affected by even small changes in the input data, which can change the output results. Therefore, in video processing, flickering is more likely to occur than with simple filter processing. To increase temporal stability and reduce flickering, an RNN structure is generally implemented that inputs the current frame and history frames simultaneously, enabling learning that includes temporal correlation (similarity between the current frame and history frames).

[0012] Note that the current frame refers to the image of the frame (current frame) that is the subject of estimation by the DNN. The history frame refers to the image obtained by inputting an image of a past frame into the DNN (an inferred image of the past frame). The past frame refers to a frame that is one or more frames before the current frame. In the present disclosure, for example, the frame that is one frame before the current frame (the most recent frame) is used as the past frame. Hereinafter, the current frame may be referred to as the "input current frame," and the history frame, which is an inferred image of the past frame, may be referred to as the "inferred history frame."

[0013] The inferred history frame, together with the input current frame, is used as input to the DNN. To ensure consistency with the input current frame, the inferred history frame undergoes motion compensation based on a motion vector. A motion vector is a vector that indicates the movement of an object between frames (the amount and direction of pixel movement). Motion compensation is a process that predicts post-motion data from pre-motion data. For example, when motion compensation is applied to an image of a past frame, the image predicted in the current frame is obtained as the motion-compensated inferred history frame. The motion-compensated inferred history frame is input to the DNN along with the input current frame.

[0014] [1-2. Data Disturbance Due to Motion Compensation] FIGS. 2 and 3 are diagrams for explaining data disturbance due to motion compensation.

[0015] 2 shows an example in which an image element IE is added to a rendered image of a 3D object as a post-processing step. The renderer obtains the motion vector MV of the 3D object during rendering. Since the image element IE is arbitrarily added in the post-processing step, the motion vector MV of the image element IE does not exist. Motion compensation is performed only for the 3D object. Therefore, there is a possibility that a discrepancy may occur between the position of the image element IE after motion compensation and the actual position of the image element IE shown in the input current frame.

[0016] For example, Figure 3 shows an example in which a 3D object serving as a subject moves diagonally upward to the right as the frames progress. In a typical renderer, velocity information (motion vector MV) is associated with the subject and stored. In many cases, the distance (depth) to the subject is also acquired simultaneously with the velocity information.

[0017] Post-processing is performed on the rendered image. In the example of FIG. 3 , image elements IE (effect images) that function as effects are added to the rendered image. Effects include those that are generated by superimposing images and those that are generated by processing images (blurring and color processing). Effects generated by superimposing images include bloom, particles, reflection, and transparency. Effects generated by processing images include depth of field (DoF), motion blur, tone mapping, and lens flare. By performing post-processing on the rendered image, the final output image includes image elements IEs whose motion is unknown.

[0018] The renderer obtains a motion vector MV of the object based on the motion of the object and the motion of the rendering viewpoint. The image element IE is not an object existing in 3D space (e.g., an object with a mesh), but is arbitrarily added in post-processing, so its motion is unknown. There is no motion vector MV for the image element IE; the motion vector MV is obtained only for the object. Because the image element IE does not have a motion vector MV, the position of the image element IE after motion compensation remains unchanged from its position before motion compensation. Therefore, there is a possibility that the position of the image element IE may be shifted between the input current frame and the inference history frame after motion compensation.

[0019] As an inference result, the DNN outputs an inference image that is a blend of the input current frame and the motion-compensated inference history frame. Therefore, in the inference image, at the positions where the image element IE of the input current frame was displayed and at the positions where the image element IE of the inference history frame was displayed, image elements IE with reduced brightness compared to their original display states are displayed. As a result, a problem of reduced brightness occurs at the positions where the image element IE should originally be reproduced, and a problem of ghosts (afterimages of the image element IE included in the inference history frame) occurs at positions where the image element IE should not be reproduced.

[0020] The present disclosure has been made to improve such defects (ghosts and brightness reduction) in inferred images. In the present disclosure, an area where defects may occur during motion compensation is acquired based on information on post-processing related to the generation of image elements IE or information on the difference between the input current frame and the motion-compensated inferred history frame. The acquired area is used as a mask area for masking data to be corrected. A motion-compensated inferred history frame (complementary history frame) in which an image of a portion corresponding to the mask area has been corrected is input to the DNN along with the input current frame.

[0021] The correction is performed using data from the input current frame. Therefore, misalignment of the image element IE between the input current frame and the complementary history frame is unlikely to occur. As a result, the above-mentioned defects in the inference image are suppressed, and high-quality inference results from the DNN are obtained. The method of the present disclosure will be described in detail below. In the following example, an example will be described in which the image element IE is an effect image added in post-processing. However, the method of the present disclosure is not limited to effect images, and can be applied to all image elements IE whose motion vectors MV are unknown.

[0022] [2. DNN Processing Example 1 for Moving Images According to the Present Disclosure] [2-1. System Configuration Example] Fig. 4 is a diagram illustrating a configuration example of an information processing device 1 that performs DNN processing for moving images according to the present disclosure. Fig. 5 is a diagram illustrating a method for improving defects caused by motion compensation of image elements IE.

[0023] In this example, after removing an image element IE from the inference history frame, motion compensation is performed, and the image element IE is again complemented in the motion-compensated image. The removal and complementation of the image element IE directly eliminates causes of ghosting and the like. The DNN processing for moving images disclosed herein is performed by an information processing device 1. For example, the information processing device 1 includes an input unit 10, a renderer 20, a scaler 30, a mask region acquisition unit 40, a restoration unit 50, a motion compensation unit 60, a history complement unit 70, a DNN 80, and an output unit 90.

[0024] The input unit 10 inputs an operation signal S to the renderer 20. For example, the operation signal S includes a rendering setting and an instruction to execute rendering.

[0025] The renderer 20 performs rendering processing based on the operation signal S. The renderer 20 performs rendering processing based on the input current frame I Ｃ The rendered image is output as the input current frame I Ｃ refers to a frame of image in a series of frames such as a video. The block diagram in Figure 4 explains the operation for one frame of a video.

[0026] The renderer 20 obtains the motion vector MV of the subject and the transparency mask TP of the image element IE based on the rendering settings. The motion vector MV is a two-dimensional vector that defines the movement of corresponding pixels between frames by the amount of pixel movement (number of pixels). The motion vector MV can be obtained by a general estimation process using a group of consecutive frames. If a CG renderer is used, the motion vector MV can be generated and obtained during rendering. The transparency mask TP is an image that indicates the distribution of the transmittance of the image element IE. The pixel value of each pixel in the transparency mask TP indicates the transmittance of the image element IE.

[0027] The scaler 30 scales the input current frame I as needed. Ｃ、 The number of pixels of the motion vector MV and the transparency mask TP is enlarged. Hereinafter, when it is necessary to distinguish the data after the number of pixels has been enlarged by the scaler 30 from the data before enlargement, a "'" is added after the symbol of the data. The scaler 30 enlarges the number of pixels of the input current frame I. Ｃ The scaler 30 sends the motion vector MV' to the restoration unit 50, the history completion unit 70, and the DNN 80. The scaler 30 sends the motion vector MV' to the motion compensation unit 60. The scaler 30 sends the transparency mask TP' to the mask region acquisition unit 40.

[0028] The architecture of the DNN 80 accepts a fixed number of pixels, while the input current frame I ＣIn cases where the number of pixels varies depending on the usage, the scaler 30 may perform pixel number expansion processing at a desired magnification to obtain a pixel number that matches the specifications of the DNN 80. The pixel number expansion method may be selected arbitrarily, for example, bilinear or bicubic. The motion vector MV is preferably expanded using the nearest neighbor method to avoid the generation of intermediate values ​​due to pixel interpolation, but is not limited to this.

[0029] The mask region acquisition unit 40 acquires the inference history frame I Ｈ When performing motion compensation on the image element IE, an area where a defect may occur due to an image element IE with an unknown motion vector is acquired as a mask area MA. Ｈ The position (ghost) of the image element IE and the input current frame I Ｃ This can occur at the position of the image element IE of ' (brightness reduction).

[0030] For example, the mask region acquisition unit 40 acquires the inference history frame I Ｈ The mask area acquisition unit 40 acquires an area corresponding to the image element IE as a mask area MA. The mask area acquisition unit 40 acquires a first mask area MA as the mask area MA. Ｈ and the second mask area MA Ｃ The first mask area MA Ｈ is the inference history frame I Ｈ The second mask area MA indicates the range of the image element IE in the Ｃ is the input current frame I Ｃ ' indicates the range of the image element IE.

[0031] The mask region acquisition unit 40 can acquire a mask region MA based on information on post-processing related to the generation of image elements IE. For example, the mask region acquisition unit 40 acquires a transparency mask TP' as information on post-processing related to the generation of image elements IE. The transparency mask TP' includes information on the positions of the image elements IE. The mask region acquisition unit 40 can estimate a region where a defect may occur during motion compensation based on the positions of the image elements IE. The mask region acquisition unit 40 acquires the estimated region as a mask region MA for masking the data to be corrected.

[0032] For example, the mask area acquisition unit 40 converts the transparency mask TP′ of the current frame acquired from the scaler 30 into the effect mask M Ｃ The mask area acquisition unit 40 converts the generated effect mask M Ｃ to the history complement unit 70. The mask area acquisition unit 40 sends the effect mask M of the previous frame acquired one frame before Ｈ is sent to the restoration unit 50.

[0033] Effect Mask M Ｃ The range of the image element IE to be removed is defined as the second mask area MA. Ｃ For example, the effect mask M Ｃ is a binary image in which pixels in the area to be removed are black (pixel value = 0) and pixels in the area not to be removed are white (pixel value = 1). The image area shown in black is the range of the image element IE to be removed (second mask area MA Ｃ The conversion process is performed by binarizing the pixel values ​​of the transparency mask TP′. The mask area acquisition unit 40 may, as necessary, obtain a second mask area MA Ｃ By performing a process of enlarging the image, the range of the image to be corrected can be widened.

[0034] The reconstruction unit 50 extracts the inference image of the past frame from the DNN 80 as the inference history frame I Ｈ The reconstruction unit 50 acquires the inference history frame I Ｈ The image elements IE whose inter-frame motion vectors MV are unknown are removed from the restored history frame I.Ｈｅ The removal process of the image element IE is performed by replacing the image element IE with a filler image. The filler image is generated by replacing the image element IE with the inference history frame I Ｈ For example, the restoration unit 50 may use an image that does not spoil the atmosphere of the inference history frame I. Ｈ The first mask area MA Ｈ Input the pixel values ​​of the part corresponding to the current frame I Ｃ ' and restore the history frame I Ｈｅ Generate.

[0035] 5 shows an example of a calculation formula for removing the image element IE. In the calculation formula, "1-M Ｈ " is the effect mask M of the past frame Ｈ This image is an image whose pixel value is the value obtained by subtracting the pixel value (0 or 1) of the effect mask M Ｈ It is an inverted image in which white (pixel value = 1) and black (pixel value = 0) are inverted.

[0036] "(1-M Ｈ ) × I Ｃ " is Effect Mask M Ｈ The pixel values ​​of the inverted image of the input current frame I Ｃ This image is an image whose pixel values ​​are the product of the pixel values ​​of the effect mask M Ｈ The mask area MA (first mask area MA Ｈ ) Input current frame I Ｃ ' is replaced with 0, that is, the first mask area MA Ｈ Input current frame I Ｃ ' means an image of

[0037] "M Ｈ ×I Ｈ " is the effect mask M of the past frame Ｈ Pixel values ​​of and inference history frame I Ｈ This image is an image whose pixel values ​​are the product of the pixel values ​​of the effect mask M Ｈ The mask area MA (first mask area MA Ｈ) Inference history frame I Ｈ The image in which the pixel values ​​of the first mask area MA are replaced with 0, i.e., Ｈ Inference history frame I of a part other than the part corresponding to Ｈ means an image of

[0038] "(1-M Ｈ ) × I Ｃ +M Ｈ ×I Ｈ " is the first mask area MA Ｈ Input current frame I Ｃ ' and the first mask area MA Ｈ Inference history frame I of a part other than the part corresponding to Ｈ This image summation creates an inference history frame I Ｈ The pixel values ​​of the image elements IE included in the input current frame I Ｃ ' (the restored history frame I Ｈｅ ) is obtained.

[0039] The motion compensation unit 60 receives the restored history frame I from the restoration unit 50. Ｈｅ The motion compensation unit 60 obtains the reconstructed history frame I based on the motion vector MV'. Ｈｅ , and perform motion compensation to obtain the corrected history frame I Ｈｅ ＭＣ The motion compensation unit 60 performs a common process known as motion compensation to generate the reconstructed history frame I. Ｈｅ can be applied to.

[0040] The motion vector MV′ is calculated based on the reconstructed history frame I Ｈｅ Input current frame I Ｃ '. By motion compensation, the reconstructed history frame I Ｈｅ and the input current frame I Ｃ The positions of corresponding pixels in the motion-compensated restored history frame I Ｈｅ (Correction History Frame I Ｈｅ ＭＣ ) to the history complement unit 70.

[0041] The history complement unit 70 converts the image of the current frame into an input current frame I Ｃ The history complement unit 70 obtains the motion-compensated inference history frame I Ｈ The part of the input current frame I that corresponds to the mask area MA is Ｃ The completed history frame I is completed based on Ｈｒ For example, the history complement unit 70 obtains the corrected history frame I Ｈｅ ＭＣ The second mask area MA Ｃ Input the pixel values ​​of the part corresponding to the current frame I Ｃ In this way, the history complement unit 70 replaces the pixel values ​​of the input current frame I Ｃ The image element IE extracted from the correction history frame I Ｈｅ ＭＣ The image added to the history frame I is complemented. Ｈｒ Generate it as:

[0042] 5 shows an example of a calculation formula for completing an image element IE. In the calculation formula, "1-M Ｃ " is the effect mask M of the current frame. Ｃ This image is an image whose pixel value is the value obtained by subtracting the pixel value (0 or 1) of the effect mask M Ｃ It is an inverted image in which white (pixel value = 1) and black (pixel value = 0) are inverted.

[0043] "(1-M Ｃ ) × I Ｃ " is the effect mask M of the current frame. Ｃ The pixel values ​​of the inverted image of the input current frame I Ｃ This image is the image whose pixel values ​​are the product of the pixel values ​​of the effect mask M Ｃ The mask area MA (second mask area MA Ｃ ) Input current frame I Ｃ ' is replaced with 0, that is, the second mask area MA Ｃ Input current frame IＣ ' means an image of

[0044] "M Ｃ ×I Ｈｅ ＭＣ " is the effect mask M of the current frame. Ｃ Pixel values ​​of the corrected history frame I Ｈｅ ＭＣ This image is the image whose pixel value is the product of the pixel value of the effect mask M Ｃ The mask area MA (second mask area MA Ｃ ) Correction history frame I Ｈｅ ＭＣ The image in which the pixel values ​​of the second mask area MA are replaced with 0, i.e., Ｃ Correction history frame I of the part other than the part corresponding to Ｈｅ ＭＣ means an image of

[0045] "(1-M Ｃ ) × I Ｃ +M Ｃ ×I Ｈｅ ＭＣ " is the second mask area MA Ｃ Input current frame I Ｃ ' and the second mask area MA Ｃ Correction history frame I of the part other than the part corresponding to Ｈｅ ＭＣ This summation of images gives the corrected history frame I Ｈｅ ＭＣ Input current frame I Ｃ The image element IE of ' is added (the complementary history frame I Ｈｒ ) is obtained. The completed history frame I Ｈｒ The position of the image element IE in the input current frame I Ｃ Therefore, ghosts and brightness reduction after DNN processing are suppressed.

[0046] The DNN 80 receives the input current frame I Ｃ ' and the complementary history frame I ＨｒThe DNN 80 outputs the inferred image of the current frame obtained by inference as the inferred current frame. The inferred current frame is the output image I Ｏ The output unit 90 outputs the output image I Ｏ General post-processing such as color conversion and codec is applied to the image. Tasks for the DNN 80 include super-resolution, noise reduction, and style transfer, but the present disclosure is not limited thereto. The DNN 80 can be applied to general video processing using an RNN structure.

[0047] As described above, in this example, the inference history frame I Ｈ The image before adding the image element IE is restored from the restored image (restored history frame I Ｈｅ ) is subjected to motion compensation. The history frame obtained by the motion compensation (corrected history frame I Ｈｅ ＭＣ ) input current frame I Ｃ The image element IE of the image is added, and the resulting image (the complementary history frame I Ｈｒ ) is the input current frame I Ｃ This inputs the current frame I Ｃ and complementary history frame I Ｈｒ As a result, defects such as ghosting and brightness reduction are suppressed, and high-quality DNN inference results are obtained.

[0048] In the conventional method described in Patent Document 1, disturbed portions of the image (noise regions) are detected from the inference image, and the DNN is optimized using a loss function weighted toward the noise regions. However, this method weakens the optimization of non-noise regions, resulting in a decrease in inference accuracy. Furthermore, detecting noise regions from the inference image is difficult even with a dedicated discriminative model. For example, simple subtraction using consecutive frames makes it difficult to detect subjects with small differences in brightness or color (such as transparent effects superimposed in post-processing). This makes it difficult to weight the loss appropriately.

[0049] On the other hand, the method of this example directly improves the cause of ghosting by removing and complementing the problematic image elements IE. Furthermore, the method of this example also suppresses the decrease in brightness of the image elements IE in the inferred image shown in Figure 2. Therefore, a greater effect can be obtained than with conventional methods.

[0050] 6 and 7 are diagrams illustrating an example of the process of removing image elements IE. In this example, particles P are used to produce an effect. The particles P have a transmittance distribution in which the transparency increases from the center to the outside. The particles P are used to produce sparks in battle scenes, for example.

[0051] The renderer 20 has a rendering engine 21 and a post-processing unit 22. The post-processing unit 22 performs post-processing on the image (rendered image) output from the rendering engine and adds particles P.

[0052] The mask region acquisition unit 40 acquires a transparency mask TP as post-processing information from the post-processing unit 22. The mask region acquisition unit 40 acquires the transmittance of a plurality of particles P to be added as an effect from the transparency mask TP. The mask region acquisition unit 40 extracts a low transmittance region in each particle P having a transmittance lower than a set reference value from the inference history frame I to be removed. Ｈ The mask area acquisition unit 40 sets the low-transmittance area as a mask area MA (pixel value = 0) and sets the area other than the low-transmittance area as a non-mask area (pixel value = 1). In this way, the mask area acquisition unit 40 converts the transparency mask TP into an effect mask M.

[0053] The setting criteria are set according to the ease with which ghosting is apparent. For example, when the transparency of the effect is high (e.g., 95%), the pixel values ​​mostly take on the value of the background of the effect. Therefore, problems such as ghosting and brightness reduction are unlikely to occur in the inference results of the DNN80. When the transparency of the effect is low (e.g., 10%), the effect is clearly displayed, and ghosting is likely to occur in the inference results of the DNN80. Therefore, the setting criteria can be set based on the magnitude of the transparency of the effect.

[0054] In the example of FIG. 7 , the transmittance of the transparency mask TP is binarized based on the threshold value th. For example, an area with a transmittance equal to or less than the threshold value th is detected as an effect area (low transmittance area) where ghosts are likely to become apparent. If the threshold value th is too high, an unnecessarily wide range will be detected as an area to be removed. If the threshold value th is too low, the area that should be detected as an area to be removed will not be detected sufficiently. The system developer can set the threshold value th appropriately depending on how easily ghosts become apparent.

[0055] The mask region acquisition unit 40 targets low-transmittance regions where transparency is low and ghosts are likely to become apparent, and excludes high-transmittance regions where transparency is high and ghosts are unlikely to become apparent. Ｈ There is an advantage to reusing the inference history frame I Ｈ Therefore, only low transmittance areas where ghosts are likely to become apparent are set as areas to be removed.

[0056] The threshold value th can also be determined based on the final image quality. Alternatively, the mask area MA may be detected based on machine learning using the input current frame and the correction history frame as student data and the effect mask M created from the transparency mask TP as teacher data. Figure 8 shows an example of a method for determining the threshold value th, which serves as a setting standard, based on the final image quality.

[0057] In the example of FIG. 8, the set criterion is the inferred current frame (output image I ＯThe image quality evaluation value of the image quality evaluation method is set as a condition for maximizing the image quality evaluation value of the image quality evaluation method (grid search). As an image quality evaluation method, PSNR (Peak Signal to Noise Ratio) or SSIM (Structural Similarity) is used.

[0058] The image quality is evaluated by the image quality evaluation unit EV. The image quality evaluation unit EV may be incorporated into the information processing device 1 or may be provided independently of the information processing device 1. The image quality evaluation unit EV acquires a plurality of inference images with different threshold values ​​th from the DNN 80. The image quality evaluation unit EV compares the acquired inference images with the ground truth and adopts the threshold value th that obtains the inference image closest to the ground truth as the setting standard. The search for the threshold value th may be performed with fewer iterations using a general method such as Bayesian optimization.

[0059] In order to more reliably prevent the occurrence of ghosts, the restoration unit 50 extracts a range that is expanded from the low transmittance region by a predetermined pixel size from the inference history frame I. Ｈ For example, the restoration unit 50 may remove pixel values ​​in a range expanded from the low transmittance region outward by a predetermined pixel size from the input current frame I Ｃ The particle P is depicted as an image in which the transmittance changes gradually from the center to the periphery. Although not as severe as the center, the outer periphery of the particle P may also cause image distortion. By expanding the removal range to include such areas, ghosting can be suppressed more effectively.

[0060] 2-3. Processing Flow FIG. 9 is a diagram showing an example of a processing flow for performing inference processing.

[0061] The scaler 30 receives the input current frame I from the renderer 20. Ｃ , motion vectors MV and transparency mask TP. The scaler 30 scales the input current frame I as needed. Ｃ , the number of pixels of the motion vector MV and the transparency mask TP are enlarged, and the input current frame I Ｃ ', motion vector MV' and transparency mask TP' are obtained (step S1).

[0062] The mask area acquisition unit 40 acquires the effect mask M of the current frame based on the transparency mask TP' of the current frame and the transparency mask TP' of the previous frame. Ｃ and effect mask M of past frames Ｈ (Step S2). The restoration unit 50 acquires the input current frame I Ｃ ', inference history frame I Ｈ , and the effect mask M of the past frame Ｈ Based on this, the inference history frame I Ｈ As a result, the restoration unit 50 removes the effect from the rendering image of the previous frame before the effect was applied (the restored history frame I Ｈｅ ) is restored (step S3).

[0063] The motion compensation unit 60 calculates the reconstructed history frame I based on the motion vector MV'. Ｈｅ As a result, the motion compensation unit 60 applies motion compensation to the object position relative to the input current frame I Ｃ Corrected history frame I combined with Ｈｅ ＭＣ (Step S4). The history complement unit 70 obtains the effect mask M Ｃ Based on the input current frame I Ｃ Correct the effect of 'History Frame I Ｈｅ ＭＣ As a result, the history complementing unit 70 adds the effect-complemented history frame I Ｈｒ is acquired (step S5).

[0064] The scaler 30 and the history complement unit 70 calculate the input current frame I Ｃ ' and the complementary history frame I Ｈｒ is input to the trained DNN 80 (step S6). The DNN 80 outputs the inference result, Ｏ , the final output, and the inference history frame I for the next video processing. Ｈ (Step S7) After that, the recursive processing of the RNN is repeated.

[0065] [2-4. Effects] The information processing device 1 includes a mask region acquisition unit 40, a history completion unit 70, and a DNN 80. The mask region acquisition unit 40 acquires an inference history frame I, which is an inference image of a past frame. Ｈ In the image element IE, the motion vector MV to be used for motion compensation is unknown. The mask area acquisition unit 40 acquires an area corresponding to the image element IE as a mask area MA. The history complement unit 70 acquires an inference history frame I after motion compensation. Ｈ The part of the input current frame I that corresponds to the mask area MA is Ｃ The completed history frame I is Ｈｒ The DNN 80 obtains the input current frame I Ｃ and complementary history frame I Ｈｒ In the information processing method disclosed herein, the processing of the information processing device 1 is executed by a computer. A computer-readable non-transitory storage medium disclosed herein stores a program that causes a computer to realize the processing of the information processing device 1.

[0066] According to this configuration, the input current frame I Ｃ and complementary history frame I Ｈｒ Therefore, defects such as ghosting and brightness reduction are suppressed, and image quality is improved.

[0067] The mask area acquisition unit 40 acquires the first mask area MA Ｈ and the second mask area MA Ｃ The first mask area MA is obtained. Ｈ is the inference history frame I Ｈ The second mask area MA indicates the range of the image element IE in the Ｃ is the input current frame I Ｃ 1 shows the range of the image element IE in

[0068] The DNN 80 outputs the inference result as the input current frame I Ｃ The inference image is a blend of the input current frame I and the history frame after motion compensation.Ｃ The area where the image element IE was drawn, and the inference history frame I Ｈ In this case, defects such as ghosting and brightness reduction tend to occur in the area where the image element IE is drawn. By correcting such an area with the image data of the input current frame, the above defects can be more effectively eliminated.

[0069] The mask area acquisition unit 40 acquires the mask area MA based on information of post-processing related to the generation of the image element IE.

[0070] According to this configuration, it is possible to accurately identify the image element IE to be removed.

[0071] The information processing device 1 includes a restoration unit 50 and a motion compensation unit 60. The restoration unit 50 restores an inference history frame I Ｈ The first mask area MA Ｈ Input the pixel values ​​of the part corresponding to the current frame I Ｃ The restored history frame I is replaced with the pixel value of Ｈｅ The motion compensation unit 60 generates the reconstructed history frame I Ｈｅ , and perform motion compensation to obtain the corrected history frame I Ｈｅ ＭＣ The history complement unit 70 generates the corrected history frame I Ｈｅ ＭＣ The second mask area MA Ｃ Input the pixel values ​​of the part corresponding to the current frame I Ｃ The pixel values ​​of the history frame I are replaced with those of the Ｈｒ Generate.

[0072] According to this configuration, image elements IE whose motion vectors MV are unknown are excluded from motion compensation, and then the image elements IE are restored, thereby more effectively suppressing the occurrence of ghosts and other artifacts caused by motion compensation of the image elements IE.

[0073] The mask area acquisition unit 40 acquires, as post-processing information, the transmittance of a plurality of particles P to be added as an effect. The mask area acquisition unit 40 acquires, as a mask area MA, a low transmittance area within each particle P having a transmittance lower than a set reference.

[0074] This configuration allows the mask area MA to be made as small as possible. Ｈ Since the data is not modified more than necessary, the inference results of DNN80 are of high quality.

[0075] The restoration unit 50 restores pixel values ​​in a range expanded from the low transmittance region to the outside by a predetermined pixel size to the input current frame I Ｃ Replace with the pixel value of

[0076] According to this configuration, the mask area is made slightly wider, which effectively suppresses the occurrence of ghosts and the like.

[0077] The set criterion is set as a condition that maximizes the image quality evaluation value of the inferred current frame.

[0078] This configuration allows the quality of the inference image to be maximized.

[0079] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0080] 3. Example 2 of DNN Processing for Moving Images of the Present Disclosure 3-1. Example of System Configuration FIG. 10 is a diagram showing an example of the configuration of another information processing device 2 that performs DNN processing for moving images of the present disclosure.

[0081] This example differs from the information processing apparatus 1 of FIG. 4 in the following two points. First, the mask area MA is Ｃ and the motion-compensated inference history frame I Ｈ (Simple Correction History Frame I Ｈ ＭＣ ) and the other is the simple correction history frame I Ｈ ＭＣ The part corresponding to the mask area MA of the input current frame I Ｃ Complete history frame I by completing with Ｈｒ The following description will focus on the differences from the information processing device 1.

[0082] The mask region is estimated by the mask region acquisition unit 45 of the information processing device 2. The motion compensation unit 60 receives the inference image of the past frame from the DNN 80 as the inference history frame I. Ｈ The motion compensation unit 60 obtains the inference history frame I based on the motion vector MV′ obtained from the scaler 30. Ｈ The mask region acquisition unit 45 performs motion compensation on the inference history frame I Ｈ Simple correction history frame I Ｈ ＭＣ The simple correction history frame I is obtained as Ｈ ＭＣ is an inference history frame I containing an image element IE. Ｈ The image element IE does not have a motion vector MV. Therefore, the simple correction history frame I Ｈ ＭＣ is an image in which motion compensation has been applied to parts other than the image element IE.

[0083] Simple Correction History Frame I Ｈ ＭＣ The position of the image element IE is the position before motion compensation (inference history frame I Ｈ The simple correction history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask region acquisition unit 45 acquires the image element IE from the simple correction history frame I Ｈ ＭＣ and the input current frame I Ｃ A mask area MA is obtained based on the comparison result with '.

[0084] For example, the mask region acquisition unit 45 acquires the simply corrected history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask area acquisition unit 45 extracts the difference between the effect mask M and the mask area MA. Ｉ and sends it to the history completion unit 70.

[0085] Effect Mask M Ｉis a mask area MA, Ｈ and the second mask area MA Ｃ For example, the effect mask M Ｉ is a binary image in which pixels in the mask area MA are black (pixel value = 0) and pixels outside the mask area MA are white (pixel value = 1). Ｈ ＭＣ The first mask area MA Ｈ and the pixel values ​​of the corresponding parts in the second mask area MA Ｃ Input the pixel values ​​of the part corresponding to the current frame I Ｃ ' and the pixel values ​​of the complementary history frame I Ｈｒ Generate.

[0086] [3-2. Complementing Process of Simply Corrected History Frame] FIG. 11 shows the process of simply corrected history frame I. Ｈ ＭＣ 10A and 10B are diagrams illustrating an example of a complementation process.

[0087] Simple Correction History Frame I Ｈ ＭＣ and the input current frame I Ｃ When comparing the image elements I and I, the image elements I added in post-processing are obtained as differences. However, if differences are simply taken, differences other than the image elements I may also be detected. For example, if a simple correction history frame I is used due to pixel interpolation during motion compensation, Ｈ ＭＣ If the resolution of the image is reduced, differences due to the reduced resolution may be detected. Therefore, a classification process is performed to detect differences due to post-processing.

[0088] For example, the mask region acquisition unit 45 acquires the simply corrected history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask region acquisition unit 45 classifies the differences (inter-frame differences) between the simply corrected history frames I and IB, and determines which of the inter-frame differences are due to post-processing. Ｈ ＭＣ and the input current frame I ＣThe mask area acquisition unit 45 acquires the range of the image that is the difference as a mask area MA.

[0089] An example of a calculation formula for performing the interpolation process is shown in FIG. 11. In the calculation formula, "1-M Ｉ " is Effect Mask M Ｉ This image is an image whose pixel value is the value obtained by subtracting the pixel value (0 or 1) of the effect mask M Ｉ It is an inverted image in which white (pixel value = 1) and black (pixel value = 0) are inverted.

[0090] "(1-M Ｉ ) × I Ｃ " is Effect Mask M Ｉ The pixel values ​​of the inverted image of the input current frame I Ｃ This image is an image whose pixel values ​​are the product of the pixel values ​​of the effect mask M Ｉ The input current frame I of the area other than the area corresponding to the mask area MA Ｃ The image in which the pixel values ​​of the input current frame I are replaced with 0, that is, the ... Ｃ means an image of

[0091] "M Ｉ ×I Ｈ ＭＣ " is Effect Mask M Ｉ Pixel values ​​of the simple correction history frame I Ｈ ＭＣ This image is an image whose pixel values ​​are the product of the pixel values ​​of the effect mask M Ｉ A simple correction history frame I of the area corresponding to the mask area MA Ｈ ＭＣ The image in which the pixel values ​​of the mask area MA are replaced with 0, i.e., the simple correction history frame I Ｈ ＭＣ means an image of

[0092] "(1-M Ｉ ) × I Ｃ +M Ｉ ×I Ｈ ＭＣ" is the input current frame I of the part corresponding to the mask area MA. Ｃ and a simple correction history frame I of the area other than the area corresponding to the mask area MA. Ｈ ＭＣ This summation of images gives a simple correction history frame I Ｈ ＭＣ Input current frame I Ｃ The image incorporating the image element IE of ' (the complementary history frame I Ｈｒ ) is obtained.

[0093] FIG. 12 is a diagram illustrating an example of a classification process for inter-frame differences.

[0094] The mask area acquisition unit 45 acquires the simply corrected history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask area acquisition unit 45 calculates the similarity between corresponding pixels of the mask area MA based on a plurality of determination methods. The mask area acquisition unit 45 applies the calculated similarity obtained by each determination method to an analogy determination criterion to acquire a mask area MA.

[0095] The similarity judgment criterion can be set arbitrarily by the system developer. The similarity judgment criterion may be a threshold indicating a similarity range, or may be a mathematical model that outputs a judgment value related to the similarity judgment. The mathematical model can output a judgment value by statistically processing the similarity obtained by each judgment method. The mathematical model may be expressed as a probability density function or a neural network.

[0096] In the example of Fig. 12, "1-diff" and "cosine similarity" are shown as similarities. "diff" is the similarity of the simple correction history frame I Ｈ ＭＣ and the input current frame I Ｃ The difference in pixel value of each color is calculated for each color, and the sum of the differences in pixel value of each color is divided by the number of colors. Ｈ , G Ｈ , B Ｈ ) is the simple correction history frame I Ｈ ＭＣ indicates the pixel values ​​of the red, green, and blue pixels of the image.Ｃ , G Ｃ , B Ｃ ) is the input current frame I Ｃ RGB represents the pixel values ​​of red (R), green (G), and blue (B) pixels. Note that YUV may be used instead of RGB as a color representation method.

[0097] FIG. 13 is a diagram showing an example of a mathematical model used as an analogy criterion.

[0098] In the example of Fig. 13, the mathematical model is modeled as a two-dimensional probability density function (2D-PDF). In Fig. 13, "μ" is the average of two types of similarity ("1-diff" and "cosine similarity"), and "Σ" is the covariance value of the two types of similarity. "n" is the number of data, and "T" is a constant. In calculating the probability density function, the input current frame I Ｃ and simple correction history frame I Ｈ ＭＣ The number of paired data is acquired, where "n" is the total number of paired data.

[0099] In Fig. 13, "x" is a two-dimensional column vector in which the values ​​of "1-diff" and "cosine similarity" are arranged. In the mathematical model of Fig. 13, the input current frame I Ｃ and simple correction history frame I Ｈ ＭＣ From the pixel values ​​of the pixels, the probability density p is calculated for each pixel.

[0100] The smaller the probability density p, the higher the probability that the image is an effect image, and the larger the probability density p, the lower the probability that the image is an effect image. ｐｄｆ is set. The probability density p is set to the threshold th ｐｄｆ The following area is determined as an effect area, and the probability density p is less than the threshold th ｐｄｆ Any area larger than this is determined as a non-effect area. The effect area is a mask area MA (pixel value "mask" = 0), and the non-effect area is a non-mask area (pixel value "mask" = 1). The system developer can set the threshold th depending on how easily ghosts become apparent. ｐｄｆcan be set appropriately.

[0101] Threshold th ｐｄｆ can also be determined based on the final image quality. ｐｄｆ 14 is a diagram showing an example of a method for determining the similarity criterion based on the final image quality. In the example of FIG. 14, the similarity criterion is the inferred current frame (output image I Ｏ ) is set as a condition for maximizing the image quality evaluation value. As the image quality evaluation method, PSNR, SSIM, etc. are used.

[0102] The image quality is evaluated by the image quality evaluation unit EV. The image quality evaluation unit EV receives a threshold value th ｐｄｆ The image quality evaluation unit EV compares the acquired inference images with the ground truth and determines the threshold th that provides the inference image closest to the ground truth. ｐｄｆ is adopted as the similarity criterion. ｐｄｆ The search can be performed using a method called grid search or Bayesian optimization.

[0103] 3-3. Processing Flow FIG. 15 is a diagram showing an example of a processing flow for performing inference processing.

[0104] The scaler 30 receives the input current frame I from the renderer 20. Ｃ and motion vectors MV. The scaler 30 scales the input current frame I as needed. Ｃ and the number of pixels of the motion vector MV is expanded, and the input current frame I Ｃ ' and motion vector MV' are obtained (step S11).

[0105] The motion compensation unit 60 receives the inference image of the past frame from the DNN 80 as the inference history frame I Ｈ The motion compensation unit 60 obtains the inference history frame I based on the motion vector MV'. Ｈ , and a simple correction history frame I Ｈ ＭＣ (Step S12). The mask region acquisition unit 45 generates the input current frame IＣ ' and simple correction history frame I Ｈ ＭＣ Based on the comparison result with the effect mask M Ｉ is acquired (step S13).

[0106] The history complement unit 70 performs the effect mask M Ｉ Based on the simple corrected history frame I Ｈ ＭＣ The part corresponding to the mask area MA of the input current frame I Ｃ In this way, the history completion unit 70 can obtain the completed history frame I ′, which is improved in terms of ghosting and brightness reduction. Ｈｒ is acquired (step S14).

[0107] The scaler 30 and the history complement unit 70 calculate the input current frame I Ｃ ' and the complementary history frame I Ｈｒ is input to the trained DNN 80 (step S15). The DNN 80 outputs the output image I Ｏ , the final output, and the inference history frame I for the next video processing. Ｈ (Step S16) After that, the recursive processing of the RNN is repeated.

[0108] [3-4. Effects] The mask region acquisition unit 45 acquires an inference history frame I including an image element IE. Ｈ A simple correction history frame I obtained by performing motion compensation on Ｈ ＭＣ and the input current frame I Ｃ A mask area MA is obtained based on the comparison result.

[0109] According to this configuration, even if post-processing information is not available from the renderer 20, it is possible to identify the image element IE to be removed.

[0110] The history complement unit 70 generates a simple corrected history frame I Ｈ ＭＣ The first mask area MA Ｈ and the pixel values ​​of the corresponding parts in the second mask area MA ＣInput the pixel values ​​of the part corresponding to the current frame I Ｃ The pixel values ​​of the history frame I are replaced with those of the Ｈｒ Generate.

[0111] According to this configuration, problems related to both ghosts and reduced brightness can be eliminated simultaneously through a single process.

[0112] The mask area acquisition unit 45 acquires the simply corrected history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask area acquisition unit 45 acquires the range of the image that is the difference as a mask area MA.

[0113] According to this configuration, it is possible to easily identify the image element IE to be removed.

[0114] The mask area acquisition unit 45 acquires the simply corrected history frame I Ｈ ＭＣ and the input current frame I Ｃ The mask area acquisition unit 45 calculates the similarity between corresponding pixels of the mask area MA based on a plurality of determination methods. The mask area acquisition unit 45 applies the calculated similarity obtained by each determination method to an analogy determination criterion to acquire a mask area MA.

[0115] According to this configuration, it is possible to accurately identify the image element IE to be removed.

[0116] The analogy criterion is set as a condition that maximizes the image quality evaluation value of the inferred current frame.

[0117] This configuration allows the quality of the inference image to be maximized.

[0118] 4. Example of Hardware Configuration FIG. 16 is a diagram showing an example of the hardware configuration of the information processing devices 1 and 2. As shown in FIG.

[0119] The information processing of the information processing devices 1 and 2 is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0120] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0121] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0122] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0123] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0124] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0125] For example, when the computer 1000 functions as the information processing devices 1 and 2 according to the embodiments, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0126] [Additional Notes] The present technology may also be configured as follows. (1) An information processing device comprising: a mask region acquisition unit that identifies image elements for which motion vectors used for motion compensation are unknown in an inference history frame, which is an inference image of a past frame, and acquires regions corresponding to the image elements as mask regions; a history completion unit that acquires a complemented history frame in which a portion of the inference history frame after the motion compensation, which corresponds to the mask region, is complemented based on an input current frame; and a DNN that outputs an inference current frame based on the input current frame and the complemented history frame. (2) The information processing device described in (1), wherein the mask region acquisition unit acquires, as the mask regions, a first mask region that indicates the range of the image element in the inference history frame and a second mask region that indicates the range of the image element in the input current frame. (3) The information processing device described in (2), wherein the mask region acquisition unit acquires the mask regions based on information of post-processing related to generation of the image elements. (4) The information processing device according to (3), comprising: a restoration unit that generates a restored history frame by replacing pixel values ​​of a portion of the inferred history frame corresponding to the first mask region with pixel values ​​of the input current frame; and a motion compensation unit that performs the motion compensation on the restored history frame to generate a corrected history frame, wherein the history complement unit generates the complemented history frame by replacing pixel values ​​of a portion of the corrected history frame corresponding to the second mask region with pixel values ​​of the input current frame. (5) The information processing device according to (4), wherein the mask region acquisition unit acquires, as the post-processing information, transmittances of a plurality of particles to be added as effects, and acquires, as the mask region, a low-transmittance region within each particle having transmittance lower than a set reference. (6) The information processing device according to (5), wherein the restoration unit replaces pixel values ​​in a range extended outward from the low-transmittance region by a predetermined pixel size with pixel values ​​of the input current frame.(7) The information processing device according to (5) or (6), wherein the setting criterion is set as a condition for maximizing an image quality evaluation value of the inferred current frame. (8) The information processing device according to (2), wherein the mask region acquisition unit acquires the mask region based on a comparison result between a simply corrected history frame obtained by performing the motion compensation on the inferred history frame including the image element and the input current frame. (9) The information processing device according to (8), wherein the history complement unit generates the complemented history frame by replacing pixel values ​​of a portion of the simply corrected history frame corresponding to the first mask region and pixel values ​​of a portion of the simply corrected history frame corresponding to the second mask region with pixel values ​​of the input current frame. (10) The information processing device according to (8) or (9), wherein the mask region acquisition unit extracts a difference resulting from post-processing related to generation of the image element from the simply corrected history frame and the input current frame, and acquires the range of the image resulting in the difference as the mask region. (11) The information processing device according to (10), wherein the mask region acquisition unit calculates similarities between corresponding pixels of the simply corrected history frame and the input current frame based on a plurality of determination methods, and acquires the mask region by applying the calculated similarities obtained by each determination method to an analogy determination criterion. (12) The information processing device according to (11), wherein the analogy determination criterion is set as a condition for maximizing an image quality evaluation value of the inferred current frame. (13) An information processing method executed by a computer, comprising: identifying image elements for which motion vectors used for motion compensation are unknown in an inferred history frame that is an inferred image of a past frame; acquiring a region corresponding to the image element as a mask region; acquiring an interpolated history frame in which a portion of the inferred history frame after the motion compensation that corresponds to the mask region is interpolated based on the input current frame; and outputting the inferred current frame based on the input current frame and the interpolated history frame.(14) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: identify image elements for which motion vectors used for motion compensation are unknown in an inference history frame, which is an inference image of a past frame; acquire an area corresponding to the image element as a mask area; acquire a complementary history frame in which a portion of the inference history frame after the motion compensation that corresponds to the mask area is complemented based on an input current frame; and output an inference current frame based on the input current frame and the complementary history frame.

[0127] 1, 2 Information processing device 40, 45 Mask area acquisition unit 50 Restoration unit 60 Motion compensation unit 70 History completion unit 80 DNN I Ｃ , I Ｃ ' Input current frame IE image element I Ｈ Inference History Frame I Ｈｅ Restore History Frame I Ｈｅ ＭＣ Correction History Frame I Ｈｒ Complementary History Frame I Ｈ ＭＣ Simple correction history frame M, M Ｃ , M Ｈ , M Ｉ Effect Mask MA Mask Area MA Ｃ Second mask area MA Ｈ First mask area MV, MV' Motion vector P Particle

Claims

1. An information processing device having: a mask area acquisition unit that identifies image elements for which the motion vector used for motion compensation is unknown in an inference history frame, which is an inference image of a past frame, and acquires an area corresponding to the image element as a mask area; a history completion unit that acquires a complemented history frame in which a portion of the inference history frame after the motion compensation that corresponds to the mask area is complemented based on an input current frame; and a DNN that outputs an inference current frame based on the input current frame and the complemented history frame.

2. The information processing device according to claim 1, wherein the mask area acquisition unit acquires as the mask area a first mask area indicating the range of the image element in the inference history frame and a second mask area indicating the range of the image element in the input current frame.

3. The information processing device according to claim 2, wherein the mask region acquisition unit acquires the mask region based on information of post-processing related to generation of the image element.

4. An information processing device as described in claim 3, comprising: a restoration unit that replaces pixel values ​​of a portion of the inferred history frame corresponding to the first mask region with pixel values ​​of the input current frame to generate a restored history frame; and a motion compensation unit that performs the motion compensation on the restored history frame to generate a corrected history frame, wherein the history complement unit replaces pixel values ​​of a portion of the corrected history frame corresponding to the second mask region with pixel values ​​of the input current frame to generate the complemented history frame.

5. The information processing device according to claim 4, wherein the mask area acquisition unit acquires the transmittance of a plurality of particles to be added as an effect as information for the post-processing, and acquires, as the mask area, a low transmittance area within each particle having a transmittance lower than a set standard.

6. The information processing device according to claim 5, wherein the restoration section replaces pixel values ​​in a range expanded outward from the low transmittance region by a predetermined pixel size with pixel values ​​of the input current frame.

7. The information processing apparatus according to claim 5, wherein the set criterion is set as a condition for maximizing an image quality evaluation value of the inferred current frame.

8. The information processing device according to claim 2, wherein the mask area acquisition unit acquires the mask area based on a comparison result between a simply corrected history frame obtained by performing the motion compensation on the inference history frame including the image element and the input current frame.

9. The information processing device according to claim 8, wherein the history complement unit generates the complemented history frame by replacing pixel values ​​of a portion of the simply corrected history frame corresponding to the first mask region and pixel values ​​of a portion of the simply corrected history frame corresponding to the second mask region with pixel values ​​of the input current frame.

10. The information processing device according to claim 8, wherein the mask area acquisition unit extracts differences resulting from post-processing related to the generation of the image elements from the simply corrected history frame and the input current frame, and acquires the range of the image that is the difference as the mask area.

11. The information processing device according to claim 10, wherein the mask area acquisition unit calculates the similarity between corresponding pixels of the simply corrected history frame and the input current frame based on a plurality of judgment methods, and acquires the mask area by applying the calculated similarity for each judgment method to an analogy judgment criterion.

12. The information processing device according to claim 11, wherein the analogy judgment criterion is set as a condition that maximizes an image quality evaluation value of the inferred current frame.

13. An information processing method executed by a computer, comprising: identifying an image element for which a motion vector used for motion compensation is unknown in an inference history frame, which is an inference image of a past frame; acquiring an area corresponding to the image element as a mask area; acquiring a complementary history frame in which a portion of the inference history frame after the motion compensation corresponding to the mask area is complemented based on an input current frame; and outputting an inference current frame based on the input current frame and the complementary history frame.

14. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: identify image elements for which the motion vector used for motion compensation is unknown in an inference history frame, which is an inference image of a past frame; obtain an area corresponding to the image element as a mask area; obtain a complementary history frame in which a portion of the inference history frame after the motion compensation that corresponds to the mask area is complemented based on an input current frame; and output an inference current frame based on the input current frame and the complementary history frame.