Information processing apparatus, information processing method, and computer-readable non-transitory storage medium

Abstract

The information processing apparatus includes a mask region acquisition unit, a history supplement unit, and a DNN. A mask region acquisition unit recognizes an image element in which a motion vector for motion compensation is unknown in an inference image of an inference history frame, i.e., a past frame. A mask region acquisition unit acquires a region corresponding to the image element as a mask region. A history supplement unit acquires a supplement history frame in which a portion corresponding to the mask region in the motion-compensated inference history frame is supplemented on the basis of the input current frame. The DNN outputs an inference current frame based on the input current frame and the supplemental historical frame.

Description

Technical Field

[0001] This invention relates to an information processing apparatus, an information processing method, and a computer-readable non-transitory storage medium. Background Technology

[0002] Deep neural network (DNN) processing for moving images typically employs a recurrent neural network (RNN) structure, which offers high time stability. This is because the RNN structure can correlate the current frame (current information) with historical frames (past information), thus making it easier to maintain consistency in the output video.

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: JP 2022-547517 W Summary of the Invention

[0006] Technical issues

[0007] Historical frames are used as input to the DNN along with the current frame. To maintain consistency with the current frame, historical frames undergo corrections that take into account motion between frames; this is called motion compensation. Motion compensation is performed as a process that predicts the current frame's data based on motion between frames from past frames. However, motion compensation cannot be performed because motion vectors cannot be obtained for image elements (e.g., effects) added in post-processing. Data perturbation occurs in the historical frames after motion compensation, and this affects the DNN's inference results.

[0008] Therefore, this disclosure provides information processing apparatus, information processing method and computer-readable non-transitory storage medium that can improve video quality.

[0009] Solution to the problem

[0010] According to this disclosure, an information processing apparatus is provided, comprising: a mask region acquisition unit configured to identify an image element with an unknown motion vector for motion compensation in an inference history frame, which is an inference image of past frames, and acquire a region corresponding to the image element as a mask region; a history supplementation unit configured to acquire a supplementary history frame in the motion-compensated inference history frame, wherein a portion corresponding to the mask region is supplemented based on an input current frame; and a DNN configured to output an inference current frame based on the input current frame and the supplementary history frame. According to this disclosure, an information processing method is provided in which the information processing of the information processing apparatus is performed by a computer, and a computer-readable non-transitory storage medium is provided storing a program for causing the computer to perform the information processing of the information processing apparatus. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating conventional DNN processing for dynamic images.

[0012] Figure 2 It is a graph used to illustrate data perturbations caused by motion compensation.

[0013] Figure 3 This diagram is used to illustrate the occurrence of ghosting due to motion compensation.

[0014] Figure 4 This is a diagram illustrating an example configuration of an information processing apparatus that performs DNN processing for moving images according to the present disclosure.

[0015] Figure 5 This is a diagram used to illustrate a technique for improving defects caused by motion compensation of image elements.

[0016] Figure 6 This is a diagram used to illustrate an example of image element removal processing.

[0017] Figure 7 This is a diagram used to illustrate an example of image element removal processing.

[0018] Figure 8 This is a diagram illustrating an example of a method for determining a threshold used as a setting criterion based on the final image quality.

[0019] Figure 9 This is a diagram illustrating an example of the processing flow used to perform inference processing.

[0020] Figure 10 This is a diagram illustrating an example configuration of another information processing apparatus for performing DNN processing for moving images according to the present disclosure.

[0021] Figure 11 This is a diagram illustrating an example of supplementary processing for simple corrected historical frames.

[0022] Figure 12 This is a diagram illustrating an example of how inter-frame differences are classified and processed.

[0023] Figure 13 This is a diagram illustrating an example of a mathematical model used as a criterion for similarity determination.

[0024] Figure 14 This is a diagram illustrating an example of a method for determining a threshold to define the range of the effect area based on the final image quality.

[0025] Figure 15 This is a diagram illustrating an example of the processing flow used to perform inference processing.

[0026] Figure 16This is a diagram illustrating an example of the hardware configuration of an information processing device. Detailed Implementation

[0027] In the following, embodiments according to this disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, the same parts are indicated by the same reference numerals, and redundant descriptions will be omitted.

[0028] Note that the descriptions will be given in the following order.

[0029] [1. Background]

[0030] [1-1. Conventional DNN processing for dynamic images]

[0031] [1-2. Data perturbations caused by motion compensation]

[0032] [2. A first example of DNN processing for dynamic images according to this disclosure]

[0033] [2-1. System Configuration Example]

[0034] [2-2. Image Element Removal Processing]

[0035] [2-3. Processing Flow]

[0036] [2-4. Effects]

[0037] [3. A second example of DNN processing for dynamic images based on this disclosure]

[0038] [3-1. System Configuration Example]

[0039] [3-2. Supplementary processing for simple correction of historical frames]

[0040] [3-3. Processing Flow]

[0041] [3-4. Effects]

[0042] [4. Hardware Configuration Example]

[0043] [1. Background]

[0044] [1-1. Conventional DNN processing for dynamic images]

[0045] Figure 1 This is a diagram illustrating conventional DNN processing for dynamic images.

[0046] DNN processing for dynamic images typically employs RNN structures with high temporal stability. Because RNN structures can correlate the current frame with historical frames, they easily maintain the consistency of the output video. While DNNs exhibit very high inference performance (e.g., sharpening in super-resolution scenarios), they are susceptible to small changes in the input data, which can alter the output. Therefore, flickering is more likely to occur in dynamic image processing compared to simple filtering. To enhance temporal stability and suppress flickering, RNN structures are typically implemented that can learn temporal correlations (the similarity between the current and historical frames) by simultaneously inputting both the current and historical frames.

[0047] Note that the current frame refers to the image of the frame to be estimated by the DNN (current frame). A history frame refers to an image obtained by inputting images of past frames into the DNN (inference image of past frames). A past frame refers to a frame that precedes the current frame (one or more frames). In this disclosure, for example, a frame that precedes the current frame (latest frame) is used as a past frame. Hereinafter, the current frame may be referred to as the "input current frame," and history frames that serve as inference images of past frames may be referred to as "inference history frames."

[0048] The inference history frame, along with the current input frame, is used as input to the DNN. To maintain consistency with the current input frame, motion compensation based on motion vectors is performed on the inference history frame. Motion vectors refer to vectors indicating the movement of the subject (the amount and direction of pixel movement) between frames. Motion compensation refers to the process of predicting data after motion based on data before motion. For example, when motion compensation is applied to images of past frames, the predicted image in the current frame is obtained as the motion-compensated inference history frame. The motion-compensated inference history frame is then input to the DNN along with the current input frame.

[0049] [1-2. Data perturbations caused by motion compensation]

[0050] Figure 2 and Figure 3 It is a graph used to illustrate data perturbations caused by motion compensation.

[0051] Figure 2 This example illustrates a rendered image where an image element (IE) is added to a 3D object as post-processing. The renderer acquires the motion vector (MV) of the 3D object during rendering. Since the image element (IE) is arbitrarily added in post-processing, its motion vector (MV) is not present. Motion compensation is performed only for the 3D object. Therefore, the position of the motion-compensated image element (IE) may deviate from its actual position as shown in the input current frame.

[0052] For example, Figure 3This example illustrates a 3D object, acting as the main subject, moving diagonally to the upper right as the frame progresses. In typical renderers, velocity information (motion vector MV) is maintained in association with the subject. The distance to the subject (depth) is usually acquired simultaneously with the velocity information.

[0053] Post-processing is performed on the rendered image. Figure 3 In the examples, image elements (IEs) used as effects are added to the rendered image. Examples of effects include those caused by video overlay and those caused by video processing (defocusing and color processing). Effects caused by video overlay include bloom, particles, reflections, and transmission. Effects generated by video processing include depth of field (DoF), motion blur, tone mapping, and lens flare. By performing post-processing on the rendered image, the final output image includes image elements (IEs) whose motion is unknown.

[0054] The renderer obtains the subject's motion vector MV based on the subject's motion and the rendering viewpoint's motion. The motion of image elements (IEs) is unknown because IEs are not objects existing in 3D space (e.g., objects with meshes) and are optionally added in post-processing. There is no motion vector MV for IEs, and the motion vector MV is obtained only for the subject. Because IEs do not have a motion vector MV, the position of a motion-compensated IE does not change from its position before motion compensation. Therefore, the position of an IE may shift between the input current frame and the motion-compensated inference history frame.

[0055] The DNN outputs the inference image obtained by mixing the current input frame and motion-compensated inference history frames as the inference result. Therefore, in the inference image, at the locations where image elements (IEs) are displayed in the current input frame and in the inference history frames, image elements (IEs) with lower brightness than their original display states are displayed. This results in a brightness reduction problem at locations where image elements (IEs) should be reproduced, and a ghosting problem (including afterimages of image elements (IEs) in the inference history frames) at locations where they should not be reproduced.

[0056] This disclosure aims to improve upon defects such as ghosting and reduced brightness in inference images. In this disclosure, regions where defects may occur during motion compensation are obtained based on post-processing information related to the generation of image elements (IEs) or information about the differences between the input current frame and motion-compensated inference history frames. The obtained regions are used as mask regions to mask the data to be corrected. Motion-compensated inference history frames (supplementary history frames) of the image, with the portions corresponding to the mask regions corrected, are input to the DNN along with the input current frame.

[0057] Correction is performed using data from the current input frame. Therefore, positional deviations of image elements (IEs) between the current input frame and supplementary historical frames are virtually eliminated. This suppresses the aforementioned defects in inference images and yields high-quality DNN inference results. The techniques of this disclosure will be described in detail below. An example of an effect image where the image element (IE) is added in post-processing is described below. However, the techniques of this disclosure can be applied not only to effect images but also, in general, to image elements (IEs) where the motion vector (MV) is unknown.

[0058] [2. A first example of DNN processing for dynamic images according to this disclosure]

[0059] [2-1. System Configuration Example]

[0060] Figure 4 This is a diagram illustrating an example configuration of an information processing apparatus 1 that performs DNN processing for moving images according to the present disclosure. Figure 5 This is a diagram used to illustrate a technique for improving defects caused by motion compensation of image elements (IE).

[0061] In this example, motion compensation is performed after removing image element IEs from the inference history frames, and image element IEs are then replenished for the motion-compensated image. By removing and replenishing image element IEs, factors such as ghosting are directly eliminated. The DNN processing for moving images according to this disclosure is performed by an information processing apparatus 1. For example, the information processing apparatus 1 includes an input unit 10, a renderer 20, a scaler 30, a mask region acquisition unit 40, a restoration unit 50, a motion compensation unit 60, a history replenishment unit 70, a DNN 80, and an output unit 90.

[0062] Input unit 10 inputs operation signals S to renderer 20. For example, operation signals S include rendering settings and rendering execution instructions.

[0063] Renderer 20 performs rendering processing based on operation signal S. Renderer 20 outputs the rendered image as input to the current frame I according to the rendering execution command. C Input the current frame I C It refers to a specific frame of an image within a series of frames, such as a moving image. Figure 4 The block diagram illustrates the operation of a frame in a moving image.

[0064] Renderer 20 obtains the motion vector MV of the subject and the alpha mask TP of the image element IE based on the rendering settings. The motion vector MV is a two-dimensional vector in which the movement of corresponding pixels between frames is defined by the amount of pixel movement (the number of pixels). The motion vector MV can be obtained through a general estimation process using consecutive frame groups. When using a CG renderer, the motion vector MV can be generated and obtained during rendering. The alpha mask TP is an image indicating the distribution of the transmittance of the image element IE. The pixel value of each pixel in the alpha mask TP indicates the transmittance of the image element IE.

[0065] Scaler 30 increases the input current frame I as needed. C The number of pixels for motion vector MV and transparency mask TP. In the following text, when it is necessary to distinguish between data after the pixel count has been increased by scaler 30 and data before the increase, an apostrophe (') will be added after the data symbol. Scaler 30 will input the current frame I. C The data is sent to the recovery unit 50, the history supplement unit 70, and the DNN 80. The scaler 30 sends the motion vector MV to the motion compensation unit 60. The scaler 30 sends the transparency mask TP to the mask region acquisition unit 40.

[0066] Given that the number of pixels received by the DNN 80 architecture is fixed, and on the other hand, the input current frame I... C The number of pixels varies depending on the usage method. The scaler 30 can perform pixel number expansion processing at the desired magnification to obtain a pixel number conforming to the DNN 80 specification. The pixel number expansion method can be arbitrarily chosen, such as a bilinear or bicubic method. Regarding the expansion method of the motion vector MV, to avoid intermediate values ​​due to pixel supplementation, it is desirable to apply a nearest neighbor method, but the invention is not limited thereto.

[0067] When considering the reasoning history frame I H During motion compensation, the mask region acquisition unit 40 acquires regions that may be defective due to image elements IE with unknown motion vectors as mask regions MA. Faults may occur in inference history frames I. H The position of the image element IE (ghosting) and the input current frame I C The image element's position in IE (brightness reduced).

[0068] For example, the mask region acquisition unit 40 in the inference history frame I H The image element IE with an unknown motion vector is identified. The mask region acquisition unit 40 acquires the region corresponding to the image element IE as a mask region MA. The mask region acquisition unit 40 includes a first mask region MA as the mask region MA. H and the second mask region MAC First mask region MA H Indicator Reasoning History Frame I H The range of the image element IE. Second mask region MA. C Indicates input current frame I C The range of image elements in IE.

[0069] The mask region acquisition unit 40 can acquire the mask region MA based on post-processing information related to the generation of the image element IE. For example, the mask region acquisition unit 40 acquires the transparency mask TP' as post-processing information related to the generation of the image element IE. The transparency mask TP' includes information related to the position of the image element IE. The mask region acquisition unit 40 can estimate the region where defects may occur during motion compensation based on the position of the image element IE. The mask region acquisition unit 40 acquires the estimated region as the mask region MA used to mask the data to be corrected.

[0070] For example, the mask region acquisition unit 40 converts the transparency mask TP' of the current frame acquired by the scaler 30 into an effect mask M. C The mask region acquisition unit 40 generates the effect mask M for the current frame. C The mask is sent to the history supplement unit 70. The mask region acquisition unit 40 acquires the effect mask M of the previous frame from the previous frame. H Send to recovery unit 50.

[0071] Effect Mask M C It indicates the range of the image element to be removed (IE) as the second mask area (MA). C The image. For example, effect mask M. C It is a binary image where pixels of the portion to be removed are black (pixel value = 0) and pixels of the portion not to be removed are white (pixel value = 1). The image area indicated by black is the extent of the image element IE to be removed (second mask area MA). C The transformation process is performed by binarizing the pixel values ​​of the transparency mask TP'. The mask region acquisition unit 40 can perform a second mask region magnification MA as needed. C The processing is used to increase the range of the image to be corrected.

[0072] Recovery unit 50 obtains inference images of past frames from DNN 80 as inference history frames I. H Recovery unit 50 retrieves inference history frame I. H Remove unknown image elements with motion vectors (MV) between frames in IE to generate a restored history frame I He The removal of image elements (IEs) is performed by replacing the IE with a supplementary image. The supplementary image can be a frame that does not impair the inference history.H Any image of the atmosphere. For example, recovery unit 50 restores the inference history frame I by... H The region corresponding to the first mask region MA H The pixel values ​​of a portion are replaced with the input current frame I. C The pixel values ​​are used to generate the restored historical frame I. He .

[0073] Figure 5 An example of a calculation formula for removing image elements in Internet Explorer is shown. In the calculation formula, " "This refers to an effect mask M that is created by subtracting past frames from 1." H The image is obtained by taking the pixel value (0 or 1) as the pixel value. This image is created by using the effect mask M of past frames. H The inverted image is obtained by inverting the white (pixel value=1) and black (pixel value=0) pixels.

[0074] Here, " This means that its pixel value is the effect mask M. H The pixel values ​​of the inverted image are compared with the input current frame I. C An image representing the product of the pixel values ​​of '. This image refers to the input of the current frame I. C 'The effect mask M corresponding to the past frame is excluded. H The mask region MA (first mask region MA) H The image whose pixel values ​​are replaced with 0 for all parts except the current frame I is input. C ' corresponds to the first mask region MA H The image is a partial image.

[0075] “ This means that its pixel values ​​are the effect mask M from past frames. H Pixel values ​​and inference history frames I H An image of the product of pixel values. This image refers to the inference history frame I. H The effect mask M corresponding to past frames H The mask region MA (first mask region MA) H The image in which part of the pixel value is replaced with 0, i.e., the inference history frame I. H The remainder corresponds to the first mask region MA H The image of the part other than the part of the image.

[0076] also," "This means by inputting the current frame I..." C ' corresponds to the first mask region MA H Partial images and reasoning history frames I H The remainder corresponds to the first mask region MAH The image is obtained by adding the images of the parts other than the part in the inference history frame I. By adding these images, we obtain the image that will be included in the inference history frame I. H The pixel values ​​of the image elements in IE are replaced with the input current frame I. C The image with pixel values ​​of ' (restoring historical frames I) He ).

[0077] Motion compensation unit 60 obtains the restored historical frame I from restoration unit 50 He The motion compensation unit 60 reconstructs historical frames I based on motion vector MV'. He Perform motion compensation to generate corrected historical frames I He MC The motion compensation unit 60 can apply general processing known as motion compensation to restore historical frame I. He .

[0078] Motion vector MV' has the same properties as when recovering historical frames I He To input current frame I C The vector related to the spatial movement of '. Historical frame I is recovered due to motion compensation. He and input current frame I C The corresponding pixels of ' ' overlap with each other. The motion compensation unit 60 restores the historical frame I after motion compensation. He (Correcting historical frames I) He MC Send to the historical supplement unit 70.

[0079] The history supplement unit 70 acquires the image of the current frame as input to the current frame I. C '. Historical Supplement Unit 70 acquires supplementary historical frames I' Hr This supplementary historical frame I Hr It is by using motion-compensated inference history frames I H The portion corresponding to the mask region MA in the input current frame I is based on the input current frame I. C 'Obtained through supplementation. For example, the history supplementation unit 70 will correct historical frame I.' He MC The corresponding region MA of the second mask C The pixel values ​​of a portion are replaced with the input current frame I. C The pixel value of '. Therefore, the history supplementation unit 70 will obtain the pixel value from the input current frame I. C Image elements extracted from 'IE are added to the correction history frame I' He MC The generated image serves as a supplementary historical frame I Hr .

[0080] Figure 5An example of a calculation formula for supplementing image elements (IE) is shown. In the calculation formula, " "This means having an effect mask M that is obtained by subtracting the current frame from 1." C The image is obtained by taking the pixel value (0 or 1) as the pixel value. This image is obtained by using the effect mask M of the current frame. C The inverted image is obtained by inverting the white (pixel value=1) and black (pixel value=0) pixels.

[0081] Here, " This means that its pixel value is the effect mask M of the current frame. C The pixel values ​​of the inverted image are compared with the input current frame I. C An image representing the product of the pixel values ​​of '. This image refers to the input of the current frame I. C 'The effect mask M corresponding to the current frame is '' C The mask region MA (second mask region MA) C The image whose pixel values ​​are replaced with 0 for all pixels except those in the current frame I is input. C ' corresponds to the second mask region MA C The image is a partial image.

[0082] “ This means that its pixel value is the effect mask M of the current frame. C Pixel values ​​and correction history frames I He MC An image of the product of pixel values. This image refers to the historical frame I that will be corrected. He MC The effect mask M corresponding to the current frame C The mask region MA (second mask region MA) C The image in which part of the pixel value is replaced with 0 is used to correct the historical frame I. He MC The remainder corresponds to the second mask region MA. C The image of the part other than the part of the image.

[0083] also," "This means by inputting the current frame I..." C ' corresponds to the second mask region MA C Partial images and correction history frames I He MC The remainder corresponds to the second mask region MA. C The image is obtained by adding the images of the parts other than the input frame I. By adding these images, we obtain the image where the current input frame I is located. C The image element IE was added to the correction history frame I. He MC Images (Supplementary historical frame I)Hr Image elements in IE supplement historical frames. Hr The position in the input current frame I C 'Overlap.' Therefore, ghosting and brightness reduction after DNN processing are suppressed.

[0084] DNN 80 is based on the input current frame I C 'and supplementary historical frame I Hr Perform inference on the current frame. The DNN 80 outputs the inferred image of the current frame obtained through inference as the inferred current frame. The inferred current frame is output as image I. O It is sent to output unit 90. Output unit 90 processes the output image I obtained from DNN 80. O General post-processing such as color conversion and encoding / decoding is applied. As a task of DNN 80, super-resolution, noise reduction, and painting style transfer are assumed, but this disclosure is not limited to these. DNN 80 can be applied to general dynamic image processing using RNN structures.

[0085] As mentioned above, in this example, from inference history frame I H Restore images from before the addition of image elements in IE and restore the restored images (Restore History Frames). He Perform motion compensation. Then, input the current frame I. C Image elements IE are added to historical frames obtained through motion compensation (corrected historical frames I). He MC ), and the obtained images (Supplementary History Frame I) Hr ) and input current frame I C Together, they are input into DNN 80. Therefore, image elements (IE) are suppressed when inputting the current frame I. C With supplementary historical frame I Hr This reduces the occurrence of positional deviations between the two. Therefore, problems such as ghosting and reduced brightness are suppressed, and high-quality DNN inference results can be obtained.

[0086] In the conventional method described in Patent Document 1, disturbed portions (noise regions) of the video are detected from the inference image, and the DNN is optimized using a loss function weighted towards the noise regions. However, in this method, optimization of non-noise regions is relatively weakened, and inference accuracy deteriorates. Furthermore, even using a dedicated recognition model, it is difficult to detect noise regions from the inference image. For example, by using simple differences between consecutive frames, it is difficult to detect subjects with small differences in brightness or color (e.g., transparency effects superimposed in post-processing). Therefore, appropriate loss weighting is also challenging.

[0087] On the other hand, in the technique of this example, the cause of ghosting is directly improved by removing and supplementing the image element IE, which is the problem. Furthermore, in the technique of this example, Figure 2 The brightness reduction of image elements (IE) in the inference image shown is also suppressed. Therefore, better results can be obtained than with conventional methods.

[0088] [2-2. Image Element Removal Processing]

[0089] Figure 6 and Figure 7 This is an example diagram illustrating the removal process of an image element (IE). In this example, as an effect, particle P is used to generate video. Particle P has a transmittance distribution with increasing transparency from the center outwards. Particle P is used to generate sparks in a battle scene.

[0090] The renderer 20 includes a rendering engine 21 and a post-processing unit 22. The post-processing unit 22 performs post-processing on the image (rendered image) output from the rendering engine to add particles P.

[0091] The mask region acquisition unit 40 acquires the transparency mask TP from the post-processing unit 22 as post-processing information. The mask region acquisition unit 40 also acquires the transmittance of multiple particles P added as effects from the transparency mask TP. The mask region acquisition unit 40 identifies low transmittance regions where the transmittance of each particle P is lower than a set standard as inference history frames I. H The image element IE to be removed is selected. The mask region acquisition unit 40 sets the low transmittance region as the mask region MA (pixel value = 0) and sets the region other than the low transmittance region as the non-mask region (pixel value = 1). Therefore, the mask region acquisition unit 40 converts the transparency mask TP into the effect mask M.

[0092] The setting criteria are based on the ease with which ghosting occurs. For example, when the effect's transmittance is high (e.g., 95%), most pixel values ​​are taken as background values. Therefore, issues such as ghosting and reduced brightness are almost nonexistent in the DNN 80 inference results. When the effect's transmittance is low (e.g., 10%), the effect is clearly displayed, and therefore, ghosting may occur in the DNN 80 inference results. Therefore, the setting criteria can be based on the magnitude of the effect's transmittance.

[0093] exist Figure 7In the example, the transmittance of the transparency mask TP is binarized based on a threshold th. For instance, regions with transmittance equal to or less than the threshold th are detected as areas prone to ghosting (low transmittance regions). If the threshold th is too high, an unnecessarily wide range is detected as a target to be removed. If the threshold th is too low, the portion detected as a target to be removed cannot be sufficiently detected. System developers can appropriately set the threshold th based on the likelihood of ghosting.

[0094] The mask region acquisition unit 40 sets low-transmittance regions with low transmittance and prone to ghosting as removal targets, and removes high-transmittance regions with high transmittance and almost no ghosting from the removal targets. In the RNN, high-quality inference history frames I output from the DNN 80 are reused. H It is advantageous, and it is not expected to require more than necessary modifications to the reasoning history frame I. H Therefore, only low-transmittance areas prone to ghosting are set as targets for removal.

[0095] The threshold th can also be determined based on the final image quality. Alternatively, the mask region MA can be detected based on machine learning, in which the current frame and the corrected historical frame are used as student data, and the effect mask M created based on the transparency mask TP is used as teacher data. Figure 8 This is a diagram illustrating an example of a method for determining the threshold th used as a setting criterion based on the final image quality.

[0096] exist Figure 8 In the example, the criterion is set to make the inference current frame (output image I) output from DNN 80 the correct value. O The condition for maximizing the image quality assessment value is (grid search). Peak signal-to-noise ratio (PSNR), structural similarity (SSIM), etc., are used as image quality assessment methods.

[0097] Image quality is evaluated by an image quality evaluation unit (EV). The image quality evaluation unit (EV) may be included in the information processing device 1 or may be set independently of the information processing device 1. The image quality evaluation unit (EV) acquires multiple inference images with different thresholds (th) from the DNN 80. The image quality evaluation unit (EV) compares the acquired inference images with the ground truth and adopts the threshold (th) obtained when the inference image closest to the ground truth is obtained as the setting criterion. The search for the threshold (th) can be achieved using general techniques such as Bayesian optimization with fewer iterations.

[0098] Recovery unit 50 can retrieve inference history frames I HThe range extending outward from the low-transmittance region by a predetermined pixel size is removed to more reliably prevent ghosting. For example, the recovery unit 50 can replace the pixel values ​​in the range extending outward from the low-transmittance region by a predetermined pixel size with the input current frame I. C The pixel values ​​are used to represent the particle P. The particle P is plotted as an image where the transmittance gradually changes from the center to the periphery. Although not as much as in the center, the outer portion of particle P can still cause interference in the video. By extending the removal range to such portions, ghosting can be further suppressed.

[0099] [2-3. Processing Flow]

[0100] Figure 9 This is a diagram illustrating an example of the processing flow used to perform inference processing.

[0101] Scaler 30 obtains the input current frame I from renderer 20. C Motion vector (MV) and transparency mask (TP). Scaler 30 zooms in on the current input frame (I) as needed. C The number of pixels for motion vector MV and transparency mask TP, and the input current frame I. C 'Motion Vector MV' and Transparency Mask TP' (Step S1).

[0102] The mask region acquisition unit 40 acquires the effect mask M of the current frame based on the transparency mask TP' of the current frame and the transparency mask TP' of past frames. C Effect mask M of past frames H (Step S2). The recovery unit 50 is based on the input current frame I. C '、Inference History Frame I H Effect mask M of past frames H From the reasoning history frame I H The effect is removed. Therefore, the recovery unit 50 restores the rendered image of past frames (restoring historical frames I) before applying the effect. He (Step S3).

[0103] The motion compensation unit 60 applies motion compensation based on motion vector MV' to restore historical frame I. He Therefore, the motion compensation unit 60 obtains the position of the main body relative to the input current frame I. C 'Matching Correction History Frame I' He MC (Step S4). The history supplement unit 70 uses the effect mask M based on the current frame. C Input current frame I C The effect is added to the correction history frame I. He MC Therefore, the historical supplementation unit 70 acquires the supplementary historical frame I, which is supplemented by the historical supplementation effect.Hr (Step S5).

[0104] Scaler 30 and history supplement unit 70 will input the current frame I C 'and supplementary historical frame I Hr Input is fed into the learned DNN 80 (step S6). The DNN 80 outputs an output image I as the inference result. O As the final output and inference history frame I H This is used for the next video processing step (step S7). The recursive processing of the RNN is repeated below.

[0105] [2-4. Effects]

[0106] Information processing device 1 includes a mask region acquisition unit 40, a history supplementation unit 70, and a DNN 80. The mask region acquisition unit 40 extracts the inference history frame I, which is the inference image of past frames. H The image element IE, whose motion vector MV is unknown, is identified for motion compensation. The mask region acquisition unit 40 acquires the region corresponding to the image element IE as the mask region MA. The history supplementation unit 70 acquires supplementary historical frames I. Hr This supplementary historical frame I Hr It is by using motion-compensated inference history frames I H The portion corresponding to the mask region MA in the input current frame I is based on the input current frame I. C This is obtained through supplementation. DNN 80 is based on the input current frame I. C and supplementary historical frames I Hr Output the current frame for inference. In the information processing method of this disclosure, the processing of the information processing device 1 is performed by a computer. The computer-readable non-transitory storage medium of this disclosure stores a program for causing the computer to perform the processing of the information processing device 1.

[0107] Based on this configuration, in the current input frame I C With supplementary historical frame I Hr There is almost no positional deviation of image elements between them. Therefore, defects such as ghosting and reduced brightness are suppressed, and video quality is improved.

[0108] Mask region acquisition unit 40 acquires the first mask region MA H and the second mask region MA C MA is the mask region. The first mask region MA H Indicator Reasoning History Frame I H The range of image elements in IE. Second mask region MA. C Indicates input current frame I C The range of image elements in IE.

[0109] DNN 80 will mix the input of the current frame I C The inference image output obtained from the motion-compensated historical frames is used as the inference result. Therefore, in the inference image, the current frame I is used as the input. C Image elements in IE are drawn in areas and infer history frames. H Image elements in the IE (Image Frame Interference) may exhibit ghosting and reduced brightness in the drawn areas. These issues are better resolved by using image data from the current frame to correct these areas.

[0110] The mask region acquisition unit 40 acquires the mask region MA based on post-processing information related to the generation of image element IE.

[0111] Based on this configuration, the image elements to be removed (IE) can be accurately identified.

[0112] The information processing device 1 includes a recovery unit 50 and a motion compensation unit 60. The recovery unit 50 performs reasoning on historical frames I... H The region corresponding to the first mask region MA H The pixel values ​​of a portion are replaced with the input current frame I. C To generate a restored historical frame I using pixel values He Motion compensation unit 60 pairs of historical frames I are restored He Perform motion compensation to generate corrected historical frames I He MC The historical supplement unit 70 corrects the historical frame I. He MC The corresponding region MA of the second mask C The pixel values ​​of a portion are replaced with the input current frame I. C The pixel values ​​are used to generate supplementary historical frames I Hr .

[0113] According to this configuration, after performing motion compensation by excluding image elements (IEs) whose motion vectors (MV) are unknown, the image elements (IEs) are then restored. Therefore, the occurrence of ghosting and other issues caused by motion compensation of image elements (IEs) is further suppressed.

[0114] The mask region acquisition unit 40 acquires the transmittance of multiple particles P added as effects as post-processing information. The mask region acquisition unit 40 acquires low transmittance regions in each particle P whose transmittance is lower than the set standard as mask regions MA.

[0115] This configuration allows the mask region MA to be as small as possible. Due to the inference history frame I... H It was not modified beyond what was necessary, so the inference results of DNN 80 are of high quality.

[0116] The recovery unit 50 replaces the pixel values ​​in a range of predetermined pixel sizes extending outward from the low transmittance region with the input current frame I. C The pixel value.

[0117] According to this configuration, by slightly widening the mask area, the occurrence of ghosting and other issues is further suppressed.

[0118] The setting criteria are configured to maximize the image quality assessment value of the current frame during inference.

[0119] This configuration allows for maximizing the quality of inference images.

[0120] Note that the effects described in this manual are merely examples and are not limited to any particular effect; other effects may be provided.

[0121] [3. A second example of DNN processing for dynamic images based on this disclosure]

[0122] [3-1. System Configuration Example]

[0123] Figure 10 This is a diagram illustrating an example configuration of another information processing apparatus 2 performing DNN processing for moving images according to the present disclosure.

[0124] This example and Figure 4 The information processing device 1 differs from the previous one in the following two aspects. First, based on the input current frame I... C With motion-compensated inference history frame I H (Simple correction of historical frames I) H MC The mask region MA is estimated by the difference between the two frames. Secondly, the historical frame I is simply corrected. H MC The portion corresponding to the mask region MA is used as input to the current frame I. C Supplement to generate supplementary history frames I Hr The following text will primarily describe the differences from information processing device 1.

[0125] The mask region is estimated by the mask region acquisition unit 45 included in the information processing device 2. The motion compensation unit 60 acquires inference images of past frames from the DNN 80 as inference history frames I. H The motion compensation unit 60 infers the historical frame I based on the motion vector MV' obtained from the scaler 30. H Perform motion compensation. Mask region acquisition unit 45 acquires the motion-compensated inference history frame I. H As a simple correction history frame I H MC Simple correction of historical frames IH MC It is through inference history frames I including image elements in IE. H Obtained by performing motion compensation. Image elements IE do not have motion vectors MV. Therefore, simple correction of historical frames I H MC It is an image for which motion compensation has been applied to parts other than the image elements (IE).

[0126] Simple correction of historical frames I H MC The position of the image element in IE is not from its position before motion compensation (inference history frame I). H The position of the image element in the image changes. The image element in the IE image undergoes simple correction of the historical frame. H MC With input current frame I C There are positional differences between them. The mask region acquisition unit 45 is based on a simple correction of historical frame I. H MC With input current frame I C The mask region MA is obtained by comparing the results between the two.

[0127] For example, the mask region acquisition unit 45 extracts simple correction history frame I. H MC With input current frame I C The difference between them. The mask region acquisition unit 45 generates an effect mask M in which the range with the difference is set as the mask region MA. I And apply the effect mask M I Send to History Supplement Unit 70.

[0128] Effect Mask M I Including the first mask region MA as the mask region MA H Second mask region MA C (See) Figure 11 For example, effect mask M I It is a binary image in which pixels in the mask region MA are set to black (pixel value = 0), and pixels in the area outside the mask region MA are set to white (pixel value = 1). The history supplementation unit 70 performs simple correction on the history frame I. H MC The region corresponding to the first mask region MA H The pixel values ​​of the portion and the corresponding second mask region MA C The pixel values ​​of a portion are replaced with the input current frame I. C The pixel values ​​are used to generate supplementary historical frames I. Hr .

[0129] [3-2. Supplementary processing for simple correction of historical frames]

[0130] Figure 11 This is used to illustrate the simple correction of historical frames I. H MC A diagram illustrating an example of supplementary processing.

[0131] When simple correction of historical frames I H MC With input current frame I C During comparison, portions of the image element IE added in post-processing are captured as differences. However, when differences are obtained in a simple sequential manner, differences other than image element IEs may also be detected. For example, in a simple correction of historical frame I... H MC When the resolution is reduced due to pixel addition during motion compensation, differences caused by the reduced resolution may be detected. Therefore, classification processing is performed to detect differences caused by post-processing.

[0132] For example, the mask region acquisition unit 45 pairs simple correction history frames I H MC With input current frame I C The differences between frames (inter-frame differences) are classified, and it is determined which part of the inter-frame difference is caused by post-processing. The mask region acquisition unit 45 obtains data from the simple correction history frame I. H MC and input current frame I C The differences caused by post-processing related to the generation of image element IE are extracted. The mask region acquisition unit 45 acquires the range of this difference in the image as the mask region MA.

[0133] Figure 11 An example of a calculation formula for performing supplementary processing is shown. In the calculation formula, " "This means having an effect mask M that can be obtained by subtracting the effect mask M from 1." I The image is obtained by taking the pixel value (0 or 1) as the pixel value. This image is obtained by using the effect mask M. I The inverted image is obtained by inverting the white (pixel value=1) and black (pixel value=0) pixels.

[0134] Here, " This means that its pixel value is the effect mask M. I The pixel values ​​of the inverted image are compared with the input current frame I. C An image of the product of pixel values. This image refers to the input of the current frame I. C The division corresponds to the effect mask M. I The image whose pixel values ​​outside the mask region MA are replaced with 0, i.e., the input current frame I CThe image corresponding to the portion of the mask region MA.

[0135] “ This means that its pixel value is the effect mask M. I Pixel values ​​and simple correction history frames I H MC An image of the product of pixel values. This image refers to the simple correction of historical frame I. H MC The corresponding effect mask M I The image where the pixel values ​​of a portion of the mask region MA are replaced with 0 is a simple correction of the historical frame I. H MC The image excluding the portion corresponding to the mask region MA.

[0136] also," "This means by inputting the current frame I..." C The image corresponding to the portion of the mask region MA and the simple corrected history frame I H MC The image is obtained by adding the images of all parts except the portion corresponding to the mask region MA. By adding the images, we obtain the image where the current input frame I is located. C The image element IE is included in the simple correction history frame I H MC Images in (Supplementary History Frame I) Hr ).

[0137] Figure 12 This is a diagram illustrating an example of how inter-frame differences are classified and processed.

[0138] Mask region acquisition unit 45 calculates simple correction history frame I based on multiple determination techniques. H MC The pixels and the input current frame I C The similarity between corresponding pixels. The mask region acquisition unit 45 acquires the mask region MA by applying the similarity calculated by each of the determination techniques to the similarity determination criteria.

[0139] The similarity determination criteria can be arbitrarily set by the system developers. These criteria can be a threshold indicating a range of similarity, or a mathematical model that outputs a definite value related to the similarity determination. The mathematical model can statistically process the similarity obtained through each determination technique and output a definite value. The mathematical model can be represented as a probability density function or as a neural network.

[0140] exist Figure 12 In the example, "1-diff" and "cosine similarity" are shown as similarities, where "diff" is obtained by obtaining a simple corrected historical frame I.H MC With input current frame I C The difference between pixel values ​​of each color is calculated by summing the differences between the corresponding pixel values ​​and dividing the sum by the number of colors. Figure 12 In the middle, (R) H G H B H Instructions for simple correction of historical frames I H MC The pixel values ​​of the red, green, and blue pixels. (R) C G C B C Indicates the input of the current frame I C The pixel values ​​of red (R), green (G), and blue (B) pixels. YUV can be used instead of RGB as a color representation method.

[0141] Figure 13 This is a diagram illustrating an example of a mathematical model used as a criterion for similarity determination.

[0142] exist Figure 13 In the example, the mathematical model is modeled as a two-dimensional probability density function (2D-PDF). Figure 13 In this context, "μ" represents the average of the two types of similarity ("1-diff", "cosine similarity"), and "∑" represents the covariance of the two types of similarity, where "n" is the number of data points and "T" is a constant. In the calculation of the probability density function, for various 3D content rendering images, the input current frame I is obtained. C and simple correction of historical frames I H MC The paired data. Here, "n" is the total number of paired data.

[0143] exist Figure 13 In this context, "x" is a two-dimensional column vector containing the values ​​of "1-diff" and "cosine similarity". Figure 13 In the mathematical model, based on the input current frame I C and simple correction of historical frames I H MC The pixel value is used to obtain the probability density p for each unit of a pixel.

[0144] The smaller the probability density p, the higher the probability of it being the effect image; conversely, the larger the probability density p, the lower the probability of it being the effect image. Figure 13 In the example, a threshold th is set for the probability density p. pdf The probability density p is equal to or less than the threshold th. pdf The region is defined as the effect region, and the probability density p is greater than the threshold th. pdfThe area is defined as the non-effect area. The effect area is the masked area MA (pixel value "mask" = 0), and the non-effect area is the non-masked area (pixel value "mask" = 1). System developers can appropriately set the threshold th based on the ease with which ghosting occurs. pdf .

[0145] Threshold th pdf It can also be determined based on the final image quality. Figure 14 This shows the threshold th used to define the range of the effect area based on the final image quality. pdf A diagram illustrating examples of the methods. Figure 14 In the example, the similarity determination criterion is set to make the inference of the current frame (output image I) from the DNN 80 output. O The condition for maximizing the image quality assessment value is defined as follows. As image quality assessment methods, PSNR, SSIM, etc., are used.

[0146] Image quality is evaluated by the Image Quality Evaluation Unit (EV). The EV obtains different threshold values ​​from the DNN 80. pdf Multiple inference images are obtained. The Image Quality Assessment Unit (EV) compares the acquired inference images with the ground truth and sets a threshold th when the inference image closest to the ground truth is obtained. pdf The similarity criterion is adopted. The threshold th is used. pdf The search can be performed using methods known as grid search or Bayesian optimization.

[0147] [3-3. Processing Flow]

[0148] Figure 15 This is a diagram illustrating an example of the processing flow used to perform inference processing.

[0149] Scaler 30 obtains the input current frame I from renderer 20. C And motion vector MV. Scaler 30 zooms in on the current input frame I as needed. C And the number of pixels of the motion vector MV, and obtain the input current frame I. C 'And motion vector MV' (step S11).

[0150] Motion compensation unit 60 obtains inference images of past frames from DNN 80 as inference history frames I. H The motion compensation unit 60 uses motion vector MV' to infer historical frames I. H Perform motion compensation and generate simple corrected historical frames I. H MC (Step S12). The mask region acquisition unit 45 obtains the mask region based on the input current frame I. C 'With simple correction of historical frames I HMC The comparison results are used to obtain the effect mask M. I (Step S13).

[0151] Historical Supplement Unit 70 Based on Effect Mask M I Use the input current frame I C 'Supplementing Simple Correction History Frames I' H MC This corresponds to the portion of the mask region MA. Therefore, the history supplementation unit 70 acquires a supplementary history frame I in which ghosting and reduced brightness issues are improved. Hr (Step S14).

[0152] Scaler 30 and history supplement unit 70 will input the current frame I C 'and supplementary historical frame I Hr Input is fed into the learned DNN 80 (step S15). The DNN 80 outputs an output image I as the inference result. O As the final output and inference history frame I H This is used for the next video processing step (step S16). The recursive processing of the RNN is repeated below.

[0153] [3-4. Effects]

[0154] Mask region acquisition unit 45 is based on inference history frames I including image elements IE H Simple correction history frame I obtained by performing motion compensation H MC With input current frame I C The mask region MA is obtained by comparing the results between them.

[0155] Based on this configuration, even if post-processing information cannot be obtained from renderer 20, the image element to be removed (IE) can be identified.

[0156] The history supplement unit 70 performs simple correction of the history frame I H MC The region corresponding to the first mask region MA H The pixel values ​​of the portion and the corresponding second mask region MA C The pixel values ​​of a portion are replaced with the input current frame I. C The pixel values ​​are used to generate supplementary historical frames I Hr .

[0157] With this configuration, problems related to both ghosting and reduced brightness can be solved simultaneously through a single process.

[0158] Mask region acquisition unit 45 obtains information from simple correction history frame I H MC and input current frame IC The differences caused by post-processing related to the generation of image element IE are extracted. The mask region acquisition unit 45 acquires the range of this difference in the image as the mask region MA.

[0159] Based on this configuration, the image elements to be removed can be easily identified in IE.

[0160] Mask region acquisition unit 45 calculates simple correction history frame I based on multiple determination techniques. H MC The pixels and the input current frame I C The similarity between corresponding pixels. The mask region acquisition unit 45 acquires the mask region MA by applying the similarity calculated by each of the determination techniques to the similarity determination criteria.

[0161] Based on this configuration, image elements to be removed (IE) can be identified with high accuracy.

[0162] The similarity determination criteria are set to maximize the image quality assessment value of the current frame for inference.

[0163] This configuration allows for maximizing the quality of inference images.

[0164] [4. Hardware Configuration Example]

[0165] Figure 16 This is a diagram illustrating an example of the hardware configuration of information processing devices 1 and 2.

[0166] Information processing in information processing devices 1 and 2 is implemented, for example, by computer 1000. Computer 1000 includes a central processing unit (CPU) 1100, random access memory (RAM) 1200, read-only memory (ROM) 1300, hard disk drive (HDD) 1400, communication interface 1500, and input / output interface 1600. These units of computer 1000 are connected via bus 1050.

[0167] The CPU 1100 operates based on the program (program data 1450) stored in the ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 loads the program stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes the processing corresponding to the various programs.

[0168] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) and programs that depend on the hardware of computer 1000, which are executed by CPU 1100 when computer 1000 is started.

[0169] HDD 1400 is a computer-readable recording medium that non-transitorily records programs executed by CPU 1100, data used by the programs, etc. Specifically, HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0170] Communication interface 1500 is an interface for connecting computer 1000 to an external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device or sends data generated by CPU 1100 to another device via communication interface 1500.

[0171] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as keyboards and mice via input / output interface 1600. Additionally, CPU 1100 sends data to output devices such as display devices, speakers, or printers via input / output interface 1600. Furthermore, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium (medium). Examples of media include optical recording media (e.g., digital multifunction discs (DVDs) or phase-change rewritable discs (PDs)), magneto-optical recording media (e.g., magneto-optical (MO) discs), magnetic tape media, magnetic recording media, semiconductor memory, etc.

[0172] For example, when computer 1000 is used as information processing apparatus 1 and 2 according to the embodiments, the CPU 1100 of computer 1000 executes an information processing program loaded on RAM 1200 to implement the functions of the aforementioned unit. Additionally, HDD 1400 stores information processing programs, various models, and various data according to this disclosure. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes the program data; however, as another example, these programs can be obtained from another device via external network 1550.

[0173] [Additional Explanation]

[0174] Note that this technology can also be configured as follows. (1)

[0176] An information processing apparatus, comprising:

[0177] A mask region acquisition unit is configured to identify image elements with unknown motion vectors for motion compensation in an inference history frame of an inference image that is a past frame, and acquire the region corresponding to the image element as a mask region.

[0178] A history supplementation unit, configured to acquire supplementary history frames, which are obtained by supplementing the portion of the motion-compensated inference history frame corresponding to the mask region based on the input current frame; and

[0179] The DNN is configured to output the inference current frame based on the input current frame and the supplementary historical frames. (2)

[0181] According to the information processing apparatus described in (1), wherein,

[0182] The mask region acquisition unit acquires a first mask region and a second mask region as the mask region. The first mask region indicates the range of the image elements in the inference history frame, and the second mask region indicates the range of the image elements in the current input frame. (3)

[0184] According to the information processing device described in (2), wherein,

[0185] The mask region acquisition unit acquires the mask region based on post-processing information related to the generation of the image elements. (4)

[0187] The information processing apparatus according to (3) further includes:

[0188] A recovery unit, configured to generate a recovered history frame by replacing pixel values ​​of the portion of the inference history frame corresponding to the first mask region with pixel values ​​of the input current frame; and

[0189] A motion compensation unit, configured to perform motion compensation on the recovered historical frame to generate a corrected historical frame, wherein...

[0190] The history supplementation unit generates the supplementary history frame by replacing the pixel values ​​of the portion of the corrected history frame corresponding to the second mask region with the pixel values ​​of the input current frame. (5)

[0192] According to the information processing device described in (4), wherein,

[0193] The mask region acquisition unit acquires the transmittance of multiple particles added as effects as information for the post-processing, and acquires the low transmittance region in each particle that is lower than the set standard as the mask region. (6)

[0195] According to the information processing apparatus described in (5), wherein,

[0196] The recovery unit replaces the pixel values ​​of the input current frame with the pixel values ​​of a range of pixels extending outward from the low transmittance region by a predetermined pixel size. (7)

[0198] According to the information processing apparatus described in (5) or (6), wherein,

[0199] The setting criteria are set to maximize the image quality assessment value of the current frame in the inference. (8)

[0201] According to the information processing device described in (2), wherein,

[0202] The mask region acquisition unit acquires the mask region based on the comparison result between the simple correction history frame and the input current frame. The simple correction history frame is obtained by performing motion compensation on the inference history frame including the image elements. (9)

[0204] According to the information processing apparatus described in (8), wherein,

[0205] The history supplementation unit replaces the pixel values ​​of the portion corresponding to the first mask region and the portion corresponding to the second mask region of the simple corrected history frame with the pixel values ​​of the input current frame to generate the supplemented history frame. (10)

[0207] According to the information processing apparatus described in (8) or (9), wherein,

[0208] The mask region acquisition unit extracts the differences caused by post-processing related to the generation of the image elements from the simple correction history frame and the input current frame, and acquires the range of the difference in the image as the mask region. (11)

[0210] According to the information processing apparatus described in (10), wherein,

[0211] The mask region acquisition unit calculates the similarity between the pixels of the simple correction history frame and the corresponding pixels of the input current frame based on multiple determination techniques, and acquires the mask region by applying the similarity calculated by each of the determination techniques to a similarity determination criterion. (12)

[0213] According to the information processing apparatus described in (11), wherein,

[0214] The similarity determination criterion is set to maximize the image quality assessment value of the current frame in the inference. (13)

[0216] An information processing method executed by a computer, the method comprising:

[0217] In the inference history frame of the inference image which is a past frame, identify image elements with unknown motion vectors for motion compensation, and obtain the region corresponding to the image element as a mask region;

[0218] Obtain supplementary historical frames, which are obtained by supplementing the portion of the motion-compensated inference historical frames corresponding to the mask region based on the input current frame; and

[0219] The current frame is inferred based on the input current frame and the supplementary historical frames. (14)

[0221] A computer-readable non-transitory storage medium storing a program for causing a computer to perform the following steps:

[0222] In the inference history frame of the inference image which is a past frame, identify image elements with unknown motion vectors for motion compensation, and obtain the region corresponding to the image element as a mask region;

[0223] Obtain supplementary historical frames, which are obtained by supplementing the portion of the motion-compensated inference historical frames corresponding to the mask region based on the input current frame; and

[0224] The current frame is inferred based on the input current frame and the supplementary historical frames.

[0225] List of reference numerals

[0226] 1, 2 Information processing devices

[0227] 40, 45 mask area acquisition units

[0228] 50 recovery units

[0229] 60 motion compensation units

[0230] 70 Historical Supplementary Units

[0231] 80DNN

[0232] I C I C 'Enter the current frame'

[0233] IE Image Elements

[0234] I H Reasoning history frames

[0235] IHe Restore historical frames

[0236] I He MC Correcting historical frames

[0237] I Hr Supplementing historical frames

[0238] I H MC Simple correction of historical frames

[0239] M, M C M H M I Effect Mask

[0240] MA mask area

[0241] MA C Second mask area

[0242] MA H First mask area

[0243] MV, MV' motion vector

[0244] P particles

Claims

1. An information processing apparatus, comprising: A mask region acquisition unit is configured to identify image elements with unknown motion vectors for motion compensation in an inference history frame of an inference image that is a past frame, and acquire the region corresponding to the image element as a mask region. A history supplementation unit is configured to acquire supplementary history frames, which are obtained by supplementing the portion of the motion-compensated inference history frame corresponding to the mask region based on the input current frame. as well as The DNN is configured to output the inference current frame based on the input current frame and the supplementary historical frames.

2. The information processing apparatus according to claim 1, wherein, The mask region acquisition unit acquires a first mask region and a second mask region as the mask region. The first mask region indicates the range of the image elements in the inference history frame, and the second mask region indicates the range of the image elements in the current input frame.

3. The information processing apparatus according to claim 2, wherein, The mask region acquisition unit acquires the mask region based on post-processing information related to the generation of the image elements.

4. The information processing apparatus according to claim 3, further comprising: A recovery unit is configured to generate a recovered history frame by replacing the pixel values ​​of the portion of the inference history frame corresponding to the first mask region with the pixel values ​​of the input current frame. as well as A motion compensation unit, configured to perform motion compensation on the recovered historical frame to generate a corrected historical frame, wherein... The history supplementation unit generates the supplementary history frame by replacing the pixel values ​​of the portion of the corrected history frame corresponding to the second mask region with the pixel values ​​of the input current frame.

5. The information processing apparatus according to claim 4, wherein, The mask region acquisition unit acquires the transmittance of multiple particles added as effects as information for the post-processing, and acquires the low transmittance region in each particle that is lower than the set standard as the mask region.

6. The information processing apparatus according to claim 5, wherein, The recovery unit replaces the pixel values ​​of the input current frame with the pixel values ​​of a range of pixels extending outward from the low transmittance region by a predetermined pixel size.

7. The information processing apparatus according to claim 5, wherein, The setting criteria are set to maximize the image quality assessment value of the current frame in the inference.

8. The information processing apparatus according to claim 2, wherein, The mask region acquisition unit acquires the mask region based on the comparison result between the simple correction history frame and the input current frame. The simple correction history frame is obtained by performing motion compensation on the inference history frame including the image elements.

9. The information processing apparatus according to claim 8, wherein, The history supplementation unit generates the supplementary history frame by replacing the pixel values ​​of the portion corresponding to the first mask region and the portion corresponding to the second mask region of the simple corrected history frame with the pixel values ​​of the input current frame.

10. The information processing apparatus according to claim 8, wherein, The mask region acquisition unit extracts the differences caused by post-processing related to the generation of the image elements from the simple correction history frame and the input current frame, and acquires the range of the difference in the image as the mask region.

11. The information processing apparatus according to claim 10, wherein, The mask region acquisition unit calculates the similarity between the pixels of the simple correction history frame and the corresponding pixels of the input current frame based on multiple determination techniques, and acquires the mask region by applying the similarity calculated by each of the determination techniques to a similarity determination criterion.

12. The information processing apparatus according to claim 11, wherein, The similarity determination criterion is set to maximize the image quality assessment value of the current frame in the inference.

13. An information processing method executed by a computer, the method comprising: In the inference history frame of the inference image which is a past frame, identify image elements with unknown motion vectors for motion compensation, and obtain the region corresponding to the image element as a mask region; Obtain supplementary historical frames, which are obtained by supplementing the portion of the motion-compensated inference historical frames corresponding to the mask region based on the input current frame; as well as The current frame is inferred based on the input current frame and the supplementary historical frames.

14. A computer-readable non-transitory storage medium storing a program for causing a computer to perform the following steps: In the inference history frame of the inference image which is a past frame, identify image elements with unknown motion vectors for motion compensation, and obtain the region corresponding to the image element as a mask region; Obtain supplementary historical frames, which are obtained by supplementing the portion of the motion-compensated inference historical frames corresponding to the mask region based on the input current frame; as well as The current frame is inferred based on the input current frame and the supplementary historical frames.

Images