Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device has a DNN, a history image acquisition unit, and a bright spot processing unit. The DNN uses a past frame as an input to perform inference. The history image acquisition unit acquires a history image corresponding to the inference result. The bright spot processing unit removes bright spots from the current input frame on the basis of the current input frame having continuity with the past frame, the history image, a motion vector between frames, and an occlusion map.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Flickering bright spots are often a problem in video production. For example, in CG rendering, a technique called TAA (Temporal Anti-Aliasing) is often used for anti-aliasing. However, when rendering moving subjects or rendering using TAA, random bright spots may appear as flickering, mainly in subjects with high specular components and high reflectivity.

[0003] Specular bright spots have very high brightness values, so if you try to suppress them using a simple image filter, you need a convolution that results in strong summation. Naturally, this results in a strong blurring of the image. Furthermore, if an image containing bright spots is input into a DNN for super-resolution, for example, the output will have strong flickering, resulting in a visually unpleasant image.

[0004] Even if a large amount of bright spot data is trained, this flickering cannot be sufficiently removed. Furthermore, it is necessary to distinguish bright spots from particles (effects) such as sparks, but this process is difficult. If it is not possible to distinguish between bright spots and particles, attempting to remove bright spots will also remove particles.

[0005] Yusuke Tokuyoshi and Anton S. Kaplanyan, “Improved Geometric Specular Antialiasing”, I3D '19, May 21-23, 2019, Montreal, QC, Canada <Internet> http: / / www. jp. square-enix. com / tech / library / pdf / ImprovedGeometricSpecularAA. pdf (searched on October 19, 2020)

[0006] Typically, when this type of flicker occurs, the solution is to reduce the reflectivity of the object material or adjust the roughness to make it smoother. However, these approaches eliminate the detail in the material's reflection and must be implemented as internal processing in the renderer (before converting to 2D imaging). Ideally, it would be possible to suppress bright spots even after converting to 2D imaging without reducing the detail.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that can suppress image defects caused by flickering bright spots.

[0008] According to the present disclosure, there is provided an information processing device including a DNN that performs inference using past frames as input, a history image acquisition unit that acquires a history image corresponding to the inference result, and a bright spot processing unit that removes bright spots from the input current frame based on an input current frame that has continuity with the past frames, the history image, a motion vector between frames, and an occlusion map.The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium that stores a program that causes a computer to realize the information processing of the information processing device.

[0009] 1 is a diagram illustrating a conventional DNN process for moving images; FIG. 2 is a diagram illustrating an example of rendering without using TAA; FIG. 3 is a diagram illustrating an example of rendering using TAA; FIG. 4 is a diagram illustrating an example of light distribution characteristics of diffuse light and reflected light; FIG. 5 is a diagram illustrating an example in which a microfacet normal distribution function (NDF) is set on the surface of a CG object; FIG. 6 is a diagram illustrating an example configuration of an information processing device that performs DNN processing for moving images according to the present disclosure; FIG. 7 is a diagram illustrating a processing flow; FIG. 8 is a diagram illustrating intermediate feature amounts; FIG. 9 is a diagram illustrating an example of bright spot detection using a history image; FIG. 10 is a diagram illustrating an example of a method for detecting an occlusion region; FIG. 11 is a diagram illustrating an example of a method for detecting an occlusion region; FIG. 12 is a diagram illustrating an example of a processing flow for performing inference processing; FIG. 13 is a diagram illustrating an example of a processing flow for performing inference processing; FIG. 14 is a diagram illustrating an example of a hardware configuration of an information processing device;

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] The description will be given in the following order: [1. Background] [1-1. Conventional DNN processing for moving images] [1-2. Occurrence of bright spot flickering] [2. Example of DNN processing for moving images according to the present disclosure] [2-1. Example of system configuration] [2-2. Detection of bright spots using history images] [2-3. Detection of occlusion areas] [2-4. Example of bright spot detection processing] [2-5. Example of bright spot removal processing] [2-6. Processing flow] [3. Example of hardware configuration] [4. Effects]

[0012] [1. Background] [1-1. Conventional DNN Processing for Video] FIG. 1 is a diagram showing conventional DNN processing for video.

[0013] DNN processing for video generally uses a recurrent neural network (RNN) structure with high temporal stability. The RNN structure can correlate the current frame with history frames, making it easier to maintain image consistency in the output frames. While DNNs have very high inference performance (e.g., sharpening effects in the case of super-resolution), they are affected by even small changes in input data, causing output results to change. Therefore, in video processing, flickering is more likely to occur than with simple filter processing. To increase temporal stability and reduce flickering, an RNN structure is generally implemented that simultaneously inputs the current frame and history frames and enables learning that includes temporal correlation (similarity between the current frame and history frames).

[0014] Note that the current frame refers to the image of the frame (current frame) that is the subject of estimation by the DNN. The history frame refers to an image (inference image) obtained by inputting an image of a past frame into the DNN, or intermediate features extracted from an image of a past frame. The past frame refers to a frame that is one or more frames before the current frame. In the present disclosure, for example, the frame that is one frame before the current frame (the most recent frame) is used as the past frame. Hereinafter, the current frame may be referred to as the "input current frame," and the history frame that is the inference result of the past frame may be referred to as the "inference history frame." Furthermore, the inference image inferred by the DNN and output as an RGB image may be referred to as the "output image."

[0015] Intermediate features refer to information on features output from the intermediate layer of a DNN when an image of a past frame is input to the DNN. It is known that using intermediate features obtained during inference as an inference history frame can produce more accurate estimation results than using an inference image (RGB image) of a past frame as an inference history frame. In the present disclosure, intermediate features obtained during inference are used as the inference history frame.

[0016] The inferred history frame, together with the input current frame, is used as input to the DNN. To ensure consistency with the input current frame, the inferred history frame undergoes motion compensation based on a motion vector. A motion vector is a vector that indicates the movement of an object between frames (the amount and direction of pixel movement). Motion compensation is a process that predicts post-motion data from pre-motion data. For example, when motion compensation is applied to an image of a past frame, the image predicted in the current frame is obtained as the motion-compensated inferred history frame. The motion-compensated inferred history frame is input to the DNN along with the input current frame.

[0017] [1-2. Occurrence of Flickering of Bright Spots] Hereinafter, examples of occurrence of flickering of bright spots will be described with reference to FIGS. 2 to 5. FIG.

[0018] In the production of CG images, an anti-aliasing technique called TAA is sometimes used. TAA is a technique that suppresses aliasing (the stepped edges (jaggies) that occur along the edges of pixels PX when drawing diagonal lines or curves) by shifting the sampling position SP of pixels in the time direction and adding up the sampling information in the time direction. Shifting the sampling position SP at the sub-pixel level is called adding jitter.

[0019] 2 is a diagram showing an example of rendering without TAA. The renderer renders 3D data of a CG object OB from the viewpoint of a CG camera CM. The renderer obtains the pixel value of each pixel PX as color data at a fixed position (e.g., the center of the pixel PX) of the pixel PX. In this case, strong aliasing remains in the rendered image.

[0020] FIG. 3 is a diagram showing an example of rendering using TAA. The pixel value of each pixel PX is calculated as a statistical value of color data acquired at multiple times while shifting the sampling position SP. In the example of FIG. 3, the pixel value at time t is calculated as a weighted average of four pieces of color data from times t, (t-1), (t-3), and (t-3). By shifting the sampling position SP and performing synthesis, edges are blurred, resulting in a smooth image.

[0021] Shifting the sampling position SP helps to improve edge smoothness. However, in areas where bright spots are drawn, pixel values ​​fluctuate significantly depending on whether the sampling position SP is set on the bright spot or off the bright spot. This fluctuation in pixel value is perceived as flickering of the bright spot. In particular, the specular component exhibits a large change in reflectance with respect to the incident reflection angle and a large luminance, making the flickering of the bright spot more noticeable. Figure 4 shows an example of the light distribution characteristics of diffused light and reflected light. In the specular reflection direction, the light source enters the eye (CG camera CM) directly, resulting in a strong bright spot.

[0022] A fine normal distribution can also be defined on the surface of a CG object OB to express complex light reflections. FIG. 5 shows an example in which a microfacet normal distribution function (NDF) is set on the surface of a CG object OB. The normal distribution expresses fine irregularities. When the sampling position SP fluctuates due to TAA, a frame suddenly exhibiting strong reflection appears within one pixel PX. This occurs because the sampling position SP overlaps with the specular reflection position. Fluctuations in the sampling position SP near the specular reflection position cause fluctuations in reflection intensity (flickering bright spots).

[0023] Non-Patent Document 1 proposes a method to avoid sudden changes in reflection intensity by smoothly filtering the normal distribution function. However, this method is an approach to improving the internal processing of the renderer and cannot be implemented without modifying the rendering engine. Furthermore, the lighting resolution decreases even in areas where there is no flickering of bright spots. If bright spots can be removed after rendering, anti-aliasing would be possible without modifying the rendering engine. From this perspective, this disclosure proposes a method to improve image quality without modifying the internal structure of the renderer.

[0024] 2. Example of DNN Processing for Video of the Present Disclosure 2-1. Example of System Configuration Fig. 6 is a diagram showing an example of the configuration of an information processing device 1 that performs DNN processing for video of the present disclosure. Fig. 7 is a diagram showing the flow of processing.

[0025] The DNN processing for moving images according to the present disclosure is performed by an information processing device 1. For example, the information processing device 1 includes an input unit 10, a renderer 20, a scaler 30, a consecutive frame determination unit 40, an occlusion detection unit 50, a bright spot processing unit 150, a history image acquisition unit 160, a DNN 100, and an output unit 110. The bright spot processing unit 150 includes a bright spot detection processing unit 60 and a bright spot removal processing unit 70. The history image acquisition unit 160 includes a history image estimation unit 80 and a motion compensation unit 90.

[0026] In this disclosure, bright points are detected based on the difference between frames, and the input current frame I ＣThe image from which bright spots have been removed is input to the DNN 100. The past image to be used for calculating the difference with respect to the input current frame is the inferred image of the past frame (the output image I of the DNN 100). Ｏ ) or an image equivalent thereto. In the present disclosure, an inferred image of a past frame or an image equivalent thereto is referred to as a history image I Ｈ，ＲＧＢ ＭＣ It is written as follows.

[0027] For example, the DNN 100 performs inference using past frames as input. The history image acquisition unit 160 acquires the inference results of the past frames output from the DNN 100 and stores the history image I corresponding to the inference results. Ｈ，ＲＧＢ ＭＣ The bright point processing unit 150 obtains the input current frame I that has continuity with the previous frame. Ｃ , history image I Ｈ，ＲＧＢ ＭＣ , the inter-frame motion vector MV, and the occlusion map OC, the input current frame I Ｃ This removes bright spots from the image, resulting in a high-quality image that is not affected by bright spots.

[0028] The input unit 10 receives an operation signal S Ｓ is input to the renderer 20. For example, the operation signal S Ｓ contains rendering settings and instructions for performing rendering.

[0029] The renderer 20 receives the operation signal S Ｓ The rendering process includes setting rendering conditions and executing rendering. The renderer 20 executes the rendering process based on the input current frame I Ｃ The input current frame I is used to obtain a rendering image and a depth image DP. Ｃ refers to an image of a certain frame among a series of frames such as a video. A depth image DP is a one-dimensional image that represents depth. The block diagram in Figure 6 explains the operation for one frame of a video.

[0030] The renderer 20 obtains the motion vector MV of the subject based on the rendering settings. The motion vector MV is a two-dimensional vector that defines the movement of corresponding pixels between frames by the amount of movement of the pixels (number of pixels). The motion vector MV can be obtained by a general estimation process using a group of consecutive frames. When a CG renderer is used, the motion vector MV can be generated and obtained during rendering. The renderer 20 calculates the motion vector MV from the input current frame I obtained by the rendering process. Ｃ , motion vectors MV and depth image DP to a scaler 30 .

[0031] The renderer 20 generates continuity determination information S based on the rendering settings. Ｒ and sends the continuity determination information S Ｒ is the input current frame I output from the renderer 20 Ｃ The information indicates whether or not there is continuity between the input current frame I and the previous frame. Ｃ This means that the current frame and the previous frame are consecutive video frames relating to a common video scene.

[0032] For example, the input current frame I Ｃ If the initial frame of rendering is a frame, or if a scene change occurs, the renderer 20 generates continuity determination information S Ｒ otherwise, i.e., the input current frame I Ｃ If the frames are consecutive frames related to a common video scene with respect to the previous frame, the renderer 20 outputs the continuity determination information S Ｒ and outputs 1.

[0033] The scaler 30 scales the input current frame I as needed. Ｃ、 The number of pixels of the motion vector MV and the depth image DP is enlarged. Hereinafter, when it is necessary to distinguish the data after the number of pixels has been enlarged by the scaler 30 from the data before enlargement, a "'" is added after the symbol of the data. The scaler 30 performs the following operations on the input current frame I. ＣThe scaler 30 sends the motion vector MV' to the consecutive frame determination unit 40, the occlusion detection unit 50, and the motion compensation unit 90. The scaler 30 sends the depth image DP' to the consecutive frame determination unit 40 and the occlusion detection unit 50.

[0034] The architecture of the DNN 100 accepts a fixed number of pixels, while the input current frame I Ｃ In cases where the number of pixels varies depending on the usage, the scaler 30 may perform pixel number expansion processing at a desired magnification to obtain a pixel number that matches the specifications of the DNN 100. The pixel number expansion method may be selected arbitrarily, for example, bilinear or bicubic. The motion vector MV is preferably expanded using the nearest neighbor method to avoid the generation of intermediate values ​​due to pixel interpolation, but is not limited to this.

[0035] The consecutive frame determination unit 40 determines the consecutiveness determination information S Ｒ Based on the input current frame I Ｃ The continuous frame determining unit 40 determines the continuity between the input current frame I' and the previous frame. Ｃ If ' is a consecutive frame, then the input current frame I Ｃ The continuous frame determination unit 40 sends the motion vector MV' to the bright spot detection processing unit 60 and the bright spot removal processing unit 70. Ｃ If ' is not a consecutive frame, then the input current frame I Ｃ ' is sent to the DNN 100.

[0036] Regarding the continuity between frames, continuity determination information S Ｒ For example, the consecutive frame determining unit 40 may determine the consecutive frame number based on information other than the input current frame I. Ｃ ' 、 Consecutive frames can also be determined by image processing using the motion vectors MV' and depth images DP'.

[0037] The bright spot removal processing unit 70 receives the input current frame I Ｃ' and correct the current frame I ＣＲ For example, the bright spot removal processing unit converts the rendering image generated using the TAA into the input current frame I Ｃ The bright spot removal processing unit removes the specular bright spots generated by the TAA from the input current frame I Ｃ The bright spots can be removed by, for example, replacing the image of the bright spot with a previous frame or an image equivalent thereto.

[0038] The motion compensation unit 90 calculates the inference history frame I obtained as a result of inferring the past frame. Ｈ , and perform motion compensation to obtain the corrected history frame I Ｈ ＭＣ The DNN 100 generates the corrected current frame I ＣＲ and correction history frame I Ｈ ＭＣ Based on the input current frame I Ｃ ' is inferred.

[0039] The DNN 100 receives the input current frame I Ｃ If ' is a consecutive frame, the corrected current frame I ＣＲ and correction history frame I Ｈ ＭＣ is used as input and the output image I Ｏ The DNN 100 outputs the intermediate features as the input current frame I Ｃ If ' is not a consecutive frame, then the input current frame I Ｃ ' and the correction history frame I Ｈ ＭＣ is used as input and the output image I Ｏ and the intermediate features are output.

[0040] The DNN 100 outputs the output image I Ｏ to the output unit 110. The DNN 100 outputs the intermediate feature to the inference history frame I ＨThe DNN 100 outputs the result to the motion compensation unit 90 as a coefficient value. Pre-trained values ​​are used as the coefficient values ​​of the DNN 100. Assumed learning tasks for the DNN 100 include, for example, super-resolution, noise reduction, and style transfer, but the present disclosure is not limited thereto. The DNN 100 is applicable to general video processing in general using an RNN structure.

[0041] The motion compensation unit 90 receives an inference history frame I, which is an inference result of a past frame, from the DNN 100. Ｈ In the example of FIG. 6, the inference history frame I Ｈ are intermediate features obtained by the DNN 100 during inference of past frames. In the present disclosure, the image and intermediate features obtained by inference may be simply referred to as inference results. The motion compensation unit 90 calculates the inference history frame I based on the motion vector MV′. Ｈ , and perform motion compensation to obtain the corrected history frame I Ｈ ＭＣ The motion compensation unit 90 performs a common process known as motion compensation on the inferred history frame I. Ｈ can be applied to.

[0042] The motion vector MV' has a vector related to the spatial movement of the subject between frames. Motion compensation is performed as a process of predicting data after motion from data before motion. The motion compensation unit 90 uses intermediate feature values ​​(inference history frame I) of past frames output from the DNN 100 as a vector. Ｈ ) by applying motion compensation. Ｃ The motion compensation unit 90 calculates the feature quantity that is close to the intermediate feature quantity of the motion-compensated inference history frame I Ｈ (Correction History Frame I Ｈ ＭＣ ) is sent to the history image estimation unit 80.

[0043] The history image estimation unit 80 estimates the corrected history frame I Ｈ ＭＣ From the motion-compensated inference image of the past frame (output image I Ｏ ) is the image corresponding to the history image I Ｈ，ＲＧＢＭＣ The history image I is obtained as Ｈ，ＲＧＢ ＭＣ The acquisition of the corrected history frame I, which is an intermediate feature of the RGB image, Ｈ ＭＣ The restoration process is performed as a process of restoring the inferred image of the past frame (output image I Ｏ ) is applied motion compensation to the image obtained by applying motion compensation to the image. Ｈ，ＲＧＢ ＭＣ is obtained as

[0044] The bright spot detection processing unit 60 detects an input current frame I that has continuity with the previous frame. Ｃ ', the bright spot detection processing unit 60 selectively detects bright spots. The bright spot detection processing unit 60 determines the presence or absence of a bright spot for each pixel PX and sends the determination result to the bright spot removal processing unit 70 as a bright spot map. The bright spot map has a determination value SH for each pixel PX regarding the presence or absence of a bright spot. For example, the bright spot detection processing unit 60 assigns a determination value SH of "1" to pixels PX that display bright spots. The bright spot detection processing unit 60 assigns a determination value SH of "0" to pixels PX that do not display bright spots.

[0045] The bright spot detection processing unit 60 receives the input current frame I Ｃ ' and history image I Ｈ，ＲＧＢ ＭＣ The bright spot detection processing unit 60 obtains the difference between the input current frame I and the bright spot detection processing unit 60 based on the inter-frame difference. Ｃ The bright spot removal processing unit 70 removes the bright spots detected by the bright spot detection processing unit 60. In this configuration, the detection and removal of bright spots are performed outside the renderer 20. Because the bright spot removal processing does not need to be performed as an internal process of the renderer 20, it is possible to improve image quality without changing the internal structure of the renderer 20.

[0046] As mentioned above, the inference history frame I Ｈ In this case, a correction called motion compensation is performed, which takes into account the motion between frames. Motion compensation is a process of predicting data for the current frame from past frames based on the motion between frames.

[0047] However, if the background portion is exposed due to the movement of the foreground, the data of the background portion is stored in the inference history frame I. Ｈ The background part cannot be predicted from the input current frame I Ｃ Since the pixel value to be referenced based on the motion vector MV' is unknown, the inference history frame I Ｈ (Correction History Frame I Ｈ ＭＣ ) produces artifacts (ghosts) in the data that indicate traces of the foreground.

[0048] In the DNN 100, the input current frame I Ｃ ' and the correction history frame I Ｈ ＭＣ The ghost-containing correction history frame I Ｈ ＭＣ If the motion vector is input to the DNN 100, an image is generated in which the foreground before the movement is reflected like a shadow, which causes a decrease in image quality. Therefore, in the present disclosure, data areas where ghosts may occur are detected in advance by the occlusion detection unit 50. By distinguishing image disturbances caused by ghosts from bright spots, unnecessary processing of bright spots can be avoided, and image quality is improved.

[0049] For example, the occlusion detection unit 50 detects a corrected history frame I in which ghosts may occur based on the depth image DP′ and the motion vector MV′. Ｈ ＭＣ The occlusion detection unit 50 detects the data area within the occlusion map OC as an occlusion area. The occlusion detection unit 50 generates a map indicating the occlusion area as an occlusion map OC and sends it to the bright spot detection processing unit 60. The bright spot detection processing unit 60 excludes the occlusion area from the bright spot detection targets. Any method may be used to detect the occlusion area. For example, the occlusion area may be identified from the depth information of the current frame and the past frame.

[0050] The occlusion detection unit 50 generates an occlusion map OC in which occlusion regions are labeled, and sends the occlusion map OC to the bright spot detection processing unit 60. For example, the occlusion map OC labels occlusion regions and regions other than occlusion regions (non-occlusion regions) with occlusion values. For example, an occlusion value of "0" is set for each pixel included in an occlusion region, and an occlusion value of "1" is set for each pixel included in a non-occlusion region.

[0051] [2-2. Detection of Bright Spots Using History Images] Fig. 8 is a diagram for explaining intermediate feature amounts. Fig. 9 is a diagram for explaining intermediate feature amounts using history images I Ｈ，ＲＧＢ ＭＣ 10A and 10B are diagrams illustrating an example of detecting a bright spot using the method of FIG.

[0052] The intermediate features refer to the internal features of the DNN 100. Ｃ is input to the DNN 100, and the feature z ｎ，ｌ Here, "n" represents the number of dimensions, and "l" represents the layer number. The DNN 100 is expressed with four layers.

[0053] The final output of DNN100 is z １，３ ~z ６４，３ 8, the number of dimensions is 64, but the number of dimensions is determined by the configuration of the DNN 100 and does not necessarily have to be 64.

[0054] In the RNN structure, the inference accuracy is improved by using such multidimensional features rather than recursively using RGB images as input. Ｈ The intermediate features recursively used as the multidimensional features (z １，ｌ ~z ｎ，ｌ In practice, it is desirable to use the feature just before the final layer (the output of the third layer in the example of FIG. 8 ), which is closer to the output, i.e., which is a feature that better represents the training data.

[0055] The history image estimation unit 80 uses a conversion model CV that converts intermediate feature quantities obtained during the inference of the DNN 100 into an RGB image to generate an inference history frame I. Ｈ From History Image I Ｈ，ＲＧＢ ＭＣ For example, the history image estimation unit 80 generates an inferred history frame I Ｈ The RGB image obtained by applying the conversion model CV to the previous frame is the inferred image (the output image I of the DNN 100). Ｏ The history image estimation unit 80 optimizes the transformation model CV so that the corrected history frame I Ｈ ＭＣ The image obtained by applying the optimized transformation model CV to the history image I Ｈ，ＲＧＢ ＭＣ It is calculated as follows.

[0056] The conversion model CV can be optimized using a general regression model. As shown in Figure 9, the conversion model CV represents a conversion matrix that converts the RNN feature into an RGB image. Figure 9 shows an example of multiple regression analysis. For example, if the red pixel value Cur Ｒ Is, Cur Ｒ = c Ｒ，０ RNN ０ +...+c Ｒ，７ RNN ７ +b Ｂ The coefficient c Ｒ，０ , ...c Ｒ，７ and b Ｂ is obtained by multiple regression analysis. Ｒ ", "Cur Ｒ " and "Cur Ｒ " is the inferred image of the past frame (output image I Ｏ ) and "c" and "b" denote coefficients of the conversion model CV.

[0057] In the example of Fig. 9, the feature has a total of eight channels (eight dimensions) from 0th to 7th, and the inferred image has a total of three channels, R, G, and B. Note that the number of dimensions of the feature and the channels of the inferred image are not limited to these. The number of dimensions of the feature may be nine or more, and the inferred image may be a YUV image.

[0058] 2-3. Detection of Occlusion Regions FIGS. 10 and 11 are diagrams showing an example of a method for detecting occlusion regions.

[0059] The occlusion detection unit 50 detects the movement of a moving object in the foreground and the movement of the background hidden behind the moving object (an image area referenced by the foreground using the motion vector MV) based on the motion vector MV. The occlusion detection unit 50 detects an occlusion area based on the movement of the foreground and background.

[0060] For example, the occlusion detection unit 50 calculates the magnitude of positional shift and the similarity of the movement direction between the foreground and background based on the motion vector MV. If the magnitude of the positional shift is not at the noise level (condition A) and the similarity of the movement direction does not satisfy the similarity condition (condition B), the occlusion detection unit 50 determines that the background area that overlapped with the foreground is an occlusion area. In the present disclosure, "the similarity between the movement direction of the foreground and the movement direction of the background does not satisfy the similarity condition" may be simply referred to as "the movement direction of the foreground and the movement direction of the background are different."

[0061] The similarity can be calculated using vector distance, cosine similarity, or the like. The noise level means that the magnitude of the positional shift is large enough to be considered noise. In other words, condition A may be considered to be a condition in which the magnitude of the positional shift between the foreground and background is greater than a threshold representing noise. The similarity condition means a condition for determining similarity. The noise level and similarity condition are hyperparameters that affect the inference accuracy of the DNN 100. These hyperparameters can be arbitrarily set by the system developer using thresholds, etc., while checking the ghosts that actually occur due to motion compensation.

[0062] That is, based on the motion vector MV, the occlusion detection unit 50 determines whether or not the following predetermined conditions are satisfied: the magnitude of the positional deviation between the foreground and the background is greater than a reference value indicating the noise level (condition A), and the direction of movement of the foreground and the direction of movement of the background are different (condition B). Based on the determination that the predetermined conditions are satisfied, the occlusion detection unit 50 determines that the area of ​​the background that overlapped with the foreground is an occlusion area.

[0063] The example in Figure 10 shows a case where the foreground is moving. If the reference of the foreground is smaller than the amount of movement of the foreground, it can be assumed that occlusion has occurred. In Figure 10, the background is stationary and the foreground is moving to the left. The motion vector MV of the foreground pixel (x, y) is (v ｘ , v ｙ ) When the foreground is moving, the occlusion area is an area that satisfies both the motion determination regarding the above-mentioned condition A (whether the moving object in the foreground has moved from its original location) and the similarity determination regarding the condition B (whether the original location has moved in the same direction as the moving object). Condition B suggests that occlusion will not occur when the background moves at a similar speed to the foreground.

[0064] For example, for a pixel (x, y), the occlusion value OC(x+v ｘ , y+v ｙ ) can be determined based on the formula shown in FIG. 10 as being "0" (included in the occlusion region) or "1" (not included in the occlusion region). "Similarity" in the formula is a function that indicates the similarity between two motion vectors MV. １ " is a reference value indicating the noise level. ２ " is a threshold value indicating the similarity condition. "MV(x+v ｘ , y+v ｙ )" is the background pixel (x+v) that is the reference of the foreground pixel (x, y). ｘ , y+v ｙ ) is the motion vector of

[0065] The example of Fig. 11 shows a case where the background is moving. As in the example of Fig. 10, when the background is moving, the occlusion area is detected as an area that satisfies both the motion determination regarding condition A and the similarity determination regarding condition B. However, as shown by the formula in Fig. 11, unlike when the foreground is moving, the coordinates of the determination result "OC(x, y)" are (x, y).

[0066] Although the explanation has been given here for the case where the foreground is moving ( FIG. 10 ) and the case where the background is moving ( FIG. 11 ), the foreground and background can be distinguished based on their respective depths. For example, the occlusion detection unit 50 distinguishes between the foreground and background based on depth information obtained from the depth image DP.

[0067] [2-4. Example of Bright Spot Detection Processing] Bright spot detection processing is performed based on, for example, the following criteria. The first criterion is whether or not the pixel value is close to white (high luminance near-white determination). Bright spots tend to have saturated luminance. Therefore, bright spots are often close to white with high luminance. If there is a pixel that is close to white with high luminance, it may be a bright spot. For example, the first criterion can be defined by the following formula (1). In formula (1), "Cur Ｒ ", "Cur Ｇ " and "Cur Ｂ " is the input current frame I Ｃ are the pixel values ​​of the red, green and blue pixels PX. Ｗ " and "th Ｓ " is a threshold value indicating the white equivalent level. The white equivalent level can be set arbitrarily by the system developer.

[0068]

[0069] The second criterion is whether or not there is a large variation in pixel values ​​between frames (determination of the magnitude of the difference between frames). When bright spot flicker occurs, pixel values ​​vary greatly between frames. If there is a pixel whose pixel value varies greatly between frames, it may be a bright spot. For example, the second criterion can be defined by the following formula (2). In formula (2), "Cur Ｒ ", "Cur Ｇ " and "Cur Ｂ" is the input current frame I Ｃ are the pixel values ​​of the red, green and blue pixels PX of "Hist Ｒ "," "Hist Ｇ " and "Hist Ｂ " is the history image I Ｈ，ＲＧＢ ＭＣ are the pixel values ​​of the red, green and blue pixels PX. ｄ " is a reference value indicating the noise level. The noise level can be set arbitrarily by the system developer.

[0070]

[0071] According to the above-mentioned criteria, the bright spot detection processing unit 60 can detect as a bright spot an image area that has a pixel value equivalent to white (first criteria) and in which the magnitude of the inter-frame difference is greater than a reference value indicating the noise level (second criteria).

[0072] The third criterion is whether or not a pixel is included in an occlusion region (non-occlusion region determination). Data disturbances called ghosts occur in occlusion regions. If ghost disturbances are confused with bright spots, data that should not be removed will be removed as bright spots. Therefore, it is desirable to exclude occlusion regions from the bright spot detection targets. The bright spot detection processing unit 60 can acquire non-occlusion regions from the occlusion map OC and selectively detect bright spots in non-occlusion regions.

[0073] The fourth criterion is whether the subject's movement is slow enough that the flickering of bright spots is easily noticeable (motion amount judgment). For fast-moving subjects, the flickering of bright spots is difficult to notice in the first place. To reduce the risk of damaging the image due to incorrect bright spot detection, it is desirable to avoid unnecessary bright spot removal processing. Therefore, the bright spot detection processor 60 sets a reference value for the subject's movement.

[0074] For example, the minimum magnitude of the motion vector MV at which the flickering of the bright spot is hardly perceptible is set as the reference value. A range of motion vectors greater than the reference value is deemed imperceptible. The bright spot detection processor 60 can exclude from the bright spot detection target image areas where the subject movement is determined to be greater than the reference value based on the motion vector MV.

[0075] The fourth determination condition can be defined by the following formula (3). ｍ " is a reference value indicating the imperceptible level. The imperceptible level can be set arbitrarily by the system developer. For example, the bright point detection processing unit 60 determines the reference value th indicating the imperceptible level from default information registered in the system. ｍ is obtained and substituted into equation (3).

[0076]

[0077] The fifth criterion is whether the video scene changes discontinuously (continuous frame determination). Ｃ If is the initial frame of rendering, or if a scene change occurs, the input current frame I Ｃ The image of the current frame is completely different from the image of the previous frame. If the bright point is determined based on the formula (1) and the formula (2), an erroneous determination may occur.

[0078] Therefore, the consecutive frame determination unit 40 uses the continuity determination information S Ｒ Based on the input current frame I Ｃ The bright spot detection processing unit 60 determines whether the input current frame I is a frame (continuous frame) that has continuity with respect to a common video scene with respect to the previous frame. Ｃ The bright spot detection process is performed only when the frames are consecutive frames.

[0079] [2-5. Example of Bright Spot Removal Processing] Bright spot removal processing is performed, for example, as follows. First, the bright spot removal processing unit 70 calculates the pixel value of a pixel PX that is deemed to display a bright spot using the following equation (4). In equation (4), "Cur'" is the input current frame I after bright spot removal. Ｃ(corrected current frame I ＣＲ ) pixel value. "Cur" is the input current frame I before bright spot removal. Ｃ "Hist" is the pixel value of the history image I Ｈ，ＲＧＢ ＭＣ "α" is a blending ratio, and the value of the blending ratio α is, for example, 0.9. The blending ratio α can be set arbitrarily by the system developer.

[0080]

[0081] The bright spot removal processing unit 70 receives the input current frame I Ｃ The pixel values ​​of the bright points included in the history image I Ｈ，ＲＧＢ ＭＣ Blend with the pixel value of the current frame I ＣＲ In equation (4), the input current frame I Ｃ and History Image I Ｈ，ＲＧＢ ＭＣ Although bright spots are removed by alpha blending in the above, the method for removing bright spots is not limited to alpha blending.

[0082] The bright spot removal processing unit 70 does not perform any particular bright spot removal processing on pixels PX that do not display bright spots. Ｃ The pixel values ​​of the corrected current frame I ＣＲ (see equation (5) below).

[0083]

[0084] 2-6. Processing Flow FIGS. 12 and 13 are diagrams showing an example of a processing flow for performing inference processing.

[0085] The scaler 30 receives the input current frame I from the renderer 20. Ｃ , motion vectors MV, depth image DP, and information (rendering information) related to rendering settings, etc. The scaler 30 scales the input current frame I as needed. Ｃ , the number of pixels of the motion vector MV and the depth image DP are enlarged, and the input current frame I Ｃ', motion vector MV' and depth image DP' are obtained (step S1).

[0086] The occlusion detection unit 50 obtains an occlusion map OC based on the depth image DP′ and the motion vector MV′ (step S2). The motion compensation unit 90 calculates intermediate features (inference history frame I) that are the DNN output from the previous frame based on the motion vector MV′. Ｈ ) and perform motion compensation on the corrected history frame I Ｈ ＭＣ (Step S3).

[0087] The continuous frame determination unit 40 receives the continuity determination information S Ｒ Based on the input current frame I Ｃ It is determined whether or not there is continuity between the frame ' and the previous frame which is the subject of inference immediately before (step S4).

[0088] If there is no continuity (step S4: No), the continuous frame determination unit 40 and the motion compensation unit 90 determine whether the input current frame I Ｃ ' and the correction history frame I Ｈ ＭＣ is input to the trained DNN 100 (step S5). The DNN 100 outputs the inference result, Ｏ The DNN 100 outputs the intermediate feature obtained by the inference to the output unit 110 as the final output. Ｈ to the motion compensation unit 90 (step S6). Thereafter, the recursive processing of the RNN is repeated.

[0089] If there is continuity (step S4: Yes), the history image estimation unit 80 converts the intermediate feature into the history image I using the conversion model CV. Ｈ，ＲＧＢ ＭＣ (Step S7). The bright spot detection processing unit 60 converts the input current frame I Ｃ ′, history image I Ｈ，ＲＧＢ ＭＣ , the motion vector MV′ and the occlusion map OC are used to calculate the input current frame I ＣThe bright spot removal processing unit 70 detects the image area of ​​the bright spot in the input current frame I' (step S8). Ｃ For the pixels showing bright points in the history image I Ｈ，ＲＧＢ ＭＣ Then, alpha blending is performed with the above to remove bright spots (step S9). After that, the process proceeds to step S5.

[0090] 3. Example of Hardware Configuration FIG. 14 is a diagram illustrating an example of the hardware configuration of the information processing device 1. As shown in FIG.

[0091] The information processing of the information processing device 1 is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0092] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0093] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0094] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0095] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0096] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0097] For example, when the computer 1000 functions as the information processing device 1 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0098] [4. Effects] The information processing device 1 includes a DNN 100, a history image acquisition unit 160, and a bright spot processing unit 150. The DNN 100 performs inference using past frames as input. The history image acquisition unit 160 acquires a history image I corresponding to the inference result. Ｈ，ＲＧＢ ＭＣ The bright point processing unit 150 obtains the input current frame I that has continuity with the previous frame. Ｃ , history image I Ｈ，ＲＧＢ ＭＣ , the inter-frame motion vector MV, and the occlusion map OC, the input current frame I Ｃ In the information processing method of the present disclosure, the processing of the information processing device 1 is executed by a computer 1000. A computer-readable non-transitory storage medium of the present disclosure stores a program that causes the computer 1000 to implement the processing of the information processing device 1.

[0099] This configuration effectively removes bright spots from the inference results, thereby suppressing image defects caused by flickering bright spots.

[0100] The history image acquisition unit 160 acquires the inference history frame I obtained as the inference result. Ｈ , and perform motion compensation to obtain the corrected history frame I Ｈ ＭＣ The history image acquisition unit 160 generates the corrected history frame I Ｈ ＭＣ Based on the history image I Ｈ，ＲＧＢ ＭＣ The bright spot processing unit 150 obtains the input current frame I Ｃ The pixel values ​​of the bright points included in the history image I Ｈ，ＲＧＢ ＭＣ Blend with the pixel value of the current frame I ＣＲ Generate.

[0101] According to this configuration, the input current frame I Ｃ Bright spots are effectively removed from the image.

[0102] The history image acquisition unit 160 stores the intermediate feature amount obtained during inference in an inference history frame I. ＨThe history image acquisition unit 160 acquires the inference history frame I Ｈ , and perform motion compensation to obtain the corrected history frame I Ｈ ＭＣ Generate.

[0103] This configuration improves the inference accuracy of the DNN 100 compared to when image data is used as input.

[0104] The history image acquisition unit 160 acquires the inference history frame I using a conversion model CV that converts the intermediate feature amount into an RGB image. Ｈ From History Image I Ｈ，ＲＧＢ ＭＣ Generate.

[0105] According to this configuration, the history image I Ｈ，ＲＧＢ ＭＣ can be easily obtained.

[0106] The history image acquisition unit 160 acquires the RGB image as the output image I of the DNN 100. Ｏ The history image acquisition unit 160 optimizes the transformation model CV so that it approximates the corrected history frame I. Ｈ ＭＣ The image obtained by applying Ｈ，ＲＧＢ ＭＣ It is calculated as follows.

[0107] According to this configuration, the motion-compensated output image I Ｏ A history image I that approximates Ｈ，ＲＧＢ ＭＣ is obtained.

[0108] The bright spot processing unit 150 receives the input current frame I Ｃ and History Image I Ｈ，ＲＧＢ ＭＣ The bright spot processing unit 150 obtains the difference between the input current frame I and the bright spot processing unit 150 based on the inter-frame difference. Ｃ Detect the bright spot.

[0109] According to this configuration, an image corresponding to an inferred image of a past frame (history image I Ｈ，ＲＧＢ ＭＣ ) is the inference history frame I ＨThe inferred image of the past frame itself (output image I Ｏ ) is not required, the implementation cost is reduced.

[0110] The bright spot processing unit 150 detects, as bright spots, image areas that have pixel values ​​at a level equivalent to white and in which the magnitude of the inter-frame difference is greater than a reference value indicating the noise level.

[0111] This configuration allows for good detection of bright spots with high brightness where flickering is easily noticeable.

[0112] The information processing device 1 includes an occlusion detection unit 50. The occlusion detection unit 50 detects occlusion regions where ghosts may occur based on the motion vectors MV, and generates an occlusion map OC. The bright spot processing unit 150 excludes occlusion regions from bright spot detection targets.

[0113] This configuration prevents ghost-induced image disturbances from being confused with bright spots, thereby preventing unnecessary bright spot processing.

[0114] The occlusion detection unit 50 determines, based on the motion vector MV, whether or not predetermined conditions are satisfied, that is, the magnitude of the positional deviation between the foreground and the background is greater than a reference value indicating the noise level, and the direction of movement of the foreground and the direction of movement of the background are different. If the predetermined conditions are satisfied, the occlusion detection unit 50 determines that the area of ​​the background that overlapped with the foreground is an occlusion area.

[0115] According to this configuration, the occlusion region can be accurately obtained.

[0116] The occlusion detector 50 distinguishes between the foreground and the background based on the depth information.

[0117] This configuration allows accurate discrimination between the foreground and the background.

[0118] The bright spot processing unit 150 excludes from the bright spot detection target image areas where it is determined that the motion of the subject is greater than a reference value based on the motion vector MV.

[0119] This configuration reduces unnecessary processing of bright spots and also reduces the risk of adverse effects on the image due to incorrectly determining bright spots.

[0120] The information processing device 1 includes a consecutive frame determination unit 40. The consecutive frame determination unit 40 determines whether an input current frame I Ｃ The bright point processing unit 150 determines the continuity between the input current frame I and the previous frame. Ｃ Selectively detect bright spots on the

[0121] This configuration prevents flicker caused by discontinuity in the image from being confused with bright spots, and prevents unnecessary processing of bright spots.

[0122] The bright point processing unit 150 converts the rendering image generated using the TAA into the input current frame I Ｃ The bright spot processing unit 150 obtains the specular bright spots generated by the TAA as the input current frame I Ｃ Remove from.

[0123] This configuration suppresses flickering of high-brightness bright spots that is likely to occur due to TAA.

[0124] In the above explanation, the inferred image of the past frame itself (output image I Ｏ ) is not used. Ｏ In a configuration example in which a VRAM for storing the inference results is pre-installed, the DNN 100 may output and record the inference results as an RGB image in the VRAM. For example, the DNN 100 may convert the RGB image obtained by inference into the output image I Ｏ The history image acquisition unit 160 outputs the history image I Ｈ，ＲＧＢ ＭＣ As a result, the input current frame I is Ｃ The RGB image corresponding to the

[0125] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0126] [Additional Notes] The present technology may also be configured as follows: (1) An information processing device comprising: a DNN that performs inference using past frames as input; a history image acquisition unit that acquires a history image corresponding to an inference result; and a bright spot processing unit that removes bright spots from the input current frame based on an input current frame that has continuity with the past frame, the history image, an inter-frame motion vector, and an occlusion map. (2) The information processing device described in (1) above, wherein the history image acquisition unit performs motion compensation on the inference history frame obtained as the inference result to generate a corrected history frame and acquires the history image based on the corrected history frame, and the bright spot processing unit blends pixel values ​​of the bright spots included in the input current frame with pixel values ​​of the history image to generate the corrected current frame. (3) The information processing device according to (2), wherein the history image acquisition unit acquires intermediate features obtained during inference as the inference history frame and performs the motion compensation on the inference history frame to generate the corrected history frame. (4) The information processing device according to (3), wherein the history image acquisition unit generates the history image from the inference history frame using a conversion model that converts the intermediate features into an RGB image. (5) The information processing device according to (4), wherein the history image acquisition unit optimizes the conversion model so that the RGB image approximates an output image of the DNN, and calculates, as the history image, an image obtained by applying the optimized conversion model to the corrected history frame. (6) The information processing device according to any one of (1) to (5), wherein the bright spot processing unit acquires a difference between the input current frame and the history image as an inter-frame difference and detects bright spots in the input current frame based on the inter-frame difference. (7) The information processing device according to (6), wherein the bright spot processing unit detects, as the bright spot, an image area having a pixel value at a level corresponding to white and in which the magnitude of the inter-frame difference is greater than a reference value indicating a noise level.(8) The information processing device according to (7), further comprising an occlusion detection unit that detects an occlusion region where ghosts may occur based on the motion vector and generates the occlusion map, wherein the bright point processing unit excludes the occlusion region from targets for detecting the bright points. (9) The information processing device according to (8), wherein the occlusion detection unit determines, based on the motion vector, whether or not predetermined conditions are satisfied, that the magnitude of positional deviation between the foreground and the background is greater than a reference value indicating a noise level and that the movement direction of the foreground and the movement direction of the background are different, and determines that the region of the background that overlapped with the foreground is the occlusion region based on the determination that the predetermined conditions are satisfied. (10) The information processing device according to (9), wherein the occlusion detection unit distinguishes between the foreground and the background based on depth information. (11) The information processing device according to any one of (7) to (10), wherein the bright spot processing unit excludes from the detection target for the bright spot an image area in which it is determined that the movement of the subject is greater than a reference value based on the motion vector. (12) The information processing device according to any one of (6) to (11), wherein the information processing device has a consecutive frame determination unit that determines the continuity between the current input frame and the previous frame, and the bright spot processing unit selectively detects the bright spot for the current input frame that has the continuity. (13) The information processing device according to any one of (1) to (12), wherein the bright spot processing unit obtains a rendering image generated using TAA (Temporal Anti-Aliasing) as the current input frame, and removes the bright spot of specular generated by the TAA from the current input frame. (14) The information processing device according to any one of (1) to (13), wherein the DNN outputs an RGB image obtained by inference to a VRAM as an output image, and the history image acquisition unit acquires the RGB image corresponding to the input current frame from the VRAM as the history image.(15) An information processing method executed by a computer, comprising: performing inference using a past frame as an input; obtaining a history image corresponding to the inference result; and removing bright spots from the input current frame based on an input current frame that has continuity with the past frame, the history image, the motion vector between frames, and an occlusion map. (16) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: performing inference using a past frame as an input; obtaining a history image corresponding to the inference result; and removing bright spots from the input current frame based on an input current frame that has continuity with the past frame, the history image, the motion vector between frames, and an occlusion map.

[0127] 1 Information processing device 40 Continuous frame determination unit 50 Occlusion detection unit 100 DNN 150 Bright spot processing unit 160 History image acquisition unit CV Conversion model I Ｃ , I Ｃ ' Input current frame I ＣＲ Corrected current frame I Ｈ Inference History Frame I Ｈ ＭＣ Correction History Frame I Ｈ，ＲＧＢ ＭＣ History image MV, MV' Motion vector OC Occlusion map

Claims

1. An information processing device having a DNN that performs inference using past frames as input, a history image acquisition unit that acquires a history image corresponding to the inference result, and a bright spot processing unit that removes bright spots from the input current frame based on an input current frame that has continuity with the past frames, the history image, motion vectors between frames, and an occlusion map.

2. The information processing device of claim 1, wherein the history image acquisition unit performs motion compensation on the inference history frame obtained as the inference result to generate a corrected history frame and acquires the history image based on the corrected history frame, and the bright spot processing unit blends the pixel values ​​of the bright spots included in the input current frame with the pixel values ​​of the history image to generate a corrected current frame.

3. The information processing device according to claim 2, wherein the history image acquisition unit acquires intermediate features obtained during inference as the inference history frame, and performs the motion compensation on the inference history frame to generate the corrected history frame.

4. The information processing device according to claim 3, wherein the history image acquisition unit generates the history image from the inference history frame using a conversion model that converts the intermediate feature amount into an RGB image.

5. The information processing device according to claim 4, wherein the history image acquisition unit optimizes the transformation model so that the RGB image approximates an output image of the DNN, and calculates an image obtained by applying the optimized transformation model to the corrected history frame as the history image.

6. The information processing device according to claim 1, wherein the bright spot processing unit obtains a difference between the current input frame and the history image as an inter-frame difference, and detects bright spots in the current input frame based on the inter-frame difference.

7. The information processing device according to claim 6, wherein the bright spot processing section detects, as the bright spot, an image area having a pixel value of a level corresponding to white and in which the magnitude of the inter-frame difference is greater than a reference value indicating a noise level.

8. The information processing device according to claim 7, further comprising an occlusion detection unit that detects an occlusion area where ghosts may occur based on the motion vector and generates the occlusion map, and the bright spot processing unit excludes the occlusion area from the detection targets for the bright spots.

9. The information processing device of claim 8, wherein the occlusion detection unit determines, based on the motion vector, whether or not predetermined conditions are satisfied, that the magnitude of the positional shift between the foreground and the background is greater than a reference value indicating a noise level, and that the direction of movement of the foreground and the direction of movement of the background are different, and determines that the area of ​​the background that overlapped with the foreground is the occlusion area, based on a determination that the predetermined conditions are satisfied.

10. The information processing device according to claim 9, wherein the occlusion detection unit distinguishes between the foreground and the background based on depth information.

11. The information processing device according to claim 7, wherein the bright spot processing section excludes an image area in which the motion of the subject is determined to be greater than a reference value based on the motion vector from the bright spot detection targets.

12. The information processing device according to claim 6, further comprising a consecutive frame determination unit that determines the continuity between the input current frame and the past frame, and the bright spot processing unit selectively performs the detection of the bright spot for the input current frame having the continuity.

13. The information processing device according to claim 1, wherein the bright spot processing unit obtains a rendering image generated using TAA (Temporal Anti-Aliasing) as the input current frame, and removes the bright spots of specular caused by the TAA from the input current frame.

14. The information processing device according to claim 1, wherein the DNN outputs an RGB image obtained by inference to a VRAM as an output image, and the history image acquisition unit acquires the RGB image corresponding to the input current frame from the VRAM as the history image.

15. An information processing method executed by a computer, comprising: performing inference using past frames as input; obtaining a history image corresponding to the inference result; and removing bright points from the input current frame based on an input current frame that has continuity with the past frames, the history image, inter-frame motion vectors, and an occlusion map.

16. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: perform inference using past frames as input; obtain a history image corresponding to the inference result; and remove bright points from the input current frame based on an input current frame that has continuity with the past frames, the history image, inter-frame motion vectors, and an occlusion map.

Images