Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device comprises a motion compensation unit, an occlusion region detection unit, a ghost removal unit, and a DNN. The motion compensation unit performs motion compensation on a past frame inference result and generates a correction history. The occlusion region detection unit detects an occlusion region in the correction history in which a ghost may occur, on the basis of a motion vector indicating motion of a subject between frames. The ghost removal unit performs a ghost removal process with respect to the occlusion region. The DNN infers the current frame on the basis of a correction history in which the ghost removal process has been performed and a current, which is an image of the current frame subject to processing.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Deep neural network (DNN) processing for video generally uses a recurrent neural network (RNN) structure with high time stability. This is because the RNN structure can correlate current (current information) with history (past information), making it easier to maintain consistency in the image of the output frame.

[0003] Special Publication No. 2022-547517

[0004] The history, along with the current, is used as input to the DNN. To ensure consistency with the current, the history undergoes a correction called motion compensation, which takes into account inter-frame motion. Motion compensation is a process that predicts the data of the current frame from past frames based on inter-frame motion. However, if the background becomes exposed due to foreground movement, the data for that background cannot be predicted from the history. As a result, data disturbances called ghosts occur in the history after motion compensation (corrected history). Ghosts affect the inference results of the DNN and cause a decrease in video quality.

[0005] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of suppressing image disturbances caused by ghosts.

[0006] According to the present disclosure, there is provided an information processing device including a motion compensation unit that performs motion compensation on an inference result of a past frame to generate a correction history, an occlusion region detection unit that detects an occlusion region in the correction history where ghosts may occur based on a motion vector that indicates the movement of a subject between frames, a ghost removal unit that performs ghost removal processing on the occlusion region, and a DNN that infers the current frame based on the correction history that has undergone the ghost removal processing and a current that is an image of the current frame to be processed.The present disclosure also provides an information processing method in which information processing of the information processing device is executed by a computer, and a computer-readable non-transitory storage medium that stores a program that causes a computer to realize the information processing of the information processing device.

[0007] 1 is a diagram illustrating a conventional DNN process for moving images; FIG. 2 is a diagram illustrating the occurrence of ghosts due to motion compensation; FIG. 3 is a diagram illustrating the occurrence of ghosts due to motion compensation; FIG. 4 is a diagram illustrating an example of a configuration of an information processing device that performs DNN processing for moving images according to the present disclosure; FIG. 5 is a diagram illustrating an example of ghost removal processing; FIG. 6 is a diagram illustrating a DNN learning method; FIG. 7 is a diagram illustrating a DNN learning method; FIG. 8 is a diagram illustrating an example of a method for detecting an occlusion region; FIG. 9 is a diagram illustrating another example of a configuration that performs DNN processing for moving images according to the present disclosure; FIG. 10 is a diagram illustrating an example of ghost removal processing; FIG. 11 is a diagram illustrating an example of a method for calculating approximate features; FIG. 12 is a diagram illustrating intermediate features; FIG. 13 is a diagram illustrating a DNN learning method; FIG. 14 is a diagram illustrating an example of a processing flow for performing inference processing; FIG. 15 is a diagram illustrating an example of a processing flow for performing inference processing;

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. Background] [1-1. Conventional DNN processing for moving images] [1-2. Occurrence of ghosts due to motion compensation] [2. Example 1 of DNN processing for moving images according to the present disclosure] [2-1. System configuration example] [2-2. Ghost removal processing] [2-3. Detection of occlusion area] [2-4. Effects] [3. Example 2 of DNN processing for moving images according to the present disclosure] [3-1. System configuration example] [3-2. Ghost removal processing using approximate features] [3-3. Processing flow] [3-4. Effects] [4. Hardware configuration example]

[0010] [1. Background] [1-1. Conventional DNN Processing for Video] FIG. 1 is a diagram showing conventional DNN processing for video.

[0011] DNN processing for video generally uses an RNN structure with high temporal stability. The RNN structure can correlate the current and history, making it easier to maintain consistency in the image of the output frame. While DNNs have very high inference performance (for example, sharpening effects in the case of super-resolution), they are affected by small changes in the input data and the output results change. As a result, in video processing, flickering is more likely to occur than in simple filter processing. To increase temporal stability and suppress flickering, an RNN structure is generally implemented that inputs the current and history simultaneously and enables learning that includes temporal correlation (similarity between the current and history).

[0012] Note that "current" refers to the image of the frame (current frame) that is the estimation target of the DNN. "history" refers to an image (estimated image) obtained by inputting an image of a past frame into the DNN, or intermediate features extracted from an image of a past frame. "Past frame" refers to a frame that is one or more frames before the current frame. In the present disclosure, for example, the frame that is one frame before the current frame (the most recent frame) is used as the past frame.

[0013] The intermediate features refer to information on the features output from the intermediate layer of a DNN when images of past frames are input to the DNN. It is known that using the intermediate features of past frames as history can produce more accurate estimation results than using images of past frames as history.

[0014] The history is used as input to the DNN along with the current. To ensure consistency with the current, the history is motion compensated based on a motion vector. A motion vector is a vector that indicates the movement of a subject between frames (the amount and direction of pixel movement). Motion compensation is the process of predicting post-motion data from pre-motion data. For example, when motion compensation is applied to the shape and features of an image in a past frame, the shape and features of the image predicted in the current frame are obtained as the corrected history. The history after motion compensation (corrected history) is input to the DNN along with the current.

[0015] [1-2. Occurrence of Ghosts Due to Motion Compensation] FIGS. 2 and 3 are diagrams for explaining the occurrence of ghosts due to motion compensation.

[0016] FIG. 2 shows an example in which a moving 3D object moves 50 pixels to the right as the frames progress. "t" indicates the frame number. "Current" is the image of the current frame (t=0). "History" is the inferred image of the previous frame (t=-1). The motion vector MV is a two-dimensional vector that indicates the movement of the subject. The motion vector MV is defined for each pixel of the current frame.

[0017] For example, let's say that the horizontal direction is the x direction and the vertical direction is the y direction. Since the 3D object is moving from the left side (-x direction), the motion vector MV of the drawing position of the 3D object is (-50, 0). Since there is no movement in the stationary part, the motion vector MV is (0, 0). The motion vector MV is expressed as a vector pointing from the current pixel position to the pixel position in the history that is the source of the movement (where it came from).

[0018] The corrected history is obtained by performing motion compensation on the history, which is performed by applying pixel values ​​in the history to destination pixels determined from the motion vector.

[0019] For example, let us consider an image region in which a 3D object is presented as a moving object region. The motion vector MV of the moving object region is (-50, 0). Therefore, in the current frame (t = 0), the moving object region has moved to a position shifted 50 pixels to the right compared to the previous frame (t = -1). Therefore, the pixel values ​​of the moving object region in the history are applied to an image region shifted 50 pixels to the right from the moving object region. For image regions other than the moving object region, the motion vector MV is (0, 0). Therefore, no change has occurred in the image between the current frame and the previous frame. Therefore, pixel values ​​are not replaced.

[0020] When a moving 3D object moves, the part hidden behind it (occlusion area) becomes visible. Because the occlusion area is only revealed in the current image, it cannot be predicted from the history. Even if motion compensation is performed on the occlusion area, the pixel value to be referenced based on the motion vector becomes unknown. As a result, data disturbances (ghosts) that indicate traces of the 3D object appear in the history (corrected history) after motion compensation. The DNN performs a process that blends the current image and the corrected history. If a corrected history containing ghosts is input to the DNN, an image is generated in which the 3D object before movement is reflected like a shadow, causing a decrease in image quality.

[0021] The present disclosure has been made in view of the above circumstances. In the present disclosure, an occlusion region in a correction history where ghosts may occur is detected based on a motion vector MV. Then, a ghost removal process is performed on the occlusion region. By performing DNN inference based on the correction history that has undergone the ghost removal process, image distortion due to ghosts is suppressed.

[0022] In conventional methods, disturbed areas (noise regions) are detected from the inference image, and the DNN is optimized using a loss function weighted toward the noise regions. However, this method weakens the optimization of non-noise regions, resulting in a decrease in inference accuracy. Furthermore, detecting noise regions from the inference image is difficult, even with a dedicated discriminative model. For example, blurring caused by pixel value replacement during motion compensation and flickering (non-ghost regions) as a visual effect are difficult to distinguish from image disturbances caused by ghosts.

[0023] The method disclosed herein accurately identifies data areas (occlusion areas) in the correction history where ghosting may occur based on motion vectors. Because ghosting is removed by directly correcting the data in the occlusion areas, the accuracy of estimating non-ghost areas, such as blur and flicker, is less likely to decrease. Therefore, compared to conventional methods, image distortion caused by ghosting is more effectively suppressed. A detailed description is provided below.

[0024] [2. DNN Processing Example 1 for Video of the Present Disclosure] [2-1. System Configuration Example] Fig. 4 is a diagram showing a configuration example of an information processing device 1 that performs DNN processing for video of the present disclosure. In the example of Fig. 4, Ｈ As the image, the inferred image of the past frame inferred by the DNN is used.

[0025] The DNN processing for moving images according to the present disclosure is performed by an information processing device 1. For example, the information processing device 1 includes an input unit 10, a scaler 20, a DNN 30, a motion compensation unit 40, an occlusion region detection unit 50, a ghost removal unit 60, and an output unit 70.

[0026] The input unit 10 receives the current I Ｃ , obtain the depth image DP and the motion vector MV. Ｃ refers to a frame of image in a series of frames, such as a video. Ｃ The image may be an image captured by a camera or a CG image generated by a CG renderer. The block diagram in Fig. 4 explains the operation in one frame of a video.

[0027] The depth image DP is a one-dimensional image that represents depth. The depth image DP can be obtained by estimation using a group of consecutive frames or by estimation using a depth estimation DNN. When using a CG renderer, the depth image DP can be generated and obtained during rendering.

[0028] A motion vector MV is a two-dimensional vector that defines the movement of corresponding pixels between frames by the amount of pixel movement (number of pixels) in the x and y directions. The motion vector MV can be obtained by a general estimation process using a group of consecutive frames. When using a CG renderer, the motion vector MV can be generated and obtained during rendering.

[0029] The scaler 20 adjusts the current I Ｃ , the number of pixels of the depth image DP and the motion vector MV is enlarged. Hereinafter, when it is necessary to distinguish data after the number of pixels has been enlarged by the scaler 20 from data before enlargement, a "'" is added after the symbol of the data. The scaler 20 Ｃ The scaler 20 sends the depth image DP' to the DNN 30 and the ghost removal unit 60. The scaler 20 sends the depth image DP' to the occlusion region detection unit 50. The scaler 20 sends the motion vector MV' to the occlusion region detection unit 50 and the motion compensation unit 40.

[0030] The architecture of the DNN 30 accepts a fixed number of pixels, while the current I Ｃ If the number of pixels in the current I changes depending on the usage, the scaler 20 may perform a pixel number expansion process at a desired magnification so as to obtain a pixel number that matches the specifications of the DNN 30. Ｃ Regarding the extension method of the depth image DP and the motion vector MV, it is desirable to apply the nearest neighbor method to avoid generation of intermediate values ​​due to pixel interpolation, but the method is not limited to this.

[0031] DNN30 is the current I Ｃ ' and the correction history I after ghost removal ＨＭＣ (Corrective History I Ｈｒ ) as input and output image I Ｏ The DNN 30 outputs the output image I Ｏ to the output unit 70. Pre-trained values ​​are used as coefficient values ​​for the DNN 30. Learning tasks for the DNN 30 are expected to include, for example, super-resolution, noise reduction, and style transfer, but the present disclosure is not limited thereto. The DNN 30 is applicable to general video processing in general using an RNN structure.

[0032] The motion compensation unit 40 obtains a history IH, which is an inference result of a past frame, from the DNN 30. In the example of Fig. 4, the history IH is an inferred image of a past frame inferred by the DNN 30. The motion compensation unit 40 performs motion compensation on the history IH based on the motion vector MV' to obtain a corrected history IH. Ｈ ＭＣ The motion compensation unit 40 can apply a common process known as motion compensation to the history IH.

[0033] The motion vector MV′ is Ｈ From Current I Ｃ By motion compensation, the history I Ｈ and Current I Ｃ The positions of the corresponding pixels in the history I match. Ｈ (Correction History I Ｈ ＭＣ ) is sent to the ghost removal unit 60.

[0034] The occlusion region detection unit 50 detects a correction history I where ghosts may occur based on the depth image DP′ and the motion vector MV′. Ｈ ＭＣ The data area within the frame is detected as an occlusion area. Any method may be used to detect the occlusion area. For example, the occlusion area may be identified from depth information of the current frame and past frames.

[0035] The occlusion region detection unit 50 generates an occlusion map OC in which occlusion regions are labeled, and sends the occlusion map OC to the ghost removal unit 60. For example, the occlusion map OC labels occlusion regions and regions other than occlusion regions (non-occlusion regions) with occlusion values. For example, an occlusion value of "0" is set for each pixel included in an occlusion region, and an occlusion value of "1" is set for each pixel included in a non-occlusion region.

[0036] The ghost removal unit 60 performs ghost removal processing on the occlusion region. The ghost removal unit 60 stores the correction history I after the ghost removal processing. Ｈ ＭＣ (Corrective History I Ｈｒ ) to the DNN 30. The DNN 30 sends the correction history I Ｈｒ and current I, which is the image of the current frame to be processed. Ｃ The DNN 30 performs inference for the current frame based on the inference results, i.e., the output image I Ｏ The output unit 70 sends the output image I obtained from the DNN 30. Ｏ Apply common post-processing such as color conversion and codec to the image.

[0037] For example, the ghost removal unit 60 can perform ghost removal processing by replacing the pixel value of each pixel in the occlusion region with blank data. Ｃ It is also possible to perform processing to replace the pixel values ​​of the image data with those of the image data ...

[0038] Blank data refers to data that indicates a specific value set by the system. Blank data can be set arbitrarily by the system developer. For example, blank data is set as data indicating 0, but data indicating an intermediate gray level may also be set as blank data. An intermediate gray level refers to a gray level between the minimum and maximum gray levels that can be displayed.

[0039] The DNN 30 calculates the pixel values ​​of the pixels in the occlusion area from the current IＣ Therefore, if 0, which indicates the lowest gradation, is used as the blank data, a high-luminance image is generated by blending the pixel value of the current I' with the blank data. Ｃ If the input is "0", the brightness may be insufficient. Ｃ In the method using the pixel value of ', there is little possibility of insufficient brightness.

[0040] 2-2. Ghost Removal Processing FIGS. 5 and 6 are diagrams showing an example of ghost removal processing.

[0041] In the example of Figure 5, data indicating "0" is used as blank data. In the RNN structure DNN 30, ghosts can occur due to motion compensation. Therefore, a process is performed to remove ghosts by directly correcting the occlusion area identified from the motion vector MV' etc. with blank data.

[0042] In the example of FIG. 5, the occlusion area is an image area where a 3D object was present in the previous frame. Ｈｒ The pixel values ​​in the occlusion region of the output image I are replaced with the lowest gray level, 0 (black). Ｏ However, the ghost of the current I Ｃ The background color of the ' is blended with the black of the blank data, resulting in a background color that is slightly less bright, which may leave artifacts.

[0043] In the example of FIG. 6, the correction history I Ｈｒ The pixel values ​​of the occlusion area of ​​the current I Ｃ The pixel value of the current I Ｃ Corrective history blended into 'I' Ｈｒ The background color of the same current Ｃ Therefore, the brightness of the background color is less likely to decrease compared to when correction is performed using blank data as in the example of FIG. 5. This makes it possible to remove ghosts with high quality.

[0044] 7 and 8 are explanatory diagrams of the learning method of the DNN 30. FIG.

[0045] The learning method of the DNN 30 differs depending on how the occlusion region is handled in actual operation. For example, the example of FIG. 7 is a learning example assuming the example of FIG. 5 as actual operation. During learning, the corrected history I in which the occlusion region is replaced with blank data is used, as in actual operation. Ｈｒ is used as the input of the DNN 30.

[0046] DNN30 has Correction History I Ｈｒ and Current I Ｃ is input, and the output image I Ｏ is obtained. The output image I Ｏ and teacher image I Ｔ The DNN 30 is optimized so that the difference between the training image I and the training image I is small. Ｔ is determined depending on the application of the DNN 30. For example, when the purpose is super-resolution, the current I Ｃ The image with higher resolution than the training image I Ｔ It is used as.

[0047] Corrected history I for input of DNN30 Ｈｒ By using the corrected history I, the inference accuracy of the DNN 30 can be improved during actual operation. Ｈｒ Instead of Correction History I Ｈ ＭＣ If the data is input to the DNN 30, the coefficients of the DNN 30 will be optimized with the data containing ghosts. In this case, the data without ghosts (corrected history I Ｈｒ ) may result in a decrease in inference accuracy. As mentioned above, by aligning the data used as input during training and during actual operation, appropriate training and inference can be achieved.

[0048] The example of FIG. 8 is a learning example assuming the example of FIG. 6 as an actual operation. In this example, too, during learning, the occlusion area is set to the current I, as in the actual operation. Ｃ Corrected history I replaced with pixel values ​​of Ｈｒ is used as the input of the DNN 30. The DNN 30 includes a correction history I Ｈｒ and Current I Ｃis input, and the output image I Ｏ is obtained. The output image I Ｏ and teacher image I Ｔ The DNN 30 is optimized so that the difference between the two is small. This improves the inference accuracy of the DNN 30 during actual operation.

[0049] 2-3. Detection of Occlusion Regions FIGS. 9 and 10 are diagrams showing an example of a method for detecting occlusion regions.

[0050] The occlusion region detection unit 50 detects the movement of a moving object in the foreground and the movement of the background hidden behind the moving object (an image region referenced by the foreground using a motion vector) based on the motion vector MV. The occlusion region detection unit 50 detects an occlusion region based on the movement of the foreground and background.

[0051] For example, the occlusion region detection unit 50 calculates the magnitude of positional shift and the similarity of the movement direction between the foreground and background based on the motion vector MV. The occlusion region detection unit 50 determines that the background region that overlapped with the foreground is an occlusion region when the magnitude of the positional shift is not at the noise level (condition A) and the similarity of the movement direction does not satisfy the similarity condition (condition B). In the present disclosure, "the similarity between the movement direction of the foreground and the movement direction of the background does not satisfy the similarity condition" may be simply referred to as "the movement direction of the foreground and the movement direction of the background are different."

[0052] The similarity can be calculated using vector distance, cosine similarity, or the like. The noise level means that the magnitude of the positional shift is large enough to be considered noise. In other words, condition A may be considered to be a condition in which the magnitude of the positional shift between the foreground and background is greater than a threshold representing noise. The similarity condition means a condition for determining similarity. The noise level and similarity condition are hyperparameters that affect the inference accuracy of the DNN 30. These hyperparameters can be arbitrarily set by the system developer using thresholds, etc., while checking the ghosts that actually occur due to motion compensation.

[0053] That is, based on the motion vector MV, the occlusion region detection unit 50 determines whether or not the following predetermined conditions are satisfied: the magnitude of the positional deviation between the foreground and the background is greater than a threshold representing noise (condition A), and the direction of movement of the foreground and the direction of movement of the background are different (condition B). Based on the determination that the predetermined conditions are satisfied, the occlusion region detection unit 50 determines that the region of the background that overlapped with the foreground is an occlusion region.

[0054] The example in Figure 9 shows a case where the foreground is moving. If the reference of the foreground is smaller than the amount of movement of the foreground, it can be assumed that occlusion has occurred. In Figure 9, the background is stationary and the foreground is moving to the left. The motion vector MV of the foreground pixel (x, y) is (v ｘ , v ｙ ) When the foreground is moving, the occlusion area is an area that satisfies both the motion determination regarding the above-mentioned condition A (whether the moving object in the foreground has moved from its original location) and the similarity determination regarding the condition B (whether the original location has moved in the same direction as the moving object). Condition B suggests that occlusion will not occur when the background moves at a similar speed to the foreground.

[0055] For example, for a pixel (x, y), the occlusion value OC(x+v ｘ , y+v ｙ ) can be determined based on the formula shown in FIG. 9 as being "0" (included in the occlusion region) or "1" (not included in the occlusion region). "Similarity" in the formula is a function that indicates the similarity between two motion vectors MV. １ " is a threshold value indicating the noise level. ２ " is a threshold value indicating the similarity condition. "MV(x+v ｘ , y+v ｙ )" is the background pixel (x+v) that is the reference of the foreground pixel (x, y). ｘ , y+v ｙ ) is the motion vector of

[0056] The example in Fig. 10 shows a case where the background is moving. As in the example in Fig. 9, when the background is moving, the occlusion area is detected as an area that satisfies both the motion determination regarding condition A and the similarity determination regarding condition B. However, as shown by the formula in Fig. 10, unlike when the foreground is moving, the coordinates of the determination result "OC(x, y)" are (x, y).

[0057] Here, the explanation has been given separately for the case where the foreground is moving (FIG. 9) and the case where the background is moving (FIG. 10), but the foreground and background can be distinguished based on their respective depths. For example, the occlusion region detection unit 50 distinguishes between the foreground and background based on depth information obtained from the depth image DP.

[0058] [2-4. Effects] The information processing device 1 includes a motion compensation unit 40, an occlusion region detection unit 50, a ghost removal unit 60, and a DNN 30. The motion compensation unit 40 performs motion compensation on the inference results of past frames to generate a corrected history I. Ｈ ＭＣ The occlusion region detection unit 50 generates a correction history I where ghosts may occur based on a motion vector MV that indicates the movement of the subject between frames. Ｈ ＭＣ The ghost removal unit 60 performs ghost removal processing on the occlusion area. The DNN 30 uses the correction history I where the ghost removal processing has been performed. Ｈ ＭＣ (Corrective History I Ｈｒ ) and the current frame image to be processed, Ｃ In the information processing method disclosed herein, the processing of the information processing device 1 is executed by a computer. The computer-readable non-transitory storage medium disclosed herein stores a program that causes a computer to realize the processing of the information processing device 1.

[0059] According to this configuration, the correction history I is generated based on the motion vector MV. Ｈ ＭＣ The occlusion area in the image is identified. The correction history I in which the ghost removal process is performed on the occlusion area is created. Ｈ ＭＣ(Corrective History I Ｈｒ ), the DNN 30 performs its inference based on the eigenvalues ​​.gamma., thereby better suppressing ghost artifacts.

[0060] The motion compensation unit 40 performs motion compensation on the inferred image of the past frame inferred by the DNN 30 to generate a correction history I. Ｈ ＭＣ Generate.

[0061] According to this configuration, the correction history I can be calculated by simple calculation. Ｈ ＭＣ Also, the correction history I Ｈ ＭＣ is Current I Ｃ Since the image data is similar to the current I Ｃ The estimation accuracy of ′ improves.

[0062] The ghost removal unit 60 performs ghost removal processing by replacing the pixel value of each pixel in the occlusion area with blank data.

[0063] According to this configuration, ghosts are suppressed by simple calculations.

[0064] As a ghost removal process, the ghost removal unit 60 calculates the pixel value of each pixel in the occlusion area by the current I Ｃ Then, the pixel values ​​of ...

[0065] This configuration effectively removes ghosts.

[0066] The occlusion region detection unit 50 determines, based on the motion vector MV, whether or not predetermined conditions are satisfied, that is, the magnitude of the positional deviation between the foreground and the background is greater than a threshold representing noise, and the direction of movement of the foreground and the direction of movement of the background are different. Based on the determination that the predetermined conditions are satisfied, the occlusion region detection unit 50 determines that the region of the background that overlapped with the foreground is an occlusion region.

[0067] According to this configuration, the occlusion region can be accurately obtained.

[0068] The occlusion region detector 50 distinguishes between the foreground and the background based on depth information.

[0069] This configuration allows accurate discrimination between the foreground and the background.

[0070] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0071] 3. Example 2 of DNN Processing for Moving Images of the Present Disclosure 3-1. Example of System Configuration Fig. 11 is a diagram showing an example of the configuration of another information processing device 2 that performs DNN processing for moving images of the present disclosure. Fig. 12 is a diagram showing an example of ghost removal processing.

[0072] The difference between this example and the information processing device 1 in FIG. 4 is that the history Ｈ The intermediate features of past frames inferred by the DNN are used as the corrected history I Ｈ ＭＣ The data of the occlusion area is the feature that approximates the current intermediate feature (approximate feature I Ｈａ ) are substituted. The following mainly describes the differences from the information processing device 1.

[0073] The information processing device 2 includes an approximate feature estimation unit 80. The motion compensation unit 40 estimates intermediate features (history I) of past frames inferred by the DNN 30. Ｈ ) is subjected to motion compensation to obtain the correction history I Ｈ ＭＣ The approximate feature estimation unit 80 generates the current I Ｃ ' to correction history I Ｈ ＭＣ The feature equivalent to the approximate feature I Ｈａ The ghost removal unit 60 performs ghost removal processing by acquiring the correction history I Ｈ ＭＣ The data of the occlusion area in the Ｈａ The data is replaced with the following data.

[0074] The motion compensation unit 40 may use the inference image of the past frame as a history I depending on the situation. ＨFor example, in a dark and monotonous video scene, it is not a problem if the inference quality is somewhat low. In that case, it is possible to replace the occlusion area with blank data as shown in Figure 5 without using approximate features that require a large amount of calculation. Ｈ The process of using the inference image of the past frame as the inference image is the same as that described in the example of FIG.

[0075] [3-2. Ghost Removal Processing Using Approximate Feature Amount] FIG. 13 shows the ghost removal processing using approximate feature amount I Ｈａ 14 is a diagram illustrating an example of a calculation method of intermediate feature amounts.

[0076] The intermediate feature refers to the internal feature of the DNN 30. Ｃ is input to the DNN 30, and the feature z ｎ，ｌ Here, "n" represents the number of dimensions, and "l" represents the layer number. The DNN 30 is expressed with four layers.

[0077] The final output of DNN30 is z １，３ ~z ６４，３ 14, the number of dimensions is 64, but the number of dimensions is determined by the configuration of the DNN 30 and does not necessarily have to be 64.

[0078] In the RNN structure, the inference accuracy is improved by using such multidimensional features rather than recursively using RGB images as input. Ｈ The intermediate features recursively used as the multidimensional features (z １，ｌ ~z ｎ，ｌ In practice, it is desirable to use the feature just before the final layer (the output of the third layer in the example of FIG. 14 ), which is closer to the output, i.e., which is a feature that better represents the training data.

[0079] The approximate feature estimation unit 80 estimates the current I Ｃ Apply the conversion process to the pixel values ​​of ＨFor example, the approximate feature estimation unit 80 approximates the feature of the current I Ｃ The intermediate feature obtained by applying the conversion model CV to the current I Ｃ The intermediate feature I obtained by inputting Ｈｉ The approximate feature estimation unit 80 optimizes the conversion model CV so that it approximates the current I Ｃ The intermediate feature obtained by applying Ｈａ It is calculated as follows.

[0080] The optimization of the conversion model can be performed using a general regression model. FIG. 13 shows an example of multiple regression analysis. For example, the zeroth-dimensional feature FM ０ is FM ０ = c ０，Ｙ Cur Ｒ +c ０，Ｙ Cur Ｇ +c ０，Ｙ Cur Ｂ +b ０ The coefficient c ０，Ｙ and b ０ is obtained by multiple regression analysis. Ｒ ", "Cur Ｒ " and "Cur Ｒ " is the current Ｃ "c" and "b" denote coefficients of the conversion model CV.

[0081] In the example of FIG. 13, the feature has a total of eight channels (eight dimensions) from 0th to 7th. Ｃ has a total of three channels: R, G, and B. The number of dimensions of the feature and the current I Ｃ The number of dimensions of the feature may be 9 or more. Ｃ may be a YUV image.

[0082] FIG. 15 is an explanatory diagram of the learning method of the DNN 30.

[0083] First, the correction history I, which has not undergone ghost removal processing, Ｈ ＭＣThe DNN 35 is trained using the correction history I, which includes ghosts (Step I). Ｈ ＭＣ This is the same as conventional learning. Ｈ ＭＣ and Current I Ｃ is input, and the output image I Ｏ is obtained. The output image I Ｏ and teacher image I Ｔ The DNN 35 is optimized so that the difference between

[0084] Next, using the DNN 35 learned in step I, one or more current I Ｃ This process infers the current I Ｃ For each intermediate feature I Ｈｉ is obtained. Current I Ｃ And this Current I Ｃ Intermediate feature I inferred from Ｈｉ A pair of data is generated by linking the current I Ｃ From the intermediate feature I Ｈｉ Approximate feature I Ｈａ The one or more calculated conversion model CVs are statistically processed to generate one conversion model CV (integrated model) (Step II).

[0085] Next, we use the integrated model to calculate the current Ｃ From the approximate feature I Ｈａ is obtained. The approximate feature I Ｈａ Correction History I Ｈ ＭＣ Correct the occlusion area and correct history I Ｈｒ Corrected History I Ｈｒ The DNN 35 is retrained using the above to obtain the final DNN 30 (Step III).

[0086] 3-3. Processing Flow FIGS. 16 and 17 are diagrams showing an example of a processing flow for performing inference processing.

[0087] The input unit 10 receives the current I Ｃ, depth image DP and motion vectors MV. The scaler 20 scales the current I Ｃ , the number of pixels of the depth image DP and the motion vector MV is enlarged, and the current I Ｃ ', a depth image DP' and a motion vector MV' are obtained (step S1).

[0088] The motion compensation unit 40 calculates the motion vector MV' based on the history I, which is the DNN output from the previous frame. Ｈ Motion compensation is performed on the Ｈ ＭＣ (Step S2). The occlusion region detection unit 50 obtains an occlusion map OC based on the depth image DP' and the motion vector MV' (Step S3).

[0089] The ghost removal unit 60 determines whether to replace the occlusion region with blank data (step S4). The type of data to use as history and the type of data to replace the occlusion region are determined by factors such as the video scene. For example, in a dark and monotonous video scene, a somewhat lower inference quality is not a problem. In such a case, the occlusion region is replaced with blank data without using approximate features that require a large amount of calculation.

[0090] If replacement with blank data is performed (step S4: Yes), the ghost removal unit 60 Ｈ ＭＣ The ghost removal unit 60 replaces the values ​​of the occlusion area of ​​the image with blank data to remove ghosts (step S5). Ｈ ＭＣ Correction History I Ｈｒ (step S6).

[0091] The scaler 20 and the ghost canceller 60 are connected to the current I Ｃ ' and Correction History I Ｈｒ is input to the DNN 30 (step S7). The DNN 30 outputs the final output image I Ｏ and history I for the next image processing Ｈ(Output image I Ｏ or intermediate features) is output (step S8). Thereafter, the recursive processing of the RNN is repeated.

[0092] If the replacement with blank data is not performed (step S4: No), the ghost removal unit 60 Ｃ ' and History I Ｈ It is determined whether the data formats are the same (step S9).

[0093] If the data formats are the same (step S9: Yes), the ghost removal unit 60 uses the correction history I Ｈ ＭＣ The occlusion area value of the current Ｃ If the data formats are different (step S9: No), the ghost removal unit 60 replaces the pixel values ​​of the correction history I with those of the correction history I. Ｈ ＭＣ The value of the occlusion region is calculated by the approximate feature I obtained from the current Ｈａ (Step S10), and then proceed to Step S6.

[0094] [3-4. Effects] The motion compensation unit 40 uses the intermediate feature I of the past frame inferred by the DNN 30. Ｈｉ Correction history I is obtained by performing motion compensation on Ｈ ＭＣ According to this configuration, the inference accuracy of the DNN 30 is improved compared to when image data is used as input.

[0095] As a ghost removal process, the ghost removal unit 60 calculates the data of the occlusion region as an approximate feature I Ｈａ According to this configuration, the current I Ｃ ' intermediate feature I Ｈｉ and Correction History I Ｈ ＭＣ This increases the similarity between the frames, which increases the continuity of the image between frames.

[0096] The approximate feature estimation unit 80 applies the current I to the optimized conversion model CV. ＣThe intermediate feature obtained by applying ' is the approximate feature I Ｈａ According to this configuration, the current I Ｃ ' intermediate feature I Ｈｉ and Correction History I Ｈ ＭＣ The similarity can be maximized.

[0097] 4. Example of Hardware Configuration FIG. 18 is a diagram showing an example of the hardware configuration of the information processing devices 1 and 2. As shown in FIG.

[0098] The information processing of the information processing device is realized by, for example, a computer 1000. The computer 1000 has a CPU (Central Processing Unit) 1100, a RAM (Random Access Memory) 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0099] The CPU 1100 operates and controls each component based on a program (program data 1450) stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0100] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000 .

[0101] The HDD 1400 is a non-transitory computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records an information processing program according to an embodiment as an example of program data 1450.

[0102] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0103] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display device, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0104] For example, when the computer 1000 functions as an information processing device according to an embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize the functions of the aforementioned components. The information processing program, various models, and various data according to the present disclosure are stored in the HDD 1400. The CPU 1100 reads and executes program data 1450 from the HDD 1400. Alternatively, the CPU 1100 may acquire these programs from another device via an external network 1550.

[0105] [Additional Notes] The present technology may also be configured as follows: (1) An information processing device comprising: a motion compensation unit that performs motion compensation on an inference result of a past frame to generate a corrected history; an occlusion region detection unit that detects an occlusion region in the corrected history where a ghost may occur based on a motion vector indicating the movement of a subject between frames; a ghost removal unit that performs a ghost removal process on the occlusion region; and a DNN that performs inference of the current frame based on the corrected history on which the ghost removal process has been performed and a current image of the current frame to be processed. (2) The information processing device according to claim 1, wherein the motion compensation unit performs the motion compensation on the inference image of the past frame inferred by the DNN to generate the corrected history. (3) The information processing device according to claim 2, wherein the ghost removal unit performs the ghost removal process by replacing pixel values ​​of each pixel in the occlusion region with blank data. (4) The information processing device according to claim 2, wherein the ghost removal unit performs the ghost removal process by replacing pixel values ​​of each pixel in the occlusion region with pixel values ​​of the current. (5) The information processing device according to claim 1, wherein the motion compensation unit generates the corrected history by performing the motion compensation on intermediate features of the past frame inferred by the DNN. (6) The information processing device according to claim 5, further comprising an approximate feature estimation unit that acquires features corresponding to the corrected history from the current as approximate features, and the ghost removal unit performs the ghost removal process by replacing data of the occlusion region with data of the approximate features. (7) The information processing device according to claim 6, wherein the approximate feature estimation unit optimizes the conversion model so that intermediate features obtained by applying a conversion model to the current approximate intermediate features obtained by inputting the current to the DNN, and calculates the intermediate features obtained by applying the current to the optimized conversion model as the approximate features.(8) The information processing device according to any one of claims 1 to 7, wherein the occlusion region detection unit determines, based on the motion vector, whether or not predetermined conditions are satisfied, that a magnitude of positional deviation between the foreground and the background is greater than a threshold representing noise and that a direction of movement of the foreground and a direction of movement of the background are different, and determines that a region of the background that overlapped with the foreground is the occlusion region based on determination that the predetermined conditions are satisfied. (9) The information processing device according to claim 8, wherein the occlusion region detection unit distinguishes the foreground and the background based on depth information. (10) An information processing method executed by a computer, comprising: generating a correction history by performing motion compensation on an inference result of a past frame; detecting an occlusion region in the correction history where a ghost may occur based on a motion vector indicating movement of a subject between frames; performing ghost removal processing on the occlusion region; and inferring the current frame based on the correction history on which the ghost removal processing has been performed and a current that is an image of the current frame to be processed. (11) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: generate a correction history by performing motion compensation on the inference results of past frames; detect an occlusion area in the correction history where ghosts may occur based on a motion vector that indicates the movement of the subject between frames; perform ghost removal processing on the occlusion area; and infer the current frame based on the correction history after the ghost removal processing and Current, which is the image of the current frame to be processed.

[0106] 1, 2 Information processing device 30 DNN 40 Motion compensation unit 50 Occlusion area detection unit 60 Ghost removal unit 80 Approximate feature estimation unit I Ｃ , I Ｃ ' Current I Ｈａ Approximate feature I Ｈｉ Intermediate feature I Ｈ ＭＣ Correction history MV, MV' motion vector

Claims

1. An information processing device having: a motion compensation unit that performs motion compensation on the inference results of past frames to generate a correction history; an occlusion area detection unit that detects occlusion areas in the correction history where ghosts may occur based on motion vectors that indicate the movement of a subject between frames; a ghost removal unit that performs ghost removal processing on the occlusion areas; and a DNN that infers the current frame based on the correction history after the ghost removal processing has been performed and a current, which is an image of the current frame to be processed.

2. The information processing device according to claim 1, wherein the motion compensation unit performs the motion compensation on an inferred image of the past frame inferred by the DNN to generate the correction history.

3. The information processing device according to claim 2, wherein the ghost removal section performs the ghost removal process by replacing pixel values ​​of each pixel in the occlusion region with blank data.

4. The information processing device according to claim 2, wherein the ghost removal section performs the ghost removal process by replacing the pixel value of each pixel in the occlusion region with the current pixel value.

5. The information processing device according to claim 1, wherein the motion compensation unit performs the motion compensation on intermediate features of the past frames inferred by the DNN to generate the corrected history.

6. An information processing device as described in claim 5, further comprising an approximate feature estimation unit that acquires a feature corresponding to the correction history from the current as an approximate feature, and the ghost removal unit performs a process of replacing data of the occlusion region with data of the approximate feature as the ghost removal process.

7. The information processing device according to claim 6, wherein the approximate feature estimation unit optimizes the conversion model so that an intermediate feature obtained by applying the conversion model to the current approximates an intermediate feature obtained by inputting the current to the DNN, and calculates the intermediate feature obtained by applying the current to the optimized conversion model as the approximate feature.

8. The information processing device of claim 1, wherein the occlusion area detection unit determines, based on the motion vector, whether or not predetermined conditions are satisfied, that the magnitude of the positional shift between the foreground and the background is greater than a threshold representing noise, and that the direction of movement of the foreground and the direction of movement of the background are different, and determines that the area of ​​the background that overlapped with the foreground is the occlusion area, based on a determination that the predetermined conditions are satisfied.

9. The information processing device according to claim 8, wherein the occlusion region detection unit distinguishes between the foreground and the background based on depth information.

10. An information processing method executed by a computer, comprising: performing motion compensation on the inference results of past frames to generate a correction history; detecting an occlusion area in the correction history where ghosts may occur based on a motion vector indicating the movement of a subject between frames; performing ghost removal processing on the occlusion area; and inferring the current frame based on the correction history after the ghost removal processing and Current, which is an image of the current frame to be processed.

11. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: generate a correction history by performing motion compensation on the inference results of past frames; detect occlusion areas in the correction history where ghosts may occur based on motion vectors that indicate the movement of the subject between frames; perform ghost removal processing on the occlusion areas; and infer the current frame based on the correction history after the ghost removal processing has been performed and Current, which is an image of the current frame to be processed.

Images