Information processing device, information processing method, and computer-readable non-transitory storage medium

Abstract

This information processing device is equipped with a DNN that subjects a current frame to inference on the basis of a blended result obtained by blending a current that is information about a current frame to be processed and a history that is recursive information including an inference result of a past frame. The DNN includes a generation unit, a correction unit, and an integration unit. The generation unit generates a minute variation component that acts to reduce the reliability of the history. The correction unit corrects the history by adding the minute variation component to the history. The integration unit integrates the current and the history so that the contribution ratio of whichever of the current and the history is more reliable is higher, and outputs the integrated result as the blended result.

Description

Information processing device, information processing method, and computer-readable non-transitory storage medium

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable non-transitory storage medium.

[0002] Deep neural network (DNN) processing for video generally uses a recurrent neural network (RNN) structure with high time stability. This is because the RNN structure can correlate the current (information on the current frame) with the history (recursive information on past frames), making it easier to maintain consistency in the output frame images.

[0003] Known systems for performing such DNN processing for moving images include a learning-type MFSR (Multi Frame Super Resolution) system that blends current and history data and uses the blending results to super-resolution the current frame.

[0004] This MFSR system can achieve more precise super-resolution results by utilizing extensive history, but on the other hand, the recursive accumulation of error components can lead to artifacts when the process is repeated over a long period of time.

[0005] To address this issue, a technique has been proposed in which the strength of the history is reduced before blending, thereby relatively reducing the contribution of the history to the blending and suppressing artifacts (see, for example, Non-Patent Document 1).

[0006] Benjamin Naoto Chiche, Arnaud Woiselle, Joana Frontera-Pons, Jean-Luc Starck, "Stable Long-Term Recurrent Video Super-Resolution", [online], [Retrieved June 22, 2023], Internet <URL: https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Chiche_Stable_Long-Term_Recurrent_Video_Super-Resolution_CVPR_2022_paper.pdf>

[0007] However, although the conventional technology described above reduces the relative contribution rate of the history in blending by reducing the history intensity, the DNN processing using the blending results performs high-order feature matching that ignores the reduced history intensity. Therefore, when using the conventional technology described above, the effect of history correction is weakened, making it difficult to suppress artifacts as intended.

[0008] Therefore, the present disclosure proposes an information processing device, an information processing method, and a computer-readable non-transitory storage medium that are capable of more effectively suppressing artifacts in DNN processing for moving images.

[0009] In order to solve the above-described problems, an information processing device according to one embodiment of the present disclosure is an information processing device including a DNN that performs inference of a current frame based on a blending result obtained by blending a current, which is information related to a current frame to be processed, and a history, which is recursive information including inference results of past frames, and the DNN includes a generation unit, a correction unit, and an integration unit. The generation unit generates a minute fluctuation component that acts to reduce the reliability of the history. The correction unit corrects the history by adding the minute fluctuation component to the history. The integration unit integrates the current and the history so that the one with a higher reliability has a higher contribution rate, and outputs the result as the blending result.

[0010] FIG. 1 is a diagram illustrating a basic configuration example of an information processing device according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating a configuration example of a blending unit according to a conventional technique. FIG. 3 is a diagram illustrating a configuration example of a blending unit according to the first embodiment of the present disclosure. FIG. 4 is a flowchart illustrating a processing procedure executed by an information processing device according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating a configuration example of a blending unit according to a second embodiment of the present disclosure. FIG. 6 is a flowchart illustrating a processing procedure executed by an information processing device according to the second embodiment of the present disclosure. FIG. 7 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0012] In the following, it is assumed that an information processing device according to an embodiment of the present disclosure executes MFSR processing as DNN processing for video. An information processing device 1 according to a first embodiment will be described using FIGS. 1 to 4. An information processing device 1A according to a second embodiment will be described using FIGS. 5 and 6. In the following, it is assumed that an information processing method according to an embodiment of the present disclosure is an information processing method executed by the information processing devices 1 and 1A.

[0013] The present disclosure will be described in the following order: 1. First embodiment 1-1. Example of basic configuration of information processing device according to first embodiment 1-2. Example of configuration of blending unit according to conventional technology 1-3. Example of configuration of blending unit according to first embodiment 1-4. Processing procedure of information processing device according to first embodiment 2. Second embodiment 2-1. Example of configuration of blending unit according to second embodiment 2-2. Processing procedure of information processing device according to second embodiment 3. Modified example 4. Hardware configuration 5. Conclusion

[0014] <<1. First embodiment>> <1-1. Basic configuration example of information processing device according to first embodiment> Fig. 1 is a diagram showing a basic configuration example of an information processing device 1 according to a first embodiment of the present disclosure. As shown in Fig. 1, the information processing device 1 has an input unit 10, a scaler 20, a DNN 30, and an output unit 40. Also, as shown in Fig. 1, the information processing device 1 has an RNN structure.

[0015] DNN processing for video, such as MFSR processing, generally uses an RNN structure with high temporal stability. The RNN structure can correlate the current Cur and history His, making it easier to maintain the consistency of the image in the output frame. While the DNN 30 has very high inference performance (e.g., the super-resolution effect in the case of MFSR processing), it is affected by even small changes in the input data and the output results change. Therefore, DNN processing for video is more prone to flickering than simple filter processing. Therefore, in order to increase temporal stability and suppress flickering, the information processing device 1 that executes DNN processing for video is configured to have an RNN structure that simultaneously inputs the current Cur and history His and enables learning including temporal correlation (similarity between the current Cur and history His).

[0016] Note that current Cur is information about the frame to be processed (current frame). Specifically, it is information indicating the Tth frame to be processed among N frames of a moving image. In this embodiment, the information format is assumed to be a general video format for moving images. More specifically, it is assumed to be three-channel image data such as YUV, which is a video transmission format in accordance with the ITU (International Telecommunication Union) standard.

[0017] Furthermore, the history His is recursive information including the super-resolution results obtained by inputting images of past frames into the DNN 30. A past frame refers to a frame that is one or more frames before the current frame. In this embodiment, the past frame is the T-1th frame of a video sequence that has N frames, with respect to the Tth frame to be processed.

[0018] Therefore, in this embodiment, the history His is assumed to be information obtained by recursively capturing the super-resolution image of the T-1th frame, as in general feedforward video processing. The format of the information is assumed to be a general video format (e.g., three-channel image data such as the above-mentioned YUV) as in the case of the current Cur.

[0019] The history His may be represented by a feature map extracted from information on past frames. More specifically, it may be a feature map of the F (>3) channel, which has an intensity for each pixel. In this case, the feature map is assumed to be information converted by a convolution operation from the super-resolved image of the T-1th frame.

[0020] Furthermore, the history His may be information that has been aligned with the current frame. In this case, the alignment is assumed to be general motion compensation. More specifically, it is assumed to be a process of converting position information of corresponding pixels using a motion vector that describes the amount of movement of corresponding pixels from a past frame to the current frame.

[0021] The input unit 10 acquires the current current image. The current current image may be an image captured by a camera or a CG (Computer Graphics) image generated by a CG renderer, etc. The scaler 20 adjusts the number of pixels of the current current image in accordance with the specifications of the DNN 30.

[0022] The DNN 30 is a deep learning model that learns and infers super-resolution of the current frame. The DNN 30 is preferably realized as a deep learning model that is trained using a super-resolved image that is the output when the current Cur and history His are input, and high-resolution training data that is paired with the super-resolved image, so as to minimize the loss between them by a steepest descent method or the like.

[0023] The DNN 30 inputs and blends the current Cur and the history His, and outputs the super-resolution result of the current frame using the blending result, while updating the history His.

[0024] 1, the DNN 30 includes a blending unit 31 and an inference unit 32. The blending unit 31 uses the current Cur and the history His to output a blending result of the two. The inference unit 32 uses the blending result to output an inference result for the current frame, that is, a super-resolution result in this embodiment, and also uses this super-resolution result to update the history His.

[0025] In this embodiment, the blending unit 31 and the inference unit 32 are assumed to be models contained in an integrated DNN 30 as shown in Figure 1, but the blending unit 31 and the inference unit 32 may each be configured by an independent DNN.

[0026] In this embodiment, the blending unit 31 is assumed to be a multi-layer convolutional neural network (CNN) that outputs a feature map. The blending unit 31 receives the current Cur and the history His, blends them, and outputs a feature map as a blending result.

[0027] In this embodiment, the inference unit 32 is assumed to be a multi-layer CNN connected after the blending unit 31. The inference unit 32 inputs the feature map output by the blending unit 31 and outputs a super-resolved image as the super-resolution result, and also accumulates the final-stage feature map as recursive information and outputs it as the history His of the next frame.

[0028] The output unit 40 applies general post-processing such as color conversion and codec to the super-resolved image acquired from the DNN 30 and outputs the image.

[0029] 1-2. Example of the configuration of a blending unit according to the prior art In an MFSR system, by making extensive use of history His, it is possible to achieve more precise super-resolution results. However, on the other hand, there is a problem in that artifacts occur when processing is repeated over a long period of time due to the recursive accumulation of error components.

[0030] To address this problem, a conventional technique has been proposed in which the strength of the history His is reduced before blending, thereby relatively reducing the contribution of the history to the blending and suppressing artifacts.

[0031] 2 is a diagram showing an example of the configuration of a blending unit 31' according to the prior art. The blending unit 31' includes an integrating unit 31a. As shown in FIG. 2, the blending unit 31' multiplies the history His by a constant to reduce the strength of the history His, thereby generating a modified history His. Ｍ The integration unit 31a calculates the current Cur and the modified history His Ｍ and are integrated and output as the blending result BM.

[0032] However, in the case of a complex model such as a DNN, inference using the blending result BM involves high-order feature matching that ignores the decrease in the intensity of the history His. Therefore, when the blending unit 31′ according to the conventional technology is used, the effect of modifying the history His is weakened, resulting in a problem that it is difficult to suppress artifacts as intended.

[0033] <1-3. Example of the configuration of the blending unit according to the first embodiment> In the information processing method according to the first embodiment, the blending unit 31 generates a small fluctuation component that has a relatively low correlation with the history His, and adds the generated small fluctuation component to the history His to generate a corrected history His. Ｍ The "small fluctuation component" is a component that acts to reduce the reliability of the history His. It may also be called a "perturbation component."

[0034] 3 is a diagram illustrating an example configuration of the blending unit 31 according to the first embodiment of the present disclosure. As illustrated in FIG. 3, the blending unit 31 includes an integration unit 31 a, a generation unit 31 b, and a correction unit 31 c.

[0035] The generator 31b generates a minute fluctuation component PM that has a relatively low degree of correlation with the history His. "Relatively low degree of correlation" includes the case where there is no correlation. For example, the generator 31b generates a minute fluctuation component PM that is not correlated with the history His.

[0036] The minute fluctuation component PM generated at this time may be any information as long as it is information that is not correlated with the history His. In this embodiment, the generator 31b generates a noise image with the same number of channels as the number of channels of the history His, for example.

[0037] The noise generation method can be a noise generation method used in general image processing. For example, the generation unit 31b generates a noise image such as Gaussian noise. The generation unit 31b then scales the generated noise image by a preset coefficient and outputs it as the minute fluctuation component PM.

[0038] The correction unit 31c uses the minute fluctuation component PM generated by the generation unit 31b to derive a corrected history His from the history His. Ｍ Calculate the corrected history His Ｍ is calculated by adding the minute fluctuation component PM to the history His.

[0039] The integration unit 31a integrates the current Cur and the modified history His Ｍ The integrating unit 31a integrates the current Cur and the modified history His and outputs the result of blending BM. Ｍ It is desirable to realize the integration by linear addition of the information.

[0040] In this case, the linear combination may be any operation, for example, it may be performed by an α blending method using a general single blending coefficient. In this embodiment, since the DNN 30 that learns and infers the super-resolution of the current frame is assumed, the current Cur and the modified history His Ｍ It is assumed that the linear combination parameters are automatically acquired so that an output image with higher resolution is output when the feature maps of

[0041] At this time, the integration unit 31a combines the current Cur and the modified history His Ｍ Compare with the modified history His Ｍ When the reliability of is high, the modified history His ＭBy increasing the weight of the modified history His, the resolution of the output image can be increased. Ｍ If the reliability of is low, the integration unit 31a uses the modified history His Ｍ By reducing the weight of , it is possible to suppress artifacts in long-term processing.

[0042] <1-4. Processing Procedure of Information Processing Device According to First Embodiment> Next, a processing procedure executed by the information processing device 1 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the processing procedure executed by the information processing device 1 according to the first embodiment of the present disclosure. Note that Fig. 4 mainly shows the processing procedure in the RNN structure described above.

[0043] 4, the blending unit 31 acquires the current Cur and the history His (step S101). Then, the generating unit 31b generates a minute fluctuation component PM having a relatively low correlation with the history His (step S102).

[0044] The correcting unit 31c corrects the history His using the minute fluctuation component PM generated by the generating unit 31b (step S103). Ｍ and output as a blending result BM (step S104).

[0045] The inference unit 32 then performs super-resolution on the current frame using the blending result BM acquired from the blending unit 31, and outputs the super-resolution result (step S105). The inference unit 32 also updates the history His based on the super-resolution result (step S106). Thereafter, the recursive processing of the RNN is repeated.

[0046] As described above, according to the information processing device 1 according to the first embodiment, it is possible to more effectively suppress artifacts in the MFSR process.

[0047] <<2. Second embodiment>> <2-1. Configuration example of blending unit according to second embodiment> Fig. 5 is a diagram showing a configuration example of a blending unit 31A according to a second embodiment of the present disclosure. Note that the basic configuration example of the information processing device 1A according to the second embodiment is similar to the configuration example of the information processing device 1 shown in Fig. 1, and therefore description thereof will be omitted here. Furthermore, since Fig. 5 corresponds to Fig. 3, the description using Fig. 5 will mainly focus on differences from Fig. 3.

[0048] As shown in FIG. 5, the blending unit 31A differs from the blending unit 31 shown in FIG. 3 in that the generating unit 31b generates a minute fluctuation component FFM that is correlated with the current Cur.

[0049] 5, the blending unit 31A differs from the blending unit 31 shown in FIG. 3 in that it further includes a map calculation unit 31d. The map calculation unit 31d generates a correction amount map MM using auxiliary information. The auxiliary information includes the amount of motion, user input, and an occlusion map. The correction amount map MM is information indicating the amount of correction for each pixel in the output image as an inference result of past frames included in the history His.

[0050] A more specific explanation will be given below. The generation unit 31b of the blending unit 31A uses the current Cur to generate a minute fluctuation component correlated with the current Cur. The minute fluctuation component generated at this time needs to be information converted using the current Cur. For example, when the current Cur is an image consisting of the above-mentioned three YUV channels and the history His is a feature map consisting of N channels, the generation unit 31b of the blending unit 31A calculates the minute fluctuation component FFM (False FM) using the following formula (a):

[0051]

[0052] Here, the coefficient S is a scaling coefficient, and is desirably set to a value equal to or less than 0.1. The generation unit 31b of the blending unit 31A calculates the minute fluctuation component FFM by multiplying the input YUV by an N-row, 4-column matrix M and the coefficient S.

[0053] In the blending unit 31 according to the first embodiment, adding minute fluctuation components PM that are not correlated with the history His has the effect of relatively reducing the contribution rate of recursive information in the integrating unit 31 a. However, if the minute fluctuation components PM are generated completely randomly, the recursive information cannot be correctly accumulated, which may result in a decrease in the sense of detail of the final super-resolution result.

[0054] In contrast to this, as in the blending unit 31A according to this embodiment, by using the current Cur to generate and add the minute fluctuation component FFM correlated with the current Cur, and by reducing the reliability of the history His, the corrected history His in the integrating unit 31a can be obtained without impairing the normal accumulation of recursive information. Ｍ This also has the effect of making it possible to maintain a high level of detail in the super-resolution result.

[0055] The map calculation unit 31d calculates and outputs a correction amount map MM using the auxiliary information. The correction amount map MM is information calculated in order for the subsequent correction unit 31c to correct the history His, and is one-channel information having the same number of pixels as the history His.

[0056] Various types of auxiliary information may be used to calculate the correction amount map MM. In this embodiment, a first correction amount map (hereinafter referred to as "correction amount map MM-1" as appropriate) is calculated using, for example, a motion amount as auxiliary information. The motion amount is the absolute value of a motion vector that describes the amount of movement of a corresponding pixel from a past frame to a current frame in a moving image to be processed. In this embodiment, the map value is set so that the smaller the motion amount of a target pixel, the larger the correction amount of the corresponding pixel in the correction amount map MM-1.

[0057] It has been confirmed that the artifact problem in long-term processing, which is a subject of the present disclosure, occurs due to the accumulation of error components in the history His in the time direction. The error components accumulated here become larger the smaller the amount of motion of the target pixel, i.e., the longer the pixel remains stationary. Therefore, by increasing the amount of correction for the corresponding pixel the smaller the amount of motion of the target pixel, as in this embodiment, the degree to which the small fluctuation component FFM is added in the subsequent correction unit 31c is increased, which has the effect of suppressing the accumulation of error components in the time direction.

[0058] Furthermore, in this embodiment, a second correction amount map (hereinafter referred to as "correction amount map MM-2" as appropriate) is calculated using, for example, user input as auxiliary information. The user input is an input by the user of the information processing device 1A to specify a correction amount for a desired pixel position. At this time, the map value is set so that the larger the input value specified by the user input, the larger the correction amount of the corresponding pixel in the correction amount map MM-2. This has the effect of allowing the correction amount of the history His to be adjusted according to the user's intention.

[0059] Furthermore, in this embodiment, a third correction amount map (hereinafter referred to as "correction amount map MM-3" as appropriate) is calculated using, for example, an occlusion map as auxiliary information. An occlusion is an area that occurs at the boundary between a foreground object and a background object in a moving image. In this embodiment, the map value is set so that the more likely a pixel area is to be occluded, the greater the correction amount for the corresponding pixel area in the correction amount map MM-3.

[0060] Generally, when occlusion occurs, the history His for that region in the past frame becomes indefinite, resulting in the problem of so-called ghosts. Therefore, by increasing the amount of correction for pixel regions that are more likely to be occluded, as in this embodiment, the contribution rate of the history His for that region is reduced in the subsequent correction unit 31c, thereby suppressing the occurrence of ghosts.

[0061] The map calculation unit 31d uses one or more of the above-mentioned correction amount maps MM-1, MM-2, and MM-3 to output the final correction amount map MM. At this time, it is desirable to select a representative value such as the average value or maximum value of the correction amount maps MM-1, MM-2, and MM-3 as each map value of the final correction amount map MM. In this embodiment, it is assumed that the maximum value is selected.

[0062] In this embodiment, the correction unit 31c calculates the corrected history His by adding the minute fluctuation component FFM based on the correction amount map MM. Ｍ Specifically, the correction unit 31c calculates and outputs the formula "corrected history His Ｍ = Correction amount map MM × minute fluctuation component FFM + (1.0 − correction amount map MM) × history His Ｍ Calculate.

[0063] This has the effect of locally controlling the contribution rate of history His to be stronger or weaker in the subsequent integration section 31a by adding more strongly the minute fluctuation component FFM that is not correlated with history His to the area specified in the correction amount map MM.

[0064] <2-2. Processing Procedure of Information Processing Device According to Second Embodiment> Next, a processing procedure executed by the information processing device 1A according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing procedure executed by the information processing device 1A according to the second embodiment of the present disclosure. Note that Fig. 6 mainly shows the processing procedure in the RNN structure described above.

[0065] 6, the blending unit 31A acquires the current Cur, the history His, and the auxiliary information (step S201). Then, the generating unit 31b generates the minute fluctuation component FFM correlated with the current Cur using the current Cur (step S202).

[0066] The map calculation unit 31d calculates a correction amount map MM using the auxiliary information (step S203), and the correction unit 31c corrects the history His using the minute fluctuation component FFM and the correction amount map MM (step S204).

[0067] Then, the integration unit 31a combines the current Cur and the modified history His Ｍ and output as a blending result BM (step S205).

[0068] The inference unit 32 then performs super-resolution on the current frame using the blending result BM acquired from the blending unit 31A and outputs the super-resolution result (step S206). The inference unit 32 also updates the history His based on the super-resolution result (step S207). Thereafter, the recursive processing of the RNN is repeated.

[0069] As described above, the blending unit 31A according to the second embodiment generates the minute fluctuation component FFM using the current Cur, while selectively lowering the contribution rate of the history His only in areas where artifacts are likely to occur, areas intended by the user, and areas where the history His is uncertain, using auxiliary information, thereby making it possible to selectively suppress artifacts while maintaining a high level of detail in the final super-resolution result.

[0070] That is, according to the information processing apparatus 1A according to the second embodiment, artifacts can be more effectively suppressed in the MFSR process.

[0071] <<3. Modifications>> Of the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the process procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0072] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0073] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0074] Furthermore, in the above-described embodiment of the present disclosure, an example was given in which the DNN processing for moving images is MFSR processing, but the DNN processing for moving images is not limited to MFSR processing, and may be, for example, colorization processing, restoration processing, blur removal processing, etc.

[0075] <<4. Hardware Configuration>> The information processing devices 1 and 1A according to the embodiments of the present disclosure described above are realized by, for example, a computer 1000 configured as shown in Fig. 7. Fig. 7 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing devices 1 and 1A. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0076] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0077] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0078] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the program according to this embodiment, which is an example of the program data 1450.

[0079] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0080] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0081] For example, when the computer 1000 functions as the information processing device 1, 1A, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the above-mentioned components. The secondary storage device 1400 stores the program according to this embodiment, various models including the DNN 30, and various data. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0082] <<5. Conclusion>> As described above, according to one embodiment of the present disclosure, the information processing device 1, 1A is an information processing device including a DNN 30 that performs inference of a current frame based on a blending result BM obtained by blending a current Cur, which is information related to the current frame to be processed, and a history His, which is recursive information including inference results of past frames. The DNN 30 includes a generator 31b, a corrector 31c, and an integrator 31a. The generator 31b generates minute fluctuation components PM and FFM that act to reduce the reliability of the history His. The corrector 31c corrects the history His by adding the minute fluctuation components PM and FFM to the history His. The integrator 31a integrates the current Cur and the history His so that the more reliable of the current Cur and the history His has a higher contribution rate, and outputs the result as the blending result BM. This allows artifacts to be more effectively suppressed in DNN processing for moving images.

[0083] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0084] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0085] Note that the present technology can also be configured as follows: (1) An information processing device including a DNN that performs inference on a current frame based on a blending result obtained by blending a current, which is information related to the current frame to be processed, and a history, which is recursive information including inference results on past frames, wherein the DNN has: a generation unit that generates a minute fluctuation component that acts to reduce the reliability of the history, a correction unit that corrects the history by adding the minute fluctuation component to the history, and an integration unit that integrates the current and the history so that the contribution rate of the current or the history with a higher reliability is higher, and outputs the result as the blending result. (2) The information processing device according to (1), wherein the generation unit uses the current to generate the minute fluctuation component correlated with the current. (3) The information processing device according to (1) or (2), further comprising a map calculation unit that calculates a correction amount map indicating a correction amount for each pixel in an output image as an inference result for the past frame included in the history, wherein the correction unit adds the small fluctuation component to the history based on the correction amount map. (4) The information processing device according to (3), wherein the map calculation unit calculates the correction amount map based on auxiliary information, wherein the auxiliary information includes at least one of an amount of motion for each pixel from the past frame, a user input, and an occlusion map indicating a possibility of occlusion. (5) The information processing device according to (4), wherein the map calculation unit calculates the correction amount map based on the amount of motion such that the correction amount increases as the amount of motion decreases. (6) The information processing device according to (4) or (5), wherein the map calculation unit calculates the correction amount map based on the user input such that the correction amount increases as the input value specified by the user input increases. (7) The information processing device according to (4), (5), or (6), wherein the map calculation unit calculates the correction amount map based on the occlusion map such that the correction amount increases in a pixel region that is more likely to be an occlusion.(8) The information processing device according to (1), wherein the generation unit generates the minute fluctuation component having a relatively low correlation with the history. (9) The information processing device according to (8), wherein the generation unit generates a noise image as the minute fluctuation component. (10) An information processing method executed by a computer, comprising: performing inference on the current frame based on a blending result obtained by blending a current, which is information about the current frame to be processed, and a history, which is recursive information including inference results of past frames; generating a minute fluctuation component that acts to reduce the reliability of the history; correcting the history by adding the minute fluctuation component to the history; and integrating the current and the history so that the contribution rate of the current or the history with a higher reliability is higher, and outputting the result as the blending result. (11) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following: inferring the current frame based on a blending result obtained by blending a current, which is information about the current frame to be processed, and a history, which is recursive information including inference results of past frames; generating a minute fluctuation component that acts to reduce the reliability of the history; modifying the history by adding the minute fluctuation component to the history; and integrating the current and the history so that the contribution rate of the current or the history with a higher reliability is higher, and outputting the result of the blending.

[0086] 1, 1A Information processing device 10 Input unit 20 Scaler 30 DNN 31, 31A Blending unit 31a Integration unit 31b Generation unit 31c Correction unit 31d Map calculation unit 32 Inference unit 40 Output unit BM Blending result Cur Current FFM Microfluctuation component His History His Ｍ Correction history MM Correction amount map PM Minute fluctuation component

Claims

1. An information processing device having a DNN that infers a current frame based on a blended result of blending a current, which is information related to the current frame to be processed, and a history, which is recursive information including inference results of past frames, wherein the DNN has: a generation unit that generates a micro-fluctuation component that acts to reduce the reliability of the history; a modification unit that modifies the history by adding the micro-fluctuation component to the history; and an integration unit that integrates the current and the history so that the contribution rate of the one with a higher reliability is higher, and outputs the result as the blended result.

2. The information processing device according to claim 1, wherein the generating unit generates the minute fluctuation component correlated with the current by using the current.

3. The information processing device according to claim 1, further comprising a map calculation unit that calculates a correction amount map indicating the amount of correction for each pixel in an output image as an inference result of the past frame included in the history, wherein the correction unit adds the small fluctuation component to the history based on the correction amount map.

4. The information processing device according to claim 3, wherein the map calculation unit calculates the correction amount map based on auxiliary information, and the auxiliary information includes at least one of an amount of motion for each pixel from the previous frame, a user input, and an occlusion map indicating the possibility of occlusion.

5. The information processing device according to claim 4, wherein the map calculation section calculates the correction amount map based on the amount of motion such that the correction amount increases as the amount of motion decreases.

6. The information processing device according to claim 4, wherein the map calculation section calculates the correction amount map based on the user input such that the correction amount increases as the input value specified by the user input increases.

7. The information processing device according to claim 4, wherein said map calculation section calculates said correction amount map based on said occlusion map such that the correction amount increases for pixel regions that are more likely to be occluded.

8. The information processing device according to claim 1, wherein the generating section generates the minute fluctuation component having a relatively low correlation with the history.

9. The information processing device according to claim 8, wherein the generating unit generates a noise image as the minute fluctuation component.

10. An information processing method executed by a computer, comprising: inferring a current frame based on a blending result obtained by blending a current, which is information related to the current frame to be processed, and a history, which is recursive information including inference results of past frames; generating a micro-fluctuation component that acts to reduce the reliability of the history; modifying the history by adding the micro-fluctuation component to the history; and integrating the current and the history so that the contribution rate of the one with a higher reliability is higher, and outputting the result as the blending result.

11. A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following steps: inferring a current frame based on a blended result obtained by blending a current, which is information related to the current frame to be processed, and a history, which is recursive information including inference results of past frames; generating a micro-fluctuation component that acts to reduce the reliability of the history; modifying the history by adding the micro-fluctuation component to the history; and integrating the current and the history so that the contribution rate of the one with a higher reliability is higher, and outputting the result as the blended result.

Images