Information processing apparatus, information processing method, and computer-readable non-transitory storage medium

Abstract

An information processing device is equipped with a DNN that reasones a current frame on the basis of a mixing result obtained by mixing a current which is information relating to a current frame to be processed and a history which is recursive information including reasoning results of past frames. The DNN includes a generation unit, a correction unit, and an integration unit. The generation unit generates a minute change component for reducing the reliability of the history. The correction unit corrects the history by adding the minute change component to the history. The integration unit integrates the current and the history so that the contribution rate of a more reliable one of the current and the history is higher, and outputs the integrated result as a mixing result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a computer-readable non-transitory storage medium. BACKGROUND

[0002] In deep neural network (DNN) processing for a moving image, a recurrent neural network (RNN) structure having high temporal stability is generally used. This is because the RNN structure can associate the current (information of the current frame) and the history (recursive information of the past frame), and thus easily maintain the consistency of the video of the output frame.

[0003] As a system that performs such DNN processing for a moving image, for example, a learning type multi-frame super resolution (MFSR) system that mixes the current and the history and performs super resolution processing on the current frame using the mixed result is known.

[0004] The MFSR system can make the super resolution result more fine by using a large amount of history. However, on the other hand, a problem of the MFSR system is an artifact generated by long-time repetitive processing due to recursive accumulation of error components.

[0005] To solve this problem, a technique has been proposed in which, before mixing, the history is corrected by reducing the strength of the history, thereby relatively reducing the contribution rate of the history in the mixing, to suppress possible artifacts (see, for example, Non-Patent Literature 1).

[0006] BIBLIOGRAPHIC LIST

[0007] NON-PATENT LITERATURE

[0008] Non-Patent Literature 1: Benjamin Naoto Chiche, Arnaud Woiselle, Joana Frontera-Pons, Jean-Luc Starck, “Stable Long-Term Recurrent Video Super-Resolution”, [online], [retrieved on June 22, 2023], Internet <URL: https: / / openaccess.thecvf.com / content / CVPR2022 / papers / Chiche_Stable_Long-Term_Recurrent_Video_Super-Resolution_CVPR_2022_paper.pdf> SUMMARY

[0009] PROBLEMS TO BE SOLVED BY THE INVENTION

[0010] However, in the above-described related art, by reducing the strength of the history, the contribution rate of the history in the mixture is relatively reduced. However, in the DNN processing using the mixture result, high-order feature matching is performed in such a manner that the reduced history strength is ignored. Therefore, the problem with the above-described related art is that the effect of the history correction is reduced, making it difficult to suppress possible artifacts as intended.

[0011] Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a computer-readable non-transitory storage medium, which can more effectively suppress possible artifacts in DNN processing for a moving image.

[0012] Solution to the problem

[0013] To solve the above-described problem, the information processing apparatus according to one aspect of the present disclosure is an information processing apparatus including a DNN configured to infer a current frame based on a mixture result obtained by mixing a current and a history, the current being information related to the current frame to be processed, the history being recursive information including an inference result for a past frame, the DNN including a generation section, a correction section, and an integration section. The generation section generates a slight change component for reducing reliability of the history. The correction section corrects the history by adding the slight change component to the history. The integration section integrates the current and the history so that one of the current and the history having higher reliability than the other has a higher contribution rate than the other, and outputs the integrated current and history as the mixture result. BRIEF DESCRIPTION OF DRAWINGS

[0014] [ Figure 1 ] Figure 1 is a diagram illustrating a basic configuration example of an information processing apparatus according to a first embodiment of the present disclosure.

[0015] [ Figure 2 ] Figure 2 is a diagram illustrating an example of a configuration of a mixture section according to a related art.

[0016] [ Figure 3 ] Figure 3 is a diagram illustrating a configuration example of a mixture section according to the first embodiment of the present disclosure.

[0017] [ Figure 4 ] Figure 4 is a flowchart illustrating a process performed by the information processing apparatus according to the first embodiment of the present disclosure.

[0018] [ Figure 5 ] Figure 5 is a diagram illustrating a configuration example of a mixture section according to a second embodiment of the present disclosure.

[0019] [ Figure 6 ] Figure 6 is a flowchart showing a process procedure executed by the information processing apparatus according to the second embodiment of the present disclosure.

[0020] [ Figure 7 ] Figure 7 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same parts are denoted by the same reference numerals, and redundant description thereof will be omitted.

[0022] Further, in the following description, the information processing apparatus according to the embodiments of the present disclosure executes MFSR processing as DNN processing for a moving image. The information processing apparatus 1 according to the first embodiment will be described with reference to Figures 1 to 4 The information processing apparatus 1A according to the second embodiment will be described with reference to Figure 5 and Figure 6 Further, in the following description, the information processing method according to the embodiments of the present disclosure is an information processing method executed by the information processing apparatuses 1, 1A.

[0023] The present disclosure will be described in the following item order.

[0024] 1. First Embodiment

[0025] 1-1. Basic configuration example of information processing apparatus according to first embodiment

[0026] 1-2. Configuration example of mixing section according to related art

[0027] 1-3. Configuration example of mixing section according to first embodiment

[0028] 1-4. Process procedure of information processing apparatus according to first embodiment

[0029] 2. Second Embodiment

[0030] 2-1. Configuration example of mixing section according to second embodiment

[0031] 2-2. Process procedure of information processing apparatus according to second embodiment

[0032] 3. Modified Example

[0033] 4. Hardware Configuration

[0034] 5. Conclusion

[0035] First Embodiment

[0036] 1-1. Basic Configuration Example of Information Processing Apparatus According to First Embodiment

[0037] Figure 1 is a diagram illustrating a basic configuration example of an information processing apparatus 1 according to a first embodiment of the present disclosure. As shown in Figure 1 , the information processing apparatus 1 includes an input section 10, a sealer 20, a DNN 30, and an output section 40. In addition, as shown in Figure 1 , the information processing apparatus 1 has an RNN structure.

[0038] In DNN processing for a moving image (for example, MFSR processing), an RNN structure having high temporal stability is generally used. The RNN structure can associate the current Cur and the history His, and thus easily maintain the consistency of a video of an output frame. The DNN 30 has very high inference ability (for example, has a super-resolution effect in the case of MFSR processing), but is affected by a slight change in input data, which changes the output result. Therefore, compared to simple filtering processing, flicker is more likely to occur in DNN processing for a moving image. Therefore, in order to improve temporal stability and suppress flicker, the information processing apparatus 1 that performs DNN processing for a moving image is configured to have an RNN structure that can simultaneously receive the current Cur and the history His as inputs, and performs learning including temporal correlation (similarity between the current Cur and the history His).

[0039] Note that the current Cur is information related to a frame to be processed (a current frame). Specifically, the current Cur refers to information indicating the T-th frame among N frames of a moving image. In the present embodiment, it is assumed that the format of the information is a video format of a general moving image. More specifically, for example, it is assumed that three-channel image data, such as YUV, which is a video transmission format in the International Telecommunication Union (ITU) standard.

[0040] In addition, the history His is recursive information including a super-resolution result obtained by inputting an image of a past frame to the DNN 30. The past frame refers to a frame of one or more frames before the current frame. In the present embodiment, in a moving image including N frames, the past frame is the (T-1)-th frame with respect to the T-th frame to be processed.

[0041] Thus, in the present embodiment, as in general feedforward moving image processing, the history His is assumed to be information obtained by recursively capturing a super-resolution image of the (T-1)th frame. As in the case of the current Cur, the format of the information is assumed to be a video format of a general moving image (for example, image data of three channels such as YUV as described above).

[0042] Note that the history His can be represented by a feature map extracted from information of past frames. More specifically, the feature map can be a feature map of F (> 3) channels having an intensity for each pixel. In this case, the feature map is assumed to be information obtained by converting a super-resolution image of the (T-1)th frame using a convolution operation.

[0043] In addition, the history His can be information that has been aligned with the current frame. In this case, the alignment is assumed to be general motion compensation. More specifically, the alignment is assumed to be processing that converts position information of a corresponding pixel using a motion vector that describes an amount of motion of the corresponding pixel from a past frame toward the current frame.

[0044] The input section 10 acquires the current Cur. The current Cur can be an image captured by a camera or a computer graphics (CG) image generated by a CG renderer or the like. The sealer 20 adjusts the number of pixels of the current Cur in accordance with the specifications of the DNN 30.

[0045] The DNN 30 is a deep learning model that learns and infers a super-resolution process performed on the current frame. The DNN 30 is preferably implemented as a deep learning model that uses a super-resolution image and high-definition training data paired with the super-resolution image, and is trained by a method of steepest descent or the like so as to minimize the loss of both the super-resolution image and the high-resolution training data, where the super-resolution image is an output when the current Cur and the history His are input.

[0046] The DNN 30 receives and mixes the current Cur and the history His, uses the mixed result to output a super-resolution result of the current frame, and updates the history His.

[0047] As Figure 1 shown, the DNN 30 includes a mixing section 31 and an inference section 32. The mixing section 31 uses the current Cur and the history His to output a mixed result of the current Cur and the history His. The inference section 32 uses the mixed result to output an inference result for the current frame (i.e., a super-resolution result in the present embodiment), and uses the super-resolution result to update the history His.

[0048] Note that, in the present embodiment, as Figure 1As shown, it is assumed that the mixing section 31 and the inference section 32 are models included in an integrated DNN 30, but the mixing section 31 and the inference section 32 can be configured using separate DNNs.

[0049] In the present embodiment, it is assumed that the mixing section 31 is a multi-layer convolutional neural network (CNN) that outputs a feature map. The mixing section 31 receives the current Cur and the history His as inputs, performs processing that mixes the current Cur and the history His, and outputs a feature map as a mixing result.

[0050] In addition, in the present embodiment, it is assumed that the inference section 32 is a CNN that includes a plurality of layers and is connected to the output of the mixing section 31. The inference section 32 receives the feature map output by the mixing section 31 as an input, outputs a super-resolution image as a super-resolution result, accumulates a feature map at a final stage as recursive information, and outputs a feature map as the history His of the next frame.

[0051] The output section 40 applies general post-processing such as color conversion, coding and decoding to the super-resolution image acquired from the DNN 30, and outputs the resulting super-resolution image.

[0052] 1-2. Configuration example of mixing section according to related art

[0053] In the MFSR system, it is possible to make the super-resolution result finer by using a large amount of history His. However, on the other hand, a problem of the MFSR system is that artifacts generated by long-time repetitive processing due to recursive accumulation of error components.

[0054] Therefore, in order to solve this problem, a related art has been proposed in which, before mixing, the history His is corrected by reducing the strength of the history His, thereby relatively reducing the contribution rate of the history in mixing, to suppress possible artifacts.

[0055] Figure 2 is a diagram showing a configuration example of the mixing section 31' according to the related art. The mixing section 31' has an integration section 31a. As Figure 2 shown, the mixing section 31' calculates a corrected history His M by multiplying the history His by a constant to reduce the strength of the history His M The integration section 31a integrates the current Cur and the corrected history His

[0056] However, in the case of a complex model such as a DNN, high-order feature matching is performed in such a way that the history His intensity is ignored in the inference using the mixed result BM. Therefore, the problem with using the mixing section 31' according to the related art is that the effect of the history His correction is reduced, making it difficult to suppress possible artifacts as intended.

[0057] 1-3. Configuration example of mixing section according to first embodiment

[0058] Therefore, in the information processing method according to the first embodiment, the mixing section 31 generates a slight change component with relatively low correlation with the history His, and calculates a corrected history His by adding the generated slight change component to the history His M The "slight change component" is a component for reducing the reliability of the history His. The "slight change component" can be replaced with a "perturbation component".

[0059] Figure 3 is a diagram illustrating a configuration example of the mixing section 31 according to the first embodiment of the present disclosure. As shown in Figure 3 , the mixing section 31 includes an integration section 31a, a generation section 31b, and a correction section 31c.

[0060] The generation section 31b generates a slight change component PM with relatively low correlation with the history His. The phrase "relatively low correlation" includes the case where there is no correlation. For example, the generation section 31b generates a slight change component PM that is not correlated with the history His.

[0061] The slight change component PM generated at this time can be any information as long as the information is not correlated with the history His. In the present embodiment, the generation section 31b generates, for example, a noise image having the same number of channels as the history His.

[0062] As a method of generating noise, a noise generation method in general image processing can be used. For example, the generation section 31b generates a noise image such as Gaussian noise. Subsequently, the generation section 31b scales the generated noise image by a coefficient set in advance, and outputs the scaled noise image as the slight change component PM.

[0063] The correction section 31c calculates a corrected history His M from the history His using the slight change component PM generated by the generation section 31b. The corrected history His M is calculated by adding the slight change component PM to the history His .

[0064] The integration section 31a integrates the current Cur and the corrected history His MThe integration is performed, and the integrated result is output as a mixed result BM. Preferably, the integration unit 31a performs the integration by performing a linear addition operation on the information of the current Cur and the corrected history His M

[0065] At this time, the linear combination can be any operation, and can be performed by, for example, an alpha blending method using a general single mixing coefficient. In the present embodiment, it is assumed that the DNN 30 learns and reasons a super-resolution process performed on the current frame, and thus, it is assumed that, when the feature maps of the current Cur and the corrected history His M are input, the parameters for the linear combination can be automatically acquired to output an output image with higher resolution.

[0066] At this time, the integration unit 31a can increase the resolution of the output image by comparing the current Cur and the corrected history His M , and increasing the weight of the corrected history His M when the linear addition operation is performed, in a case where the corrected history His M has higher reliability. On the other hand, in a case where the corrected history His M has lower reliability, the integration unit 31a can suppress possible artifacts in long-time processing by decreasing the weight of the corrected history His M when the linear addition operation is performed.

[0067] 1-4. Processing procedure of the information processing apparatus according to the first embodiment

[0068] Now, a processing procedure performed by the information processing apparatus 1 according to the first embodiment will be described with reference to Figure 4 Figure 4 is a flowchart illustrating a processing procedure performed by the information processing apparatus 1 according to the first embodiment of the present disclosure. Note that Figure 4 mainly illustrates the processing procedure in the above-described RNN structure.

[0069] As illustrated in Figure 4 , the mixing unit 31 acquires the current Cur and the history His (step S101). Then, the generation unit 31b generates a slight change component PM having relatively low correlation with the history His (step S102).

[0070] In addition, the correction unit 31c corrects the history His using the slight change component PM generated by the generation unit 31b (step S103). Then, the integration unit 31a integrates the current Cur and the corrected history His M , and outputs the integrated current Cur and the corrected history His M ​​The mixing result BM is output as a result of the mixing (step S104).

[0071] Then, the inference unit 32 performs super-resolution processing on the current frame using the mixing result BM acquired from the mixing unit 31, and outputs a super-resolution result (step S105). In addition, the inference unit 32 updates the history His based on the super-resolution result (step S106). Subsequently, the recursive processing of the RNN is repeated.

[0072] As described above, the information processing apparatus 1 according to the first embodiment can more effectively suppress possible artifacts in the MFSR processing.

[0073] 2. Second Embodiment

[0074] 2-1. Configuration Example of Mixing Unit According to Second Embodiment

[0075] Figure 5 is a diagram illustrating a configuration example of the mixing unit 31A according to the second embodiment of the present disclosure. Note that the basic configuration example of the information processing apparatus 1A according to the second embodiment is the same as that of the information processing apparatus 1 illustrated in Figure 1 , and thus the description thereof is omitted here. In addition, Figure 5 corresponding to Figure 3 , and thus the description using Figure 5 mainly relates to the difference from Figure 3 .

[0076] As illustrated in Figure 5 , the mixing unit 31A differs from the mixing unit 31 illustrated in Figure 3 in that the generating unit 31b generates a fine change component FFM related to the current Cur.

[0077] In addition, as illustrated in Figure 5 , the mixing unit 31A differs from the mixing unit 31 illustrated in Figure 3 in that the mixing unit 31A further includes a map calculating unit 31d. The map calculating unit 31d generates a correction amount map MM using auxiliary information. The auxiliary information includes a motion amount, a user input, and an occlusion map. The correction amount map MM is information indicating a correction amount of each pixel in an output image as an inference result for past frames included in the history His.

[0078] This will be described more specifically. The generation section 31b of the mixing section 31A generates a slight change component related to the current Cur using the current Cur. At this time, the generated slight change component needs to be information obtained by conversion using the current Cur. For example, in a case where the current Cur is an image including three channels of the above-described YUV and the history His is a feature map including N channels, the generation section 31b of the mixing section 31A calculates the slight change component FFM (false FM) by the following equation (A).

[0079] [Equation 1]

[0080] ...(a)

[0081] Here, the coefficient S is a scaling coefficient, and is preferably set to a value of 0.1 or less. The generation section 31b of the mixing section 31A calculates the slight change component FFM by multiplying the input YUV by the matrix M of N rows and 4 columns and the coefficient S.

[0082] The mixing section 31 according to the first embodiment adds the slight change component PM not related to the history His to the history His to effectively relatively reduce the contribution rate of the recursive information in the integration section 31a. However, in a case where the slight change component PM is completely randomly generated, the recursive information cannot be correctly accumulated, and thus the fineness of the final super-resolution result can be reduced.

[0083] In contrast, as in the mixing section 31A according to the present embodiment, by generating the slight change component FFM related to the current Cur using the current Cur and adding the slight change component FFM to the history His to reduce the reliability of the history His, it is possible to relatively reduce the contribution rate of the corrected history His M in the integration section 31a without impairing the normal accumulation of the recursive information. In addition, this is effective for maintaining the high fineness of the super-resolution result.

[0084] The map calculation section 31d calculates a correction amount map MM using the auxiliary information, and outputs the correction amount map MM. The correction amount map MM is information calculated in the correction section 31c after the map calculation section 31d for correcting the history His, and is information of one channel having the same number of pixels as the number of pixels in the history His.

[0085] Assume that a correction amount map MM is calculated using various types of auxiliary information. In the present embodiment, for example, a motion amount is used as the auxiliary information to calculate a first correction amount map (hereinafter, appropriately referred to as "correction amount map MM-1"). The motion amount is an absolute value of a motion vector that describes an amount of motion of a corresponding pixel from a past frame to a current frame in a moving image to be processed. In the present embodiment, the map value is set so as to increase a correction amount of a corresponding pixel in the correction amount map MM-1 as a motion amount of a target pixel decreases.

[0086] It has been confirmed that the artifact problem in the long-time processing, which is a problem of the present disclosure, is caused by accumulation of an error component of the history His in the time direction. Here, a size of the accumulated error component increases as a motion amount of a target pixel decreases, that is, the size is larger for a pixel that remains still. Therefore, as in the present embodiment, by increasing a correction amount for a corresponding pixel as a motion amount of a target pixel decreases, an addition degree of the minute change component FFM in the correction section 31c after the map calculation section 31d is effectively increased, and accumulation of the error component in the time direction is suppressed.

[0087] In addition, in the present embodiment, for example, a user input is used as the auxiliary information to calculate a second correction amount map (hereinafter, appropriately referred to as "correction amount map MM-2"). The user input is an input in which a user of the information processing apparatus 1A specifies a correction amount for a desired pixel position using the information processing apparatus 1A. At this time, the map value is set so that a correction amount for a corresponding pixel in the correction amount map MM-2 increases as an input value specified by the user input increases. This allows a correction amount for the history His to be effectively adjusted in accordance with an intention of the user.

[0088] In addition, in the present embodiment, for example, an occlusion map is used as the auxiliary information to calculate a third correction amount map (hereinafter, appropriately referred to as "correction amount map MM-3"). The occlusion is a region that occurs at a boundary between a foreground object and a background object of a moving image. In the present embodiment, the map value is set so that a correction amount for a corresponding pixel region in the correction amount map MM-3 increases as a possibility of the pixel region becoming an occlusion increases.

[0089] In general, a problem of the occurrence of the occlusion is that the history His related to a region of interest in a past frame is uncertain, resulting in the occurrence of so-called ghosting. Therefore, as in the present embodiment, by increasing a correction amount for a pixel region having a higher possibility of becoming an occlusion, a contribution rate of the history His of the region of interest in the correction section 31c after the map calculation section 31d is effectively reduced, and the occurrence of the ghosting is suppressed.

[0090] The map calculation section 31d outputs a final correction amount map MM using one or more of the above correction amount maps MM-1, MM-2, MM-3. At this time, each map value of the final correction amount map MM is preferably a representative value such as an average value or a maximum value of the correction amount maps MM-1, MM-2, MM-3. In the present embodiment, it is assumed that the maximum value is selected.

[0091] In the present embodiment, the correction section 31c calculates a corrected history His M based on the correction amount map MM by addition of the slight change component FFM, and outputs the corrected history His M . Specifically, the correction section 31c calculates the corrected history His M = correction amount map MM x slight change component FFM + (1.0 - correction amount map MM) x history His" by the expression "correction amount map MM x slight change component FFM + (1.0 - correction amount map MM) x history His". M .

[0092] Therefore, by more intensively adding the slight change component FFM, which is irrelevant to the history His, to the region designated by the correction amount map MM, the integration section 31a after the correction section 31c can control the contribution rate of the history His to locally increase or decrease the contribution rate.

[0093] 2-2. Processing procedure of information processing apparatus according to second embodiment

[0094] Now, a processing procedure performed by the information processing apparatus 1A according to the second embodiment will be described with reference to Figure 6 . Figure 6 is a flowchart showing a processing procedure performed by the information processing apparatus 1A according to the second embodiment of the present disclosure. Note that Figure 6 mainly shows a processing procedure in the above-described RNN structure.

[0095] As shown in Figure 6 , the mixing section 31A acquires the current Cur, the history His, and the auxiliary information (step S201). Then, the generation section 31b generates the slight change component FFM related to the current Cur using the current Cur (step S202).

[0096] In addition, the map calculation section 31d calculates the correction amount map MM using the auxiliary information (step S203). Then, the correction section 31c corrects the history His using the slight change component FFM and the correction amount map MM (step S204).

[0097] Then, the integration section 31a integrates the current Cur and the corrected history His M , and integrates the current Cur and the corrected history His MThe mixing result BM is output as a result of the mixing (step S205).

[0098] Then, the inference unit 32 performs super-resolution processing on the current frame using the mixing result BM acquired from the mixing unit 31A, and outputs a super-resolution result (step S206). In addition, the inference unit 32 updates the history His based on the super-resolution result (step S207). Subsequently, the recursive processing of the RNN is repeated.

[0099] As described above, the mixing unit 31A according to the second embodiment can generate a fine change component FFM by using the current Cur, while selectively reducing the contribution rate of the history His using the auxiliary information only for a region in which an artifact is likely to occur, a region expected by the user, and a region in which the history His is uncertain, thereby selectively suppressing a possible artifact and maintaining high fineness of the final super-resolution result.

[0100] That is, the information processing apparatus 1A according to the second embodiment allows a possible artifact to be more effectively suppressed in the MFSR processing.

[0101] 3. Modification

[0102] Among the processing operations described in the above-described embodiments of the present disclosure, all or some of the processing operations described as being automatically performed can be manually executed, or all or some of the processing operations described as being manually executed can be automatically executed by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters described in the above documents and drawings can be arbitrarily changed. For example, the various types of information shown in the drawings are not limited to the shown information.

[0103] In addition, the components of each apparatus shown in the drawings are conceptually functional, and are not necessarily physically configured as shown in the drawings. That is, the specific form of distribution and integration of the apparatus is not limited to those shown in the drawings, and all or some of the apparatuses can be physically distributed or integrated in any unit according to various loads, usage conditions, and the like.

[0104] In addition, the above-described embodiments of the present disclosure can be appropriately combined as long as the processing contents do not contradict each other. In addition, the order of the steps shown in the sequence charts or flowcharts of the present embodiment can be appropriately changed.

[0105] In addition, in the example of the above-described embodiments of the present disclosure, the DNN processing for a moving image is MFSR processing. However, the DNN processing for a moving image is not limited to MFSR processing, and can be, for example, colorization, restoration processing, deblurring, or the like.

[0106] 4. Hardware configuration

[0107] In addition, the information processing apparatus 1, 1A according to the embodiment of the present disclosure described above is implemented by, for example, a computer 1000 configured as shown in Figure 7 Fig. 1. Figure 7 Fig. 1 is a hardware configuration diagram showing an example of a hardware configuration of the computer 1000 that implements the functions of the information processing apparatus 1, 1A. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage apparatus 1400, a communication interface 1500, and an input / output interface 1600. The respective parts of the computer 1000 are connected together via a bus 1050.

[0108] The CPU 1100 operates based on a program stored in the ROM 1300 or the secondary storage apparatus 1400, and controls each part. For example, the CPU 1100 loads a program stored in the ROM 1300 or the secondary storage apparatus 1400 into the RAM 1200, and executes processing corresponding to the various programs.

[0109] The ROM 1300 stores a boot program such as a basic input / output system (BIOS), a program depending on the hardware of the computer 1000, and the like, which is executed by the CPU 1100 at the time of startup of the computer 1000.

[0110] The secondary storage apparatus 1400 is a computer-readable recording medium that non-transitorily records a program executed by the CPU 1100, data used by the program, and the like. Specifically, the secondary storage apparatus 1400 is a recording medium that records a program according to the present embodiment, which is an example of program data 1450.

[0111] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550. For example, the CPU 1100 receives data from another apparatus via the communication interface 1500, and transmits data generated by the CPU 1100 to an apparatus via the communication interface 1500.

[0112] The input / output interface 1600 is an interface for connecting an input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard and a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface that reads a program and the like recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase-change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a tape medium, a magnetic recording medium, a semiconductor memory, and the like.

[0113] For example, in a case where the computer 1000 is used as the information processing apparatus 1, 1A, the CPU 1100 of the computer 1000 executes a program loaded on the RAM 1200 to realize the function of each of the above-described sections. The secondary storage device 1400 stores the program according to the present embodiment, various models including the DNN 30, and various types of data. Note that the CPU 1100 reads the program data 1450 from the secondary storage device 1400 and executes the program data, but as another example, the CPU can acquire these programs from other apparatuses via the external network 1550.

[0114] 5. Conclusion

[0115] As described above, according to the embodiment of the present disclosure, the information processing apparatus 1, 1A includes the DNN 30 that reasons a current frame to be processed based on a mixture result BM obtained by mixing a current Cur and a history His, where the current Cur is information related to the current frame, and the history His is recursive information including a reasoning result for a past frame, and the DNN 30 includes a generation section 31b, a correction section 31c, and an integration section 31a. The generation section 31b generates a slight change component PM and FFM for reducing the reliability of the history His. The correction section 31c corrects the history His by adding the slight change component PM and FFM to the history His. The integration section 31a integrates the current Cur and the history His so that one of them having higher reliability than the other has a higher contribution rate than the other, and outputs the integrated current Cur and history His as the mixture result BM. This allows more effective suppression of possible artifacts in DNN processing for a moving image.

[0116] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments themselves but can be variously modified without departing from the spirit of the present disclosure. In addition, components of different embodiments and modified examples can be appropriately combined.

[0117] In addition, the effects of the embodiments described herein are merely examples and are not limiting, and other effects can be derived from the following description.

[0118] Note that the present technology can also adopt the following configurations. (1)

[0120] An information processing apparatus including a DNN configured to infer a current frame to be processed based on a mixture result obtained by mixing a current and a history, the current being information related to the current frame, the history being recursive information including an inference result for a past frame,

[0121] wherein the DNN includes:

[0122] a generation section configured to generate a slight change component for reducing reliability of the history;

[0123] a correction section configured to correct the history by adding the slight change component to the history; and

[0124] an integration section configured to integrate the current and the history so that one of the current and the history having higher reliability than the other has a higher contribution rate than the other, and output the integrated current and history as the mixture result. (2)

[0126] The information processing apparatus according to (1),

[0127] wherein the generation section generates the slight change component related to the current by using the current. (3)

[0129] The information processing apparatus according to (1) or (2), further comprising:

[0130] a map calculation section configured to calculate a correction amount map indicating a correction amount for each pixel in an output image as an inference result for the past frame included in the history,

[0131] wherein the correction section adds the slight change component to the history based on the correction amount map. (4)

[0133] The information processing apparatus according to (3),

[0134] wherein the map calculation section calculates the correction amount map based on auxiliary information,

[0135] the auxiliary information including at least one of a motion amount for each pixel from a past frame, a user input, or an occlusion map indicating a likelihood of becoming an occlusion. (5)

[0137] The information processing apparatus according to (4),

[0138] The map calculating section calculates the correction amount map based on the motion amount so as to increase the correction amount as the motion amount decreases. (6)

[0140] The information processing apparatus according to (4) or (5),

[0141] The map calculating section calculates the correction amount map based on the user input so as to increase the correction amount as an input value specified by the user input increases. (7)

[0143] The information processing apparatus according to (4), (5), or (6),

[0144] The map calculating section calculates the correction amount map based on the occlusion map so as to increase the correction amount for a pixel region having a higher possibility of being occluded. (8)

[0146] The information processing apparatus according to (1),

[0147] The generating section generates the slight change component with a relatively low correlation with the history. (9)

[0149] The information processing apparatus according to (8),

[0150] The generating section generates a noise image as the slight change component. (10)

[0152] An information processing method executed by a computer, the information processing method comprising:

[0153] inferring a current frame to be processed based on a mixture result obtained by mixing a current and a history, the current being information related to the current frame, the history being recursive information including an inference result for a past frame,

[0154] generating a slight change component for reducing reliability of the history;

[0155] correcting the history by adding the slight change component to the history; and

[0156] integrating the current and the history so that one of the current and the history having higher reliability than the other has a higher contribution rate than the other, and outputting the integrated current and history as the mixture result. (11)

[0158] A computer-readable non-transitory storage medium storing a program that causes a computer to execute:

[0159] infer a current frame to be processed based on a mixture result obtained by mixing a current and a history, the current being information related to the current frame, the history being recursive information including an inference result for a past frame,

[0160] generate a small change component for reducing reliability of the history;

[0161] correct the history by adding the small change component to the history; and

[0162] integrate the current and the history so that one of the current and the history having higher reliability than the other has a higher contribution rate than the other, and output the integrated current and history as the mixture result.

[0163] LIST OF REFERENCE NUMERALS

[0164] 1, 1A information processing apparatus

[0165] 10 input unit

[0166] 20 sealer

[0167] 30 DNN

[0168] 31, 31A mixture unit

[0169] 31a integration unit

[0170] 31b generation unit

[0171] 31c correction unit

[0172] 31d graph calculation unit

[0173] 32 inference unit

[0174] 40 output unit

[0175] BM mixture result

[0176] Cur current

[0177] FFM small change component

[0178] His history

[0179] His M corrected history

[0180] MM correction amount graph

[0181] PM small change component

Claims

1. An information processing apparatus comprising a deep neural network (DNN) configured to infer a current frame to be processed based on a mixture of results, the mixture of results being obtained by mixing current and history, wherein the current is information related to the current frame, and the history is recursive information including inference results for past frames. wherein The DNN includes: A generation unit is configured to generate minute variation components to reduce the reliability of the history; A correction unit, configured to correct the history by adding the minute change component to the history; and An integration unit is configured to integrate the current and the history such that the one with higher reliability than the other has a higher contribution rate than the other, and outputs the integrated current and history as the hybrid result.

2. The information processing device according to claim 1, wherein The generation unit generates the minute change components related to the current state by using the current state.

3. The information processing apparatus according to claim 1, further comprising: The image calculation unit is configured to calculate a correction map indicating the amount of correction for each pixel in the output image, as an inference result for the past frames included in the history. The correction unit adds the minute change component to the history based on the correction map.

4. The information processing device according to claim 3, wherein, The graph calculation unit calculates the correction graph based on auxiliary information, and The auxiliary information includes at least one of the following: motion for each pixel from past frames, user input, or an occlusion map indicating the likelihood of occlusion.

5. The information processing apparatus according to claim 4, wherein The graph calculation unit calculates the correction graph based on the amount of exercise, and increases the correction amount as the amount of exercise decreases.

6. The information processing apparatus according to claim 4, wherein The graph calculation unit calculates the correction graph based on the user input, such that the correction amount increases as the input value specified by the user input increases.

7. The information processing apparatus according to claim 4, wherein The image calculation unit calculates the correction amount map based on the occlusion map, so as to increase the correction amount for pixel regions with a higher probability of being occluded.

8. The information processing apparatus according to claim 1, wherein The generation unit generates minute change components that have relatively low correlation with the history.

9. The information processing apparatus according to claim 8, wherein The generating unit generates a noisy image as the minute change component.

10. An information processing method executed by a computer, the information processing method comprising: The current frame to be processed is inferred based on the mixed results, which are obtained by mixing the current and the history, where the current is information related to the current frame and the history is recursive information including inference results for past frames. Generate minute variation components to reduce the reliability of the history; The history is corrected by adding the minute change component to the history; as well as The current and the history are integrated so that one of the current and the history having higher reliability than the other has higher contribution rate than the other, and the integrated current and history are output as the mixed result.

11. A computer-readable non-transitory storage medium storing a program causing a computer to execute: inference of a current frame to be processed based on a mixed result obtained by mixing a current and a history, the current being information related to the current frame, the history being recursive information including an inference result for a past frame; generation of a slight change component for reducing reliability of the history; correction of the history by adding the slight change component to the history; and integration of the current and the history so that one of the current and the history having higher reliability than the other has higher contribution rate than the other, and output of the integrated current and history as the mixed result.

Images