Image processing device and image processing method

Abstract

To provide a technique for reducing blur generated as a result of reduction of fluctuation while preventing correction remaining of fluctuation from being enhanced.SOLUTION: Fluctuation in an image is reduced on the basis of fluctuation information, and the contrast of the image is corrected on the basis of the fluctuation information and brightness variation in a spatial direction in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for reducing fluctuations in an input image. [Background technology]

[0002] In surveillance camera use cases such as port surveillance and infrastructure monitoring, for example, when taking telephoto shots of ships or aircraft, it is known that fluctuations in the subject image caused by uneven changes in the refractive index of the atmosphere (such as heat haze) can reduce the visibility of the subject. In response to this issue, for example, Patent Document 1 discloses a technology that determines the fluctuation intensity using an input image and corrects the fluctuation by averaging frame images in the time direction (frame direction) according to the determined fluctuation intensity. The technology disclosed in Patent Document 1 makes it possible to appropriately correct the fluctuation even when the fluctuation intensity changes depending on the shooting environment. Patent Document 1 also discloses a technology that reduces the blurring caused by image averaging by sharpening the image after fluctuation reduction according to the fluctuation intensity, since image averaging causes the image to become blurred. Patent Document 1 also discloses that the filter size of the image sharpening filter should be increased as the fluctuation intensity increases. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2015 / 132826 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technology disclosed in Patent Document 1, image sharpening may emphasize jitter, potentially weakening the jitter reduction effect. For example, with the technology disclosed in Patent Document 1, the degree of image sharpening is increased as the jitter intensity increases. However, the greater the jitter intensity, the greater the risk of jitter remaining after correction. Furthermore, when the subject is moving, averaging frame images causes blurring of the subject image, making it impossible to increase the number of frame images to be averaged. This may result in jitter remaining after correction because jitter reduction cannot be applied strongly. Increasing the degree of sharpening when jitter remaining after correction exists may emphasize the jitter remaining after correction due to sharpening, potentially weakening the jitter reduction effect. Furthermore, the emphasis on jitter remaining after correction is more noticeable in areas of the subject image where spatial variation is small, such as areas where the image brightness is nearly flat. Even if the residual jitter correction is emphasized in areas where the spatial change of the subject image is large, the subject image is also emphasized at the same time, making the residual jitter correction less noticeable. On the other hand, if slight jitter occurring in flat areas of the image is emphasized, the subject is not emphasized very much because the spatial change of the subject image is small, and the emphasis of the residual jitter correction becomes relatively more noticeable, which may result in a weakening of the jitter reduction effect. The present invention provides a technology for reducing blurring caused by jitter reduction while preventing the emphasis of the residual jitter correction. [Means for solving the problem]

[0005] One aspect of the present invention is characterized by comprising a reduction means for reducing fluctuations in an image based on fluctuation information, and a correction means for correcting the contrast of the image based on the fluctuation information and a spatial luminance change in the image. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a technique for reducing blur caused by reducing fluctuations while preventing the uncorrected fluctuations from being emphasized. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 2] 10 is a flowchart of the operation of the image processing device. [Figure 3] FIG. 10 is a diagram for explaining the influence of fluctuations on an input image. [Figure 4] FIG. 10 is a diagram for explaining a method for acquiring fluctuation information in an input image. [Figure 5] 4A and 4B are diagrams for explaining an example of fluctuation reduction processing by a reduction unit 102. [Figure 6] FIG. 1 is a diagram for explaining a problem with the conventional technology. [Figure 7] FIG. 4 is a diagram for explaining the effect of the first embodiment. [Figure 8] FIG. 4 is a diagram for explaining the effect of the first embodiment. [Figure 9] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 10] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 11] FIG. 10 is a diagram for explaining the effect of the third embodiment. [Figure 12] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus. [Figure 13] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0009] [First embodiment] First, an example of the functional configuration of an image processing device according to this embodiment will be described using the block diagram in Fig. 1. The image processing device acquires input images for each frame. The input images for each frame may be images for each frame in a moving image, or may be still images of each frame captured periodically or irregularly.

[0010] The input image is a color image in which each pixel has pixel values ​​of multiple color components. For example, the input image is an RGB color image in which each pixel has pixel values ​​for red (Red: R), green (Green: G), and blue (Blue: B). Such an input image is, for example, an image generated according to the amount of light that passes through color filters corresponding to each color provided on an image sensor and is converted into an electrical signal by the image sensor. However, the input image is not limited to a color image and may be a monochrome image, for example. The input image may contain fluctuations in the subject image due to uneven changes in the refractive index of the atmosphere (such as heat haze).

[0011] The method by which the image processing device acquires such input images is not limited to a specific method. For example, if the image processing device is an imaging device, the image processing device acquires, as input images, captured images of each frame captured by its own imaging unit. The image processing device may also acquire, as input images, captured images captured by an external imaging device or captured images stored in a computer device such as an external server device. Thus, the method and configuration by which the image processing device acquires input images are not limited to a specific method or configuration.

[0012] The acquisition unit 101 acquires fluctuation information indicating the amount of fluctuation (degree of fluctuation) contained in an input image from an input image acquired by an image processing device. Here, the influence of fluctuation on an input image will be described with reference to FIG. 3. FIG. 3(a) shows an example of a captured image (non-fluctuation captured image) obtained by capturing an image of a stationary subject in a state without fluctuation. FIG. 3(b) shows an example of a captured image (fluctuation captured image) obtained by capturing an image of a stationary subject in a state with fluctuation.

[0013] As shown in Figures 3(a) and 3(b), even when capturing an image of a stationary subject, if the image is captured in a state where there is shaking, the image will be distorted. Figure 3(c) shows the pixel value (solid line) at pixel position P of the image captured without shaking for each frame, and the pixel value (dotted line) at pixel position P of the image captured with shaking for each frame. In Figure 3(c), the horizontal axis represents time (frame), and the vertical axis represents pixel value.

[0014] 3(c), the pixel value at pixel position P in the non-jitter captured image of each frame is almost constant, while the pixel value at pixel position P in the jitter captured image of each frame changes. Therefore, when there is jitter, a phenomenon occurs in which an actually stationary subject is captured as if it were a moving subject.

[0015] Next, a method by which the acquisition unit 101 acquires fluctuation information in an input image will be described with reference to Fig. 4. Fig. 4 shows pixel values ​​at the same pixel position Q in the input image of each frame, with the horizontal axis representing time (frame) and the vertical axis representing pixel value. The change in pixel value shown in Fig. 4(b) is larger than the change in pixel value shown in Fig. 4(a), and the change in pixel value shown in Fig. 4(c) is larger than the change in pixel value shown in Fig. 4(b).

[0016] Here, the frame corresponding to time t2 is assumed to be the current frame, and the frame corresponding to time t1 is assumed to be a past frame that is at least one frame before the current frame. In this case, the acquisition unit 101 calculates the difference between the pixel value at pixel position Q in the input image of the frame corresponding to time t2 and the pixel value at pixel position Q in the input image of the frame corresponding to time t1. There are various methods for calculating the difference between one pixel value and another pixel value (the difference between pixel values ​​between captured images), and the method is not limited to a specific method. For example, the acquisition unit 101 may calculate the absolute value of the difference between one pixel value and the other pixel value as the difference between the one pixel value and the other pixel value. Alternatively, the acquisition unit 101 may calculate the square of the difference between the one pixel value and the other pixel value as the difference between the one pixel value and the other pixel value. In this way, for each pixel position in the input image of the current frame, the acquisition unit 101 calculates the difference between the pixel value of that pixel position in the input image and the pixel value of that pixel position in a frame that is older than the current frame (a frame that is one or more frames older than the current frame).

[0017] The acquisition unit 101 then calculates fluctuation information for the input image of the current frame based on the difference calculated for each pixel position in the input image of the current frame. The method for calculating the fluctuation information for the input image of the current frame based on the difference calculated for each pixel position in the input image of the current frame is not limited to a specific method. For example, the acquisition unit 101 calculates, as the fluctuation information for the input image of the current frame, an average value, a sum value, or a normalized value of the sum value of the differences calculated for each pixel position in the input image of the current frame.

[0018] The reduction unit 102 determines the intensity of the fluctuation reduction process, which is a process for reducing fluctuations in the input image, based on the fluctuation information acquired from the input image by the acquisition unit 101. Then, the reduction unit 102 executes the fluctuation reduction process with the determined intensity on the input image. As the fluctuation reduction process, a smoothing process (smoothing (averaging) process in the time direction (frame direction)) is used, which smooths images of multiple frames to generate an image of the current frame, such as a simple moving average or a weighted moving average between input images of multiple frames (an input image of the current frame and input images of one or more frames prior to the current frame).

[0019] The reduction unit 102 increases the intensity of the jitter reduction process as the amount of jitter represented by the jitter information increases (the degree of jitter increases). For example, the reduction unit 102 increases the number of frames used in the smoothing process in the time direction (frame direction) as the amount of jitter represented by the jitter information increases (the degree of jitter increases) (i.e., the intensity of the jitter reduction process increases). Also, for example, the reduction unit 102 smoothes frames captured over a longer period of time as the amount of jitter represented by the jitter information increases (the degree of jitter increases) (i.e., the intensity of the jitter reduction process increases).

[0020] Furthermore, the reduction unit 102 weakens the intensity of the jitter reduction process as the amount of jitter represented by the jitter information becomes smaller (the degree of jitter becomes weaker). For example, the reduction unit 102 reduces the number of frames used in the smoothing process in the time direction (frame direction) as the amount of jitter represented by the jitter information becomes smaller (the degree of jitter becomes weaker) (i.e., the intensity of the jitter reduction process becomes weaker). Also, for example, the reduction unit 102 smoothes frames captured over a shorter period of time as the amount of jitter represented by the jitter information becomes smaller (the degree of jitter becomes weaker) (i.e., the intensity of the jitter reduction process becomes weaker).

[0021] An example of the jitter reduction process by the reduction unit 102 will be described with reference to FIG. 5. In FIG. 5, the horizontal axis represents frames (time), and the vertical axis represents pixel values. FIG. 5(a) shows pixel values ​​at the same pixel position P in jitter-captured images of each frame. FIG. 5(b) shows changes in pixel values ​​at pixel position P in an image obtained by performing a smoothing process, which is jitter correction at a first intensity, on the input image (jitter-captured image) of FIG. 5(a). FIG. 5(c) shows changes in pixel values ​​at pixel position P in an image obtained by performing a smoothing process, which is jitter correction at a second intensity (>first intensity), on the input image (jitter-captured image) of FIG. 5(a). Changes in pixel values ​​due to jitter are approximated to a normal distribution based on a predetermined position, and can therefore be reduced by smoothing in the frame direction.

[0022] Furthermore, the reduction unit 102 may change the weighting used in the smoothing process according to the strength of the jitter reduction process. For example, the reduction unit 102 increases the weighting value as the strength of the jitter reduction process increases, and decreases the weighting value as the strength of the jitter reduction process decreases. The reduction unit 102 then performs the smoothing process using the weighting value adjusted (changed) in this manner to generate an image of the current frame. In this way, the parameters in the smoothing process that are changed according to the strength of the jitter reduction process are not limited to a specific form.

[0023] The calculation unit 104 calculates a histogram of brightness values ​​in the input image (brightness histogram). The correction unit 103 corrects the contrast of the reduced image generated by the reduction unit 102 performing fluctuation reduction processing on the input image, based on the fluctuation information acquired by the acquisition unit 101 and the brightness histogram calculated by the calculation unit 104.

[0024] The correction unit 103 emphasizes the contrast of the reduced image more as the amount of fluctuation represented by the fluctuation information increases (the degree of fluctuation increases). Furthermore, the correction unit 103 calculates "spatial luminance change of the input image" from the luminance histogram. For example, the correction unit 103 detects multiple luminance peaks from the luminance histogram and calculates the distance (luminance) between the peaks as "spatial luminance change of the input image." The "distance (luminance) between peaks" may be the maximum distance among the distances between peaks, or may be the average value of the distances between peaks. The correction unit 103 emphasizes the contrast of the reduced image more as the distance (luminance) between peaks increases.

[0025] The correction unit 103 then outputs the image obtained by the correction (the input image with the contrast corrected) as an output image. The output destination of the output image is not limited to a specific output destination. For example, the correction unit 103 may display the output image on a display unit (not shown) of the image processing apparatus, store the output image in a memory (not shown) of the image processing apparatus, or transmit the output image to an external device via a network interface (not shown) of the image processing apparatus.

[0026] Next, the operation of the image processing device will be described with reference to the flowchart in Fig. 2. Note that the processing according to the flowchart in Fig. 2 is performed for each input image input to the image processing device.

[0027] In step S1, the acquisition unit 101 acquires fluctuation information from an input image acquired by the image processing device. In step S2, the calculation unit 104 calculates a luminance histogram, which is a histogram of luminance values ​​in the input image.

[0028] In step S3, the reduction unit 102 determines whether the amount of fluctuation (degree of fluctuation) represented by the fluctuation information acquired in step S1 is large. For example, if the amount of fluctuation (degree of fluctuation) represented by the fluctuation information acquired in step S1 is equal to or greater than a threshold, the reduction unit 102 determines that the amount of fluctuation (degree of fluctuation) represented by the fluctuation information is large. On the other hand, if the amount of fluctuation (degree of fluctuation) represented by the fluctuation information acquired in step S1 is less than the threshold, the reduction unit 102 determines that the amount of fluctuation (degree of fluctuation) represented by the fluctuation information is small.

[0029] If it is determined that the amount of fluctuation (degree of fluctuation) represented by the fluctuation information is large, the process proceeds to step S4. On the other hand, if it is determined that the amount of fluctuation (degree of fluctuation) represented by the fluctuation information is small, the process according to the flowchart in FIG. 2 ends.

[0030] In step S4, the reduction unit 102 determines the strength of the fluctuation reduction process according to the amount of fluctuation (degree of fluctuation) represented by the fluctuation information acquired in step S1. Then, the reduction unit 102 executes the fluctuation reduction process of the determined strength on the input image.

[0031] In step S5, the correction unit 103 calculates the "spatial luminance change of the input image" from the luminance histogram and determines whether the "spatial luminance change of the input image" is large. For example, the correction unit 103 detects multiple luminance peaks from the luminance histogram. If the distance (luminance) between the peaks is equal to or greater than a threshold, the correction unit 103 determines that the "spatial luminance change of the input image" is large, and if the distance (luminance) between the peaks is less than the threshold, the correction unit 103 determines that the "spatial luminance change of the input image" is small. As described above, the "distance (luminance) between peaks" may be the maximum distance between the peaks or the average value of the distances between the peaks.

[0032] If it is determined that the "spatial brightness change of the input image" is large as a result of this determination, the processing proceeds to step S6; if it is determined that the "spatial brightness change of the input image" is small, the processing according to the flowchart in Figure 2 ends.

[0033] In step S6, the correction unit 103 corrects the contrast of the reduced image generated by the reduction unit 102 performing fluctuation reduction processing on the input image, based on the fluctuation information acquired by the acquisition unit 101 and the luminance histogram calculated by the calculation unit 104. Then, the correction unit 103 outputs the image obtained by the correction as an output image.

[0034] The determination process in step S3 and the determination process in step S5 may be omitted as appropriate.

[0035] Here, the problems with the conventional technology will be explained using Fig. 6. Fig. 6(a) shows an image having a high-luminance region (region on the left) which is a region of a pixel group having a high luminance value, and a low-luminance region (region on the right) which is a region of a pixel group having a low luminance value.

[0036] Graphs (1) to (4) shown in Fig. 6(b) show the luminance values ​​at each pixel position on horizontal line A of the image shown in Fig. 6(a). The horizontal axis of these graphs indicates the pixel position on horizontal line A, and the vertical axis indicates the luminance value.

[0037] Graph (1) shows the luminance value at each pixel position on horizontal line A of the image when no fluctuation occurs, and graphs (2), (3), and (4) show the luminance value at each pixel position on horizontal line A of the image when fluctuation occurs.

[0038] More specifically, graph (2) shows the luminance values ​​at each pixel position on horizontal line A of an image that has not undergone fluctuation reduction processing. Without fluctuation reduction processing, luminance fluctuations due to fluctuations occur in the luminance range between the luminance of the high-luminance area and the luminance of the low-luminance area within the fluctuation displacement range shown in Figure 6(b). An image of the luminance fluctuations due to this fluctuation is expressed by the vertical solid line within the fluctuation displacement range in graph (2).

[0039] Graph (3) shows the luminance values ​​at each pixel position on horizontal line A of the image that has undergone jitter reduction processing. Graph (3) shows that the effect of jitter reduction processing has made the luminance fluctuation due to jitter smaller than that of graph (2). Furthermore, within the range of jitter displacement, the closer to the high-luminance region, the more likely the jitter is to increase toward the high-luminance side, and the closer to the low-luminance region, the more likely the jitter is to decrease toward the low-luminance side. Therefore, the central value (average value) of the luminance fluctuation due to jitter after jitter reduction processing, shown in graph (3), becomes higher the closer to the high-luminance region, and becomes lower the closer to the low-luminance region. Due to the effect of this phenomenon, the image after jitter reduction processing appears blurrier than the image before jitter reduction processing.

[0040] Graph (4) shows the image of jitter when image sharpening is performed to correct the blur of the image after jitter reduction processing. Sharpening emphasizes local image changes, which in turn emphasizes the brightness fluctuations caused by jitter. In other words, the brightness fluctuations caused by jitter shown in graph (4) are larger than the brightness fluctuations caused by jitter shown in graph (3). As a result, the remaining jitter is emphasized, weakening the jitter reduction effect.

[0041] Furthermore, because fluctuations are caused by random changes in the refractive index of the atmosphere, images with fluctuations exhibit random distortion and shaking of the subject, as well as artifacts such as mosquito noise.Since these fluctuations appear as relatively high-frequency image changes, there is a risk that emphasizing high-frequency components through image sharpening processing will also emphasize the fluctuations.

[0042] Next, the effects of this embodiment will be described with reference to Figs. 7 and 8. Fig. 7(a) shows image P having a high-luminance region (region on the left) and a low-luminance region (region on the right). Fig. 7(b) shows image Q having a high-luminance region (region on the left) and a low-luminance region (region on the right). The difference in luminance value between the high-luminance region and the low-luminance region in image P is larger than the difference in luminance value between the high-luminance region and the low-luminance region in image Q. In other words, image P is an image with large changes in luminance in the spatial direction, while image Q is an image with small changes in luminance in the spatial direction.

[0043] Graphs (1) and (2) in Fig. 7(c) show the luminance values ​​at each pixel position on horizontal line A of image P shown in Fig. 7(a). Graphs (3) and (4) in Fig. 7(c) show the luminance values ​​at each pixel position on horizontal line A of image Q shown in Fig. 7(b). The horizontal axis of these graphs indicates the pixel position on horizontal line A, and the vertical axis indicates the luminance value.

[0044] Graph (1) shows the luminance values ​​at each pixel position on horizontal line A of the image obtained by performing jitter reduction processing on image P. Graph (2) shows the luminance values ​​at each pixel position on horizontal line A of the image obtained by enhancing the contrast of the image obtained by performing jitter reduction processing on image P. Graph (3) shows the luminance values ​​at each pixel position on horizontal line A of the image obtained by performing jitter reduction processing on image Q. Graph (4) shows the luminance values ​​at each pixel position on horizontal line A of the image obtained by enhancing the contrast of the image obtained by performing jitter reduction processing on image Q.

[0045] As a method for enhancing contrast, for example, taking the brightness histogram shown in FIG. 8(a) (the horizontal axis represents brightness values, and the vertical axis represents the frequency of brightness values), first, the brightness peaks in the high brightness area and the low brightness area are detected. Then, as shown in FIG. 8(b), the gradation of the input image is converted using a gradation characteristic that allocates more output gradation values ​​to gradations corresponding to the interval between the peaks, thereby obtaining the gradation of the output image. In FIG. 8(b), the horizontal axis represents the gradation of the input image, and the vertical axis represents the gradation of the output image.

[0046] As shown in graph (2), if the spatial luminance change of the image is sufficiently large compared to the luminance change caused by fluctuations, it is possible to enhance the contrast of the image as a whole and achieve a blur reduction effect, even if the luminance change due to fluctuations is large due to contrast enhancement. On the other hand, as shown in graph (4), if the spatial luminance change of the image is small compared to the luminance change caused by fluctuations, even if contrast enhancement is performed, the spatial luminance change of the image is buried in the luminance change caused by fluctuations, making it difficult to achieve a contrast improvement effect. Furthermore, the effect of contrast enhancement emphasizes the fluctuations, increasing the luminance change caused by fluctuations, thereby weakening the effect of the fluctuation reduction process. Therefore, in such cases, it is better not to perform contrast enhancement or to reduce the degree of contrast enhancement.

[0047] That is, when the spatial luminance change of the image is large, the blurring caused by the fluctuation reduction process can be reduced by performing contrast enhancement on the image after fluctuation reduction. On the other hand, when the spatial luminance change of the image is small, the contrast enhancement on the image after fluctuation reduction can be prevented from emphasizing the uncorrected fluctuation. That is, it is possible to reduce the blurring caused by the fluctuation reduction process while preventing the uncorrected fluctuation from being emphasized.

[0048] <Modification> In this embodiment, the correction unit 103 controls the contrast enhancement of the image based on the degree or amount of fluctuation contained in the input image, but it may also control the contrast enhancement of the image based on, for example, the strength of the fluctuation reduction process. The strength of the fluctuation reduction process may be, for example, the number of frames or the smoothing period used in the smoothing process in the time direction (frame direction) performed by the reduction unit 102. Furthermore, the strength of the fluctuation reduction process may be, for example, the strength of the fluctuation reduction process set by the user via a setting unit (not shown), or a value specifying the area in the input image to which the fluctuation reduction process is applied (for example, the size of the area). The correction unit 103 enhances the contrast of the reduced image more as the strength of the fluctuation reduction process is stronger.

[0049] Furthermore, the correction unit 103 determines the magnitude of the "spatial luminance change of the input image" using a luminance histogram, but the method for determining the magnitude of the "spatial luminance change of the input image" is not limited to a specific method.

[0050] For example, the correction unit 103 detects edges in the input image using a general edge detection technique. Then, for each region, such as a peripheral region of a pixel detected as an edge (edge ​​pixel) or a specified image region (e.g., an image region having the same attribute), the correction unit 103 counts the number of edge pixels belonging to the region. If there is a region where the number of counted edge pixels is equal to or greater than a threshold, the correction unit 103 determines that the "spatial luminance change of the input image" is large, and if there is no such region, the correction unit 103 determines that the "spatial luminance change of the input image" is small.

[0051] Furthermore, the correction unit 103 may determine the magnitude of the "spatial luminance change of the input image" based on the edge strength in the input image. For example, the correction unit 103 may determine that the "spatial luminance change of the input image" is large if an index such as the maximum edge strength in the input image or the average edge strength in the input image is equal to or greater than a threshold. On the other hand, the correction unit 103 may determine that the "spatial luminance change of the input image" is small if the index is less than the threshold.

[0052] In addition, in this embodiment, the correction unit 103 emphasizes the contrast of the reduced image more as the amount of fluctuation represented by the fluctuation information increases (the degree of fluctuation increases), but other conditions for controlling the emphasis of the contrast of the reduced image are also possible. For example, the correction unit 103 may emphasize the contrast more as the strength of the fluctuation reduction process increases.

[0053] In addition, the correction unit 103 may weaken the degree of contrast enhancement when the condition that "the amount of fluctuation (degree of fluctuation) represented by the fluctuation information is greater than or equal to a threshold value and the strength of the fluctuation reduction processing is less than a threshold value" is met, compared to the degree of contrast enhancement when the condition is not met.

[0054] It is generally assumed that the amount of jitter contained in an image and the strength of the jitter reduction process are proportional to each other. However, since jitter reduction process causes image blurring of moving subjects (motion blur), it is conceivable to weaken the strength of the jitter reduction process in scenes with many moving subjects, even if the degree of jitter contained in the image is large. If the degree of jitter contained in the image is large and the strength of the jitter reduction process is weak, the jitter is not reduced and remains (residual jitter). Furthermore, if contrast is enhanced when residual jitter exists, the jitter will be emphasized. Therefore, by weakening the degree of contrast enhancement when the degree of jitter contained in the image is large and the strength of the jitter reduction process is weak, it is possible to prevent the jitter from being emphasized.

[0055] In addition, in the present embodiment, the acquiring unit 101 uses the inter-frame difference of the input image when acquiring the degree or amount of fluctuation as fluctuation information, but this is not limiting. For example, the acquiring unit 101 may use the magnitude of the fluctuation displacement shown in Fig. 6 or 7 as the degree (amount) of fluctuation, and acquire information representing the degree (amount) as fluctuation information.

[0056] The greater the magnitude of the fluctuation displacement, the greater the degree of fluctuation. The fluctuation displacement is a quantity that represents the maximum or average extent to which a captured image of a subject moves (appears to move) in image space due to the influence of fluctuation. Alternatively, the number of frames (fluctuation period) required for the cumulative value of inter-frame differences to reach a predetermined value or more may be used as the degree of fluctuation. The shorter the fluctuation period (the fewer the number of frames required for the cumulative value of inter-frame differences to reach a predetermined value or more), the greater the degree of fluctuation.

[0057] Furthermore, in this embodiment, the calculation unit 104 calculates a luminance histogram from the input image, but the luminance histogram is merely one piece of information for acquiring "information required to calculate spatial luminance variations of the input image." In other words, the calculation unit 104 may acquire information other than a luminance histogram as long as such information can be acquired. For example, the calculation unit 104 may calculate, as such information, statistics related to the colors in the input image, or statistics of R, G, and B pixel values ​​in the input image.

[0058] Alternatively, the image processing device may divide the input image into multiple pixel blocks and perform processing for each pixel block according to the flowchart in Figure 2. In this case, the image processing device acquires pixel blocks of the output image corresponding to each pixel block in the input image. In this case, fluctuation information and a luminance histogram are calculated for each pixel block, so the intensity of the fluctuation reduction process and the degree of contrast enhancement / correction gradation characteristics correspond to the fluctuation information of each pixel block. This enables more precise control, thereby enabling better optimization of the fluctuation reduction effect and blur reduction effect.

[0059] [Second embodiment] In each of the following embodiments, including this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that they are the same as the first embodiment. First, an example of the functional configuration of an image processing device according to this embodiment will be described using the block diagram of Fig. 9. In Fig. 9, functional units that are the same as those shown in Fig. 1 are assigned the same reference numerals, and descriptions of these functional units will be omitted.

[0060] The acquisition unit 201 operates in the same manner as the acquisition unit 101, and calculates fluctuation information indicating the amount of fluctuation (degree of fluctuation) contained in the image obtained by the reduction unit 202 performing fluctuation reduction processing on the nth (n is an integer greater than or equal to 2) frame (past frame).

[0061] The reduction unit 202 operates in the same manner as the reduction unit 102, and performs jitter reduction processing on the image of the (n+1)th frame (current frame) output from the correction unit 203 based on the jitter information calculated by the acquisition unit 201 for the nth frame, to generate an output image of the (n+1)th frame. Note that if the amount of jitter (degree of jitter) represented by the jitter information calculated by the acquisition unit 201 for the nth frame is equal to or greater than a threshold, the reduction unit 202 performs jitter reduction processing on the image of the (n+1)th frame with a stronger intensity than the jitter reduction processing performed for the nth frame. Note that if the amount of jitter (degree of jitter) represented by the jitter information calculated by the acquisition unit 201 for the nth frame is less than the threshold, the reduction unit 202 performs jitter reduction processing on the image of the (n+1)th frame with the same intensity as the jitter reduction processing performed for the nth frame. This causes the intensity of jitter reduction to converge. Alternatively, when the reduction unit 202 detects that the amount (degree) of fluctuation has converged to a predetermined value, the reduction unit 202 may converge the intensity of the fluctuation reduction process. Then, the reduction unit 202 outputs the generated output image. The output destination of the output image is not limited to a specific output destination, as in the first embodiment.

[0062] The correction unit 203 operates in the same manner as the correction unit 103, and corrects the contrast of the input image of the (n+1)th frame based on the fluctuation information calculated by the acquisition unit 201 for the nth frame and the luminance histogram calculated by the calculation unit 104 for the (n+1)th frame. Note that the correction unit 203 corrects the contrast when the amount (degree) of fluctuation represented by the fluctuation information calculated by the acquisition unit 201 for the nth frame is less than a threshold value or when it is determined that the amount (degree) of fluctuation has converged.

[0063] In this way, according to this embodiment, the intensity of the jitter reduction process and the degree of contrast enhancement for the next frame can be determined according to the amount (level) of jitter after the jitter reduction process. Therefore, it is possible to prevent the intensity of the jitter reduction process and the degree of contrast enhancement from being excessive or insufficient, and to optimize the effects of jitter reduction and blur reduction.

[0064] [Third embodiment] This embodiment improves the blurring of the image caused by the jitter reduction process by performing a sharpening process on the image after the jitter reduction process, and by performing the sharpening process according to the amount (degree) of jitter, it prevents the uncorrected jitter from being emphasized.

[0065] An example of the functional configuration of the image processing device according to this embodiment will be described with reference to the block diagram of Fig. 10. In Fig. 10, the same functional units as those shown in Fig. 1 are assigned the same reference numerals, and descriptions of these functional units will be omitted.

[0066] The sharpening unit 303 performs sharpening processing on the reduced image obtained by the reduction unit 102 in accordance with the fluctuation information calculated by the acquisition unit 101, thereby generating a "sharpened reduced image." The sharpening unit 303 then outputs the generated "sharpened reduced image" as an output image. As in the first embodiment, the output destination of the output image is not limited to a specific output destination. The sharpening unit 303 will be described in more detail below.

[0067] The extraction unit 3031 extracts high-frequency components from the reduced image. For example, the extraction unit 3031 applies a spatial low-pass filter (LPF) to the reduced image to extract low-frequency components, and obtains the result of subtracting the low-frequency components from the reduced image as the high-frequency components.

[0068] The filter unit 3032 performs filtering on the high-frequency components extracted by the extraction unit 3031 according to the fluctuation information calculated by the acquisition unit 101. The filtering is, for example, processing using a general Gaussian filter. The filter unit 3032 applies a Gaussian filter to the high-frequency components so that the greater the amount of fluctuation represented by the fluctuation information (the stronger the degree of fluctuation) or the stronger the intensity of the fluctuation reduction processing, the greater the blurring of the high-frequency components. The greater the amount (degree) of fluctuation, the more likely it is that the uncorrected fluctuation will be emphasized. Therefore, it is better to blur the high-frequency components as the amount (degree) of fluctuation increases. Furthermore, image fluctuations (fluctuation components) caused by fluctuations contained in high-frequency components tend to appear as particularly high-frequency signals among the high-frequency components. Furthermore, the greater the amount (degree) of fluctuation, the greater the image fluctuations (fluctuation components) caused by fluctuations contained in high-frequency components. Therefore, by blurring the high frequency components to a greater extent as the amount (degree) of fluctuation increases, the fluctuation components contained in the high frequency components can be removed or reduced. In this way, by processing the high frequency components based on the fluctuation information, the emphasis of the uncorrected fluctuation can be suppressed.

[0069] The multiplication unit 3033 multiplies the high-frequency components that have been subjected to the filtering process by a gain corresponding to the fluctuation information calculated by the acquisition unit 101. The multiplication unit 3033 increases the gain as the amount of fluctuation represented by the fluctuation information increases (the degree of fluctuation increases) or as the intensity of the fluctuation reduction process increases. The greater the amount (degree) of fluctuation or the greater the intensity of the fluctuation reduction process, the blurrier the image becomes after the fluctuation reduction process. Therefore, by increasing the gain and increasing the intensity of the high-frequency components, it is possible to more easily achieve a blur reduction effect. The addition unit 3034 generates an output image by adding the high-frequency components multiplied by the gain by the multiplication unit 3033 to the reduced image, and outputs the generated output image.

[0070] The effects of this embodiment will be described below with reference to Fig. 11. Graphs (1) to (4) shown in Fig. 11 show the luminance values ​​at each pixel position in a portion of an image (for example, the above-mentioned horizontal line A). The horizontal axis of these graphs indicates the pixel position in the portion, and the vertical axis indicates the luminance value.

[0071] Graph (1) shows the brightness values ​​at each pixel position of a part of an image without fluctuation. Graph (2) shows an image of brightness fluctuations due to fluctuations. Compared to graph (1), the edges have become blurred due to the fluctuation reduction process, and brightness fluctuations have occurred due to fluctuations.

[0072] The graph on the left of (3) shows the high-frequency components extracted from the image in (2) in a conventional example where the emphasis of residual fluctuations is not suppressed. Because fluctuations appear in the image as spatial high-frequency fluctuations, the extracted high-frequency components contain brightness fluctuations due to the fluctuations.

[0073] The graph on the left of (4) shows the result of sharpening the image of graph (2) in a conventional example that does not suppress the emphasis of remaining fluctuation. That is, the graph on the left of (4) is the result of adding the graphs (2) and (3). As shown in the graph on the left of (4), conventional sharpening processes perform sharpening without considering the degree of fluctuation, so that the fluctuation is also emphasized along with the sharpening of the image, resulting in a weakened effect of the fluctuation reduction process.

[0074] The graph on the right of (3) shows the high-frequency components extracted from the image of graph (2) in this embodiment, which suppresses the emphasis of residual fluctuations. By applying a Gaussian filter or the like to the high-frequency components shown in the graph on the left of (3), the high-frequency components shown in the graph on the right of (3) do not contain brightness fluctuations due to fluctuations.

[0075] The graph on the right of (4) shows the results of sharpening the image of graph (2) in this embodiment, which suppresses the emphasis of residual fluctuation. That is, the graph on the right of (4) is the result of adding the graphs on the right of (2) and (3). As shown in the graph on the right of (4), the sharpening process of this embodiment removes or reduces fluctuation components from high-frequency components before adding them to the original signal. Therefore, sharpening can be performed while preventing the emphasis of residual fluctuation. That is, blurring caused by the fluctuation reduction process (image averaging in the frame direction) can be reduced while preventing the emphasis of residual fluctuation correction.

[0076] <Modification> The filter unit 3032 may remove low-amplitude signals of high-frequency components. That is, the filter unit 2032 may replace high-frequency components whose absolute values ​​are less than a threshold value with 0. This makes it possible to sharpen portions with large fluctuations in brightness, such as the edges of an object, while preventing the enhancement of fluctuation components with relatively small fluctuations in brightness, thereby improving blurring caused by reduced fluctuations.

[0077] Furthermore, the sharpening unit 303 may perform the above-mentioned sharpening process only when it is determined that there is a large change in luminance in the spatial direction of the image, or may perform the sharpening process only on edge portions (strong edge portions) having edge strength equal to or greater than a threshold value.

[0078] When the spatial luminance variation of an image is small, for example, when high-frequency components are extracted as shown in the left graph of (3), the difference between the high-frequency components resulting from spatial luminance variation, such as edges, and the high-frequency components resulting from luminance fluctuations due to fluctuations becomes small. This makes it difficult to emphasize only the high-frequency components resulting from spatial luminance variation, which may result in the uncorrected fluctuations being emphasized or the blurring caused by the fluctuation reduction process not being reduced. By performing sharpening processing only when the spatial luminance variation of an image is large or by performing sharpening processing only on strong edge portions, it becomes easier to separate the high-frequency components resulting from spatial luminance variation, such as edges, from the high-frequency components resulting from luminance fluctuations due to fluctuations. As a result, blurring caused by the fluctuation reduction process can be reduced in areas with large luminance variation, such as the edges of a subject, while suppressing the emphasis on the uncorrected fluctuations in areas with small luminance variation, thereby preventing a weakening of the fluctuation reduction effect.

[0079] [Fourth embodiment] In this embodiment, the compression rate of the input image is corrected according to index values ​​such as the amount of fluctuation, the degree of fluctuation, and the strength of the fluctuation reduction process. When the index value is large, the compression rate of the input image is reduced to retain the high-frequency components of the input image and prevent blurring of the input image.

[0080] An example of the functional configuration of the image processing device according to this embodiment will be described with reference to the block diagram of Fig. 12. In Fig. 12, the same functional units as those shown in Fig. 1 are assigned the same reference numerals, and descriptions of these functional units will be omitted.

[0081] The correction unit 403 determines the compression rate according to the fluctuation information calculated by the acquisition unit 101, performs compression encoding processing according to the determined compression rate on the reduced image generated by the reduction unit 102, and outputs the compressed and encoded reduced image as an output image. The output destination of the output image is not limited to a specific output destination, as in the first embodiment.

[0082] For example, the larger the index value represented by the fluctuation information (the larger the amount of fluctuation, the stronger the degree of fluctuation, the stronger the intensity of the fluctuation reduction process), the lower the compression rate the correction unit 403. On the other hand, the smaller the index value (the smaller the amount of fluctuation, the weaker the degree of fluctuation, the weaker the intensity of the fluctuation reduction process), the higher the compression rate the correction unit 403. For example, the larger the index value, the more the correction unit 403 sets the quantization table used for compression encoding so as to retain high-frequency components in the input image.

[0083] By lowering the compression rate when the jitter reduction process is strong, the high frequency components of the image can be retained, thereby reducing the blurring of the image caused by the jitter reduction process. Furthermore, when the jitter reduction process is strong, the correlation between frames increases, which increases the coding compression efficiency of the moving image, so even if the compression rate is lowered, the increase in the data volume of the moving image can be suppressed.

[0084] Furthermore, because fluctuations tend to appear as high-frequency components in an image, reducing the high-frequency components by determining a compression rate that increases as the fluctuation level increases makes the fluctuation less noticeable. Furthermore, in this case, there is no need to apply strong fluctuation reduction processing, so blurring caused by fluctuation reduction processing can be prevented.

[0085] [Fifth embodiment] The functional units shown in Figures 1, 9, 10, and 12 may be implemented as software (computer programs) or hardware. In the former case, a computer device capable of executing the software is applicable to the image processing devices of the above-described embodiments and modifications. An example of the hardware configuration of such a computer device will be described using the block diagram of Figure 13.

[0086] The CPU 1301 executes various processes using computer programs and data stored in the RAM 1302. As a result, the CPU 1301 controls the operation of the entire computer device, and also executes or controls various processes described as processes performed by the image processing devices of the above-described embodiments and modifications.

[0087] The RAM 1302 has an area for storing computer programs and data loaded from the ROM 1303 or the storage device 1306, and an area for storing computer programs and data received from the outside via the I / F 1307. The RAM 1302 also has a work area used by the CPU 1301 when executing various processes. In this way, the RAM 1302 can provide various areas as needed.

[0088] The ROM 1303 stores setting data for the computer device, computer programs and data relating to the startup of the computer device, computer programs and data relating to the basic operation of the computer device, and the like.

[0089] The operation unit 1304 is a user interface such as a keyboard, a mouse, a touch panel screen, etc., and allows the user to input various instructions and information to the computer device by operating it.

[0090] The display unit 1305 has a liquid crystal screen or a touch panel screen, and can display the processing results of the CPU 1301 as images, text, etc. The display unit 1305 may be a projection device such as a projector that projects images and text.

[0091] The storage device 1306 is a large-capacity information storage device such as a hard disk drive, etc. The storage device 1306 stores an OS, computer programs and data for causing the CPU 1301 to execute or control the various processes described as processes performed by the image processing device of each of the above embodiments and modifications.

[0092] The I / F 1307 is a communication interface for performing data communication with an external device via a network such as a LAN or the Internet. For example, the computer device can acquire an input image from an external imaging device or server device via the I / F 1307.

[0093] The CPU 1301, RAM 1302, ROM 1303, operation unit 1304, display unit 1305, storage device 1306, and I / F 1307 are all connected to a system bus 1308. Note that the configuration shown in Fig. 13 is merely an example of the hardware configuration of a computer device capable of executing the various processes described as processes performed by the image processing devices of the above-mentioned embodiments and modifications, and can be modified / altered as appropriate.

[0094] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in the above-mentioned embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0095] Furthermore, some or all of the above-described embodiments and modifications may be used in appropriate combination, and some or all of the above-described embodiments and modifications may be used selectively.

[0096] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) a reduction means for reducing fluctuation in an image based on fluctuation information; a correction means for correcting the contrast of the image based on the fluctuation information and a spatial luminance change in the image; An image processing device comprising: (Item 2) The image processing device according to item 1, characterized in that the correction means calculates a histogram of brightness values ​​in the image, detects brightness value peaks in the histogram, calculates the distance between the peaks as the brightness change, and the greater the distance, the more the contrast of the image is emphasized. (Item 3) 3. The image processing device according to item 2, wherein the correction means does not perform the enhancement when the distance is less than a threshold value. (Item 4) The image processing device described in item 1, characterized in that the correction means determines whether the luminance change is large based on the number of edge pixels in the image or the edge strength in the image, and performs the correction if it is determined that the luminance change is large, and does not perform the correction if it is determined that the luminance change is small. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the correction means enhances the contrast of the image more as the amount of fluctuation represented by the fluctuation information increases. (Item 6) 6. The image processing device according to item 5, wherein the correction means does not perform the enhancement when the amount of fluctuation represented by the fluctuation information is less than a threshold value. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the correction means emphasizes the contrast of the image more as the intensity of the reduction increases. (Item 8) The image processing device described in any one of items 1 to 7, characterized in that the correction means weakens the degree of contrast enhancement when the condition that the amount of fluctuation represented by the fluctuation information is equal to or greater than a threshold and the intensity of the reduction is less than a threshold is met, compared to the degree of contrast enhancement when the condition is not met. (Item 9) the reduction means reduces fluctuation in an image of a current frame based on fluctuation information of a past frame; The correction means corrects the image of the current frame based on fluctuation information of past frames and luminance changes in the spatial direction in the image of the current frame. 2. The image processing device according to item 1, (Item 10) Item 10. The image processing device according to item 9, wherein the reduction means reduces the fluctuation in the image of the current frame with a strength greater than the strength with which the fluctuation in the image of the past frame is reduced if the amount of fluctuation represented by the fluctuation information of the past frame is equal to or greater than a threshold value. (Item 11) Item 11. The image processing device according to item 9 or 10, characterized in that, if the amount of fluctuation represented by the fluctuation information of a past frame is less than a threshold, the reduction means reduces the fluctuation in the image of the current frame with the same intensity as the intensity with which the fluctuation in the image of the past frame is reduced. (Item 12) a reduction means for reducing fluctuation in an image based on fluctuation information; a sharpening means for processing high-frequency components in a reduced image generated by the reduction based on the fluctuation information, and generating an image by adding the processed high-frequency components to the reduced image; An image processing device comprising: (Item 13) The image processing device described in item 12 is characterized in that the sharpening means performs a filter process that blurs high-frequency components the greater the amount of fluctuation represented by the fluctuation information, and multiplies the high-frequency components that have been subjected to the filter process by a gain that increases the greater the amount of fluctuation represented by the fluctuation information. (Item 14) 14. The image processing device according to item 12 or 13, wherein the sharpening means removes low amplitude signals of high frequency components. (Item 15) 15. The image processing device according to any one of items 12 to 14, wherein the sharpening means performs the processing and the generation when it is determined that there is a large change in luminance in the spatial direction in the image. (Item 16) 16. The image processing device according to any one of items 12 to 15, wherein the sharpening means performs the processing and the generation on an edge portion having an edge strength equal to or greater than a threshold value. (Item 17) a reduction means for reducing fluctuation in an image based on fluctuation information; a compression encoding means for compressing and encoding an image at a compression rate according to the fluctuation information; An image processing device comprising: (Item 18) Item 18. The image processing device according to item 17, wherein the compression encoding means reduces the compression rate as the amount of fluctuation represented by the fluctuation information increases. (Item 19) Item 18. The image processing device according to item 17, wherein the compression encoding means reduces the compression rate as the degree of fluctuation represented by the fluctuation information increases. (Item 20) Item 18. The image processing device according to item 17, wherein the compression encoding means decreases the compression rate as the strength of the reduction increases. (Item 21) 21. The image processing device according to any one of items 1 to 20, wherein the image is an input image or each pixel block obtained by dividing the input image. (Item 22) moreover, 22. The image processing device according to any one of items 1 to 21, comprising an imaging means. (Item 23) An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a correction step in which a correction means of the image processing device corrects the contrast of the image based on the fluctuation information and a spatial luminance change in the image; An image processing method comprising: (Item 24) An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a sharpening step in which a sharpening means of the image processing device processes high-frequency components in the reduced image generated by the reduction based on the fluctuation information, and generates an image in which the processed high-frequency components are added to the reduced image; An image processing method comprising: (Item 25) An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a compression encoding step in which a compression encoding means of the image processing device compresses and encodes an image at a compression rate according to the fluctuation information; An image processing method comprising: (Item 26) 22. A computer program for causing a computer to function as each means of the image processing device according to any one of items 1 to 21.

[0098] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0099] 101: Acquisition unit 102: Reduction unit 103: Correction unit 104: Calculation unit

Claims

1. a reduction means for reducing fluctuation in an image based on fluctuation information; a correction means for correcting the contrast of the image based on the fluctuation information and a spatial luminance change in the image; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the correction means calculates a histogram of brightness values ​​in the image, detects brightness value peaks in the histogram, calculates the distance between the peaks as the brightness change, and the greater the distance, the more the contrast of the image is enhanced.

3. 3. The image processing apparatus according to claim 2, wherein the correction means does not perform the enhancement when the distance is less than a threshold value.

4. 2. The image processing device according to claim 1, wherein the correction means determines whether the luminance change is large based on the number of edge pixels in the image or the edge strength in the image, and performs the correction if it determines that the luminance change is large, and does not perform the correction if it determines that the luminance change is small.

5. 2. The image processing apparatus according to claim 1, wherein the correction means enhances the contrast of the image more as the amount of fluctuation represented by the fluctuation information increases.

6. 6. The image processing apparatus according to claim 5, wherein the correction means does not perform the enhancement when the amount of fluctuation represented by the fluctuation information is less than a threshold value.

7. 2. The image processing apparatus according to claim 1, wherein the correction means enhances the contrast of the image more as the intensity of the reduction increases.

8. The image processing device described in claim 1, characterized in that the correction means weakens the degree of contrast enhancement when the condition that the amount of fluctuation represented by the fluctuation information is greater than or equal to a threshold and the strength of the reduction is less than a threshold is met, compared to the degree of contrast enhancement when the condition is not met.

9. the reduction means reduces fluctuation in an image of a current frame based on fluctuation information of a past frame; The correction means corrects the image of the current frame based on fluctuation information of past frames and luminance changes in the spatial direction in the image of the current frame.

2. The image processing device according to claim 1, wherein:

10. The image processing device according to claim 9, characterized in that the reduction means reduces the fluctuation in the image of the current frame with a strength greater than the strength with which the fluctuation in the image of the past frame is reduced if the amount of fluctuation represented by the fluctuation information of the past frame is equal to or greater than a threshold value.

11. 10. The image processing device according to claim 9, wherein the reduction means reduces the fluctuation in the image of the current frame with the same intensity as that used to reduce the fluctuation in the image of the previous frame if the amount of fluctuation represented by the fluctuation information of the previous frame is less than a threshold value.

12. a reduction means for reducing fluctuation in an image based on fluctuation information; a sharpening means for processing high-frequency components in a reduced image generated by the reduction based on the fluctuation information, and generating an image by adding the processed high-frequency components to the reduced image; An image processing device comprising:

13. The image processing device according to claim 12, characterized in that the sharpening means performs a filter process that blurs high-frequency components the greater the amount of fluctuation represented by the fluctuation information, and multiplies the high-frequency components that have been subjected to the filter process by a gain that increases the greater the amount of fluctuation represented by the fluctuation information.

14. 13. The image processing apparatus according to claim 12, wherein said sharpening means removes low-amplitude signals of high-frequency components.

15. 13. The image processing apparatus according to claim 12, wherein the sharpening means performs the processing and the generation when it is determined that there is a large change in luminance in the spatial direction in the image.

16. 13. The image processing apparatus according to claim 12, wherein the sharpening means performs the processing and the generation on an edge portion having an edge strength equal to or greater than a threshold value.

17. a reduction means for reducing fluctuation in an image based on fluctuation information; a compression encoding means for compressing and encoding an image at a compression rate according to the fluctuation information; An image processing device comprising:

18. 18. The image processing apparatus according to claim 17, wherein the compression encoding means reduces the compression rate as the amount of fluctuation represented by the fluctuation information increases.

19. 18. The image processing apparatus according to claim 17, wherein said compression encoding means reduces the compression rate as the degree of fluctuation indicated by said fluctuation information increases.

20. 18. The image processing apparatus according to claim 17, wherein the compression encoding means decreases the compression rate as the strength of the reduction increases.

21. 2. The image processing apparatus according to claim 1, wherein the image is an input image or each pixel block obtained by dividing the input image.

22. moreover, 2. The image processing device according to claim 1, further comprising an imaging means.

23. An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a correction step in which a correction means of the image processing device corrects the contrast of the image based on the fluctuation information and a spatial luminance change in the image; An image processing method comprising:

24. An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a sharpening step in which a sharpening means of the image processing device processes high-frequency components in the reduced image generated by the reduction based on the fluctuation information, and generates an image in which the processed high-frequency components are added to the reduced image; An image processing method comprising:

25. An image processing method performed by an image processing device, a reduction step in which a reduction means of the image processing device reduces fluctuation in the image based on fluctuation information; a compression encoding step in which a compression encoding means of the image processing device compresses and encodes an image at a compression rate according to the fluctuation information; An image processing method comprising:

26. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 21.

Images