Image processing apparatus, image processing method, program, and storage medium

Abstract

To provide a technique to correct fluctuations with a correction strength considering the degree of influence of the fluctuations for every area.SOLUTION: An image processing apparatus has: dividing means that divides an input image into a plurality of areas; acquisition means that acquires information on a subject distance for each of the plurality of areas; correction means that performs correction processing for correcting fluctuations in the input image for each of the plurality of areas; and determination means that determines the correction strength of the correction processing for each of the plurality of areas on the basis of the information on the subject distance for each of the plurality of areas. Of the plurality of areas, at least two areas different in the information on the subject distance have a difference in the determined correction strength.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a storage medium. [Background technology]

[0002] In use cases of surveillance cameras, such as port surveillance and infrastructure monitoring, it is known that when taking telephoto shots of ships or aircraft, for example, fluctuations in the subject image caused by uneven changes in the refractive index of the atmosphere (such as heat haze) can reduce the visibility of the subject.

[0003] Conventionally, a method of smoothing an image in the time direction has been known as a method of reducing fluctuations because changes in pixel values ​​that occur when a subject that includes fluctuations is photographed are approximated to a normal distribution based on a predetermined position. However, this method of smoothing an image in the time direction has the drawback of causing a blurring phenomenon of a moving subject (hereinafter referred to as motion blur) when the smoothing time is extended.

[0004] Furthermore, since fluctuations change depending on the environment, time, etc., when smoothing processing in the time direction is used to reduce fluctuations, it is necessary to change the strength of the smoothing processing in the time direction depending on the degree of fluctuations.

[0005] In response to this, for example, Patent Document 1 discloses a technique in which pixels at the same position in an input image and a correction image are grouped by luminance, and a time-domain smoothing process is applied to each group. The technique disclosed in Patent Document 1 makes it possible to appropriately correct the fluctuations according to the degree of the fluctuations while suppressing the adverse effects of time-domain smoothing. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2015 / 040731 Summary of the Invention [Problem to be solved by the invention]

[0007] The problem to be solved by the present invention is to provide a technique for correcting fluctuations with a correction strength that takes into account the degree of influence of fluctuations in each region. [Means for solving the problem]

[0008] In order to solve the above problem, an image processing device according to one aspect of the present invention comprises a division means for dividing an input image into a plurality of regions, an acquisition means for acquiring information relating to subject distance for each of the plurality of regions, a correction means for performing a correction process for correcting fluctuations in the input image for each of the plurality of regions, and a determination means for determining a correction intensity of the correction process for each of the plurality of regions based on the information relating to the subject distance for each of the plurality of regions, and is characterized in that the determined correction intensity differs in at least two of the plurality of regions for which the information relating to the subject distance differs. [Effects of the Invention]

[0009] According to the present invention, fluctuations can be corrected with a correction strength that takes into account the degree of influence of fluctuations in each region. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus. [Figure 2] FIG. 1 is a block diagram showing the functional configuration of an image processing apparatus. [Figure 3] FIG. 10 is a diagram showing the influence of fluctuations in an input image. [Figure 4] FIG. 10 is a diagram showing an example of fluctuation amount. [Figure 5] FIG. 10 is a diagram showing an example of fluctuation correction. [Figure 6] 10A and 10B are diagrams showing the relationship between correction strength and motion blur. [Figure 7] 10 is a flowchart showing an example of a correction intensity determination process. [Figure 8] FIG. [Figure 9] 10 is a flowchart showing an example of a correction intensity determination process. [Figure 10] FIG. 10 is a diagram illustrating fluctuation information for each region. [Figure 11] 10 is a flowchart showing an example of a correction intensity determination process. [Figure 12] FIG. 4 is a diagram illustrating moving object information for each region. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] First Embodiment First, an example of the hardware configuration of an image processing device according to this embodiment will be described using the block diagram of Fig. 1. Note that Fig. 1 shows the main components related to the following description, but does not show all of the components of the image processing device. Furthermore, the hardware configuration shown in Fig. 1 is merely an example of a hardware configuration applicable to the image processing device according to this embodiment, and can be modified / altered as appropriate.

[0013] (Device configuration) The image processing device in this embodiment has an imaging optical system 101, an imaging element 102, a CPU 103, a RAM 104, and a ROM 105, and each part is electrically connected via a bus 106. Light incident on the imaging optical system 101 is focused as a subject image, and the subject image is captured by the imaging element 102.

[0014] The imaging optical system 101 is composed of a lens group consisting of one or more lenses. Examples of such lenses include a zoom lens whose focal length can be adjusted by moving in the optical axis direction. Other lenses that can be used include a focus lens whose focus can be adjusted by moving in the optical axis direction, and a teleconverter (extender) whose optical characteristics can be changed. Note that, in this embodiment, an example is described in which the image processing device and the imaging optical system 101 are integrated, but the present invention is not limited to this. For example, the imaging optical system 101 may be configured to be provided outside the image processing device, such as an interchangeable lens, as long as it is connected to the image processing device and is capable of communicating with it.

[0015] The image sensor 102 is a photoelectric conversion element that captures an image of a subject through the imaging optical system 101 and generates image data or an image signal consisting of multiple pixels. The image data or image signal contains information on multiple colors. The multiple colors are, for example, red, green, and blue. The image data passes through color filters corresponding to each color provided in front of the image sensor 102 and is converted into electrical signals of red, green, and blue signals by the image sensor 102 and processed. The color filters transmit not only visible light components corresponding to red, green, and blue, but also some infrared light components included in the invisible light range. The image sensor 102 may be a complementary metal oxide semiconductor (CMOS), a charge-coupled device (CCD), a single photon avalanche diode (SPAD), or the like. The image sensor 102 may be provided externally to the image processing device as imaging means for obtaining an input image for the image processing device. In this case, the imaging means outputs the image to the image processing device.

[0016] The CPU 103 is a central processing unit that performs overall control of the image processing device. The CPU 103 executes various processes using computer programs and data loaded into the RAM 104. As a result, the CPU 103 controls the operation of the entire image processing device, and also executes or controls various processes that will be described as processes performed by the image processing device.

[0017] The RAM 104 is a non-volatile memory that provides a work area used by the CPU 103 when it executes processing. Computer programs and data loaded from the ROM 105 are stored in the RAM 104. The RAM 104 also functions as a frame memory and a buffer memory. The ROM 105 also stores computer programs and data that cause the CPU 103 to execute or control various processes that will be described as processes performed by the image processing device.

[0018] In addition, the image processing device may have a processor such as a GPU or an image processing circuit, in which case some or all of the various processes described as processes performed by CPU 103 may be performed by the processor or the image processing circuit.

[0019] (Functional configuration) Next, an example of the functional configuration of the image processing device according to this embodiment for correcting fluctuations in an input image will be described with reference to the block diagram of FIG. 2. In this embodiment, a case will be described in which each functional unit shown in FIG. 2 is implemented by software (computer program). In the following, the functional units of FIG. 2 will sometimes be described as the subject of processing, but in reality, the functions of the functional units are realized by the CPU 103 executing a computer program corresponding to the functional units. Note that one or more of the functional units shown in FIG. 2 may be implemented by hardware.

[0020] A RAW image of each frame is output from the image sensor 102. When a moving image is captured by the image sensor 102, the RAW image of each frame corresponds to an image of each frame in the moving image, and when still images are captured periodically or irregularly by the image sensor 102, the RAW image of each frame corresponds to the still image.

[0021] The acquisition unit 111 acquires, as an input image, a captured image generated by the CPU 103. Then, the acquisition unit 111 acquires, as fluctuation information, the amount of fluctuation corresponding to each pixel of the acquired input image.

[0022] Here, the influence of fluctuation on an input image will be explained using Fig. 3. Fig. 3(a) shows an example of a captured image (non-fluctuation captured image) obtained by capturing an image of a stationary subject in a state without fluctuation. Fig. 3(b) shows an example of a captured image (fluctuation captured image) obtained by capturing an image of a stationary subject in a state with fluctuation.

[0023] As shown in Figures 3(a) and 3(b), even when capturing an image of a stationary subject, if the image is captured in a state where there is shaking, the captured image will be distorted. Figure 3(c) shows the pixel value (solid line) at pixel position P of the image captured without shaking for each frame, and the pixel value (dotted line) at pixel position P of the image captured with shaking for each frame. In Figure 3(c), the horizontal axis represents time (frame), and the vertical axis represents pixel value.

[0024] As shown in Fig. 3(c), the pixel value at pixel position P in the non-jitter captured image of each frame is almost constant, while the pixel value at pixel position P in the jitter captured image of each frame changes. Therefore, when there is jitter, a phenomenon occurs in which an actually stationary subject is captured as if it were a moving subject.

[0025] Next, a method by which the acquisition unit 111 acquires fluctuation information for each pixel position in an input image will be described with reference to Fig. 4. Fig. 4 shows pixel values ​​at the same pixel position Q in the input image of each frame, with the horizontal axis representing time (frame) and the vertical axis representing pixel value. The change in pixel value (amount of fluctuation) shown in Fig. 4(b) is larger than the change in pixel value shown in Fig. 4(a), and the change in pixel value (amount of fluctuation) shown in Fig. 4(c) is larger than the change in pixel value shown in Fig. 4(b).

[0026] Here, the frame corresponding to time t2 is assumed to be the current frame, and the frame corresponding to time t1 is assumed to be a past frame that is at least one frame before the current frame. In this case, the acquisition unit 111 calculates the difference between the pixel value at pixel position Q in the input image of the frame corresponding to time t2 and the pixel value at pixel position Q in the input image of the frame corresponding to time t1. The calculated difference is then acquired as fluctuation information at pixel position Q in the input image of the frame corresponding to time t2. There are various methods for calculating the difference between one pixel value and another pixel value (the difference between pixel values ​​between captured images), and the method is not limited to a specific method. For example, the acquisition unit 111 may calculate the absolute value of the difference between one pixel value and the other pixel value as the difference between the one pixel value and the other pixel value. Alternatively, the acquisition unit 111 may calculate the square of the difference between the one pixel value and the other pixel value as the difference between the one pixel value and the other pixel value.

[0027] In this way, the acquisition unit 111 acquires fluctuation information for each pixel position in the input image of the current frame. Specifically, the acquisition unit 111 acquires the difference between the pixel value of that pixel position in the input image and the pixel value of that pixel position in a frame earlier than the current frame (a frame earlier than the current frame by one or more frames). The acquired difference is used as fluctuation information for that pixel position in the current frame.

[0028] In other words, information about fluctuations in the input image is obtained based on the difference between the pixel values ​​of the input image and the pixel values ​​of at least one image input before or after the input image.

[0029] The acquisition unit 112 acquires current focal length information from the imaging optical system 101. The focal length information is information about the focal length, such as information about the angle of view. For example, the acquisition unit 112 acquires information about the current position of the zoom lens as focal length information from an encoder that detects the position of the zoom lens included in the imaging optical system 101. Note that if the image processing device has a functional unit that changes the angle of view, such as a teleconverter or digital zoom, or is capable of communicating with such a functional unit, the acquisition unit 112 may acquire the current focal length information from such a functional unit.

[0030] The acquisition unit 113 acquires current subject distance information from the imaging optical system 101. The subject distance information is information relating to the distance from the imaging optical system 101 to the subject (subject distance), and for example, the acquisition unit 113 acquires the subject distance information from an "encoder that detects the position of the focus lens" included in a lens driving unit of the imaging optical system 101. In this embodiment, the subject distance information is assumed to be information indicating the position of the focus lens when in focus on a main subject (hereinafter referred to as the main subject).

[0031] The method for acquiring subject distance information is not limited to a specific method. For example, the acquisition unit 113 may use a LiDAR (Light Detection and Ranging) method, which calculates distance from the time it takes for a sensor to receive reflected light from an object and the phase of the reflected light. Alternatively, if the image processing device is equipped with a light source and a sensor, a TOF (Time of Flight) method may be used, which calculates subject distance from the time it takes for light from the light source to reflect off an object and return to the sensor and the speed of light. Furthermore, by using the above method, it is possible to create a distance map by calculating subject distance information for each region in the input image. While this embodiment will be described assuming that subject distance information is acquired using the above method, a defocus map showing the distribution of defocus amounts may also be used.

[0032] The determination unit 114 determines the strength of the correction process for correcting fluctuations in the input image. Details of the operation of the determination unit 114 will be described later.

[0033] The correction unit 115 performs correction processing with the strength determined by the determination unit 114 on the input image and outputs the fluctuation-corrected input image as an output image. The output destination of the output image is not limited to a specific output destination. For example, the correction unit 115 may display the output image on a display unit (not shown) of the image processing device, store the output image in a memory (not shown) of the image processing device, or transmit the output image to an external device via a network interface (not shown) of the image processing device. An example of fluctuation correction by the correction unit 115 will be described with reference to FIG. 5.

[0034] 5(a) shows the pixel values ​​at the same pixel position P in the fluctuation captured image of each frame, where the horizontal axis represents the frame (time) and the vertical axis represents the pixel value. For fluctuation correction, a smoothing process such as a simple moving average or a weighted moving average is used to smooth the input images of multiple frames to generate an image of the current frame.

[0035] Fig. 5(b) shows the change in pixel value at pixel position P of an image obtained by performing smoothing processing, which is fluctuation correction at a first correction intensity, on the input image (fluctuation-captured image) of Fig. 5(a). Fig. 5(c) shows the change in pixel value at pixel position P of an image obtained by performing smoothing processing, which is fluctuation correction at a second correction intensity (>first correction intensity), on the input image (fluctuation-captured image) of Fig. 5(a). The change in pixel value due to fluctuation is approximated to a normal distribution based on a predetermined position, and can be reduced by smoothing processing in the frame direction.

[0036] The number of frames used in the smoothing process may be changed according to the correction strength. For example, the stronger the correction strength, the more frames the correction unit 115 uses in the smoothing process, and the weaker the correction strength, the more frames the correction unit 115 uses in the smoothing process. The correction unit 115 then performs the smoothing process using the input image with the number of frames adjusted (changed) in this way to generate an image of the current frame.

[0037] The correction unit 115 may also change the weighting used in the smoothing process according to the correction strength. For example, the stronger the correction strength, the greater the weighting value the correction unit 115 increases, and the weaker the correction strength, the greater the weighting value the correction unit 115 decreases. The correction unit 115 then performs the smoothing process using the weighting value adjusted (changed) in this way to generate an image of the current frame.

[0038] As described above, the parameters in the smoothing process that are changed according to the correction strength are not limited to a specific form. However, if the correction strength of the frame-wise smoothing process is increased, the change in pixel values ​​in the frame direction is reduced, but motion blur may occur if the input image contains a moving subject. FIG. 6 shows the relationship between correction strength and fluctuation. The curve 601 indicates the change in pixel values ​​in the frame direction, and the straight line 602 indicates the degree of motion blur. As shown in FIG. 6, increasing the correction strength makes it possible to correct large fluctuations, but the degree of motion blur also increases accordingly, which may result in reduced visibility depending on the subject.

[0039] The division unit 116 divides the input image into multiple regions according to a predetermined number of horizontal divisions and vertical divisions. The functions of the aforementioned acquisition units 111, 112, 113, determination unit 114, and correction unit 115 can all be executed for each of the multiple divided regions. In other words, it is possible to acquire fluctuation information, focal length information, and subject distance information for each of the multiple divided regions. In addition, the correction strength of the correction process can also be determined for each of the multiple divided regions, and correction processes can be performed on the input image to correct fluctuations with different correction strengths.

[0040] (Operation description) The operation of the determination unit 114 will be described below with reference to the flowchart in Fig. 7. In the jitter correction performed by the correction unit 115, the amount of jitter can be reduced by increasing the correction strength, but this has the drawback of increasing blur caused by moving objects. In other words, if the correction strength of jitter correction is increased more than necessary, it may actually result in a decrease in visibility. In light of this, in order to reduce the amount of jitter while minimizing blur caused by moving objects, it is necessary to detect the amount of jitter with higher accuracy and determine the correction strength.

[0041] In step S11, the division unit 116 divides the input image into a plurality of regions by setting a mesh frame of a predetermined size for the image. When dividing the image into m horizontal divisions and n vertical divisions, m×n mesh frames are set. FIG. 8(b) shows an example of setting a mesh frame for the image of FIG. 8(a), where a mesh frame with 20 horizontal divisions and 8 vertical divisions is set. When the division of the input image is complete, the division unit 116 proceeds to step S12.

[0042] In step S12, the acquisition unit 113 acquires subject distance information (distance map), which is information related to the subject distance for each of the multiple regions (mesh frames) divided by the division unit 116. FIG. 8(c) is an example of the acquired subject distance information. The closer the distance of the subject existing within each mesh frame, the smaller the value is represented, and the farther the distance, the larger the value is represented. When the acquisition of the subject distance information for each mesh frame is complete, the acquisition unit 113 proceeds to step S13.

[0043] In step S13, the division unit 116 divides the image into regions by grouping multiple mesh frames based on the subject distance information. Region division can be performed by regarding mesh frames with the same subject distance information as belonging to the same group. In practice, subject distance information may contain errors depending on the accuracy of the acquired subject distance information. Therefore, a predetermined tolerance may be set, and mesh frames with matching subject distance information within the tolerance may be grouped together. In other words, regions with approximately the same subject distance information are grouped together. The result of dividing the image into regions by grouping mesh frames in this way is shown in Figure 8(d). Regions 81 to 86 are the divided regions. If the upper left corner is the origin, the xth mesh frame in the horizontal direction and the yth mesh frame in the vertical direction is expressed as (x, y). In this case, for example, region 82 is a region consisting of ten mesh frames: (10,1), (11,1), (9,2), (10,2), (11,2), (12,2), (10,3), (11,3), (10,4), and (11,4). The dividing unit 116 stores the position of the mesh frame for each region as coordinate information. Note that, here, mesh frames with matching subject distance information are considered to be the same region; however, there are various conditions for region division. It is sufficient that the image is divided into at least two regions, such as two regions for close distance and long distance, depending on whether the subject distance is equal to or greater than a predetermined threshold. When the dividing unit 116 has completed the region division of the image, it proceeds to step S14.

[0044] In step S14, the dividing unit 116 adds distance information to each region. The distance information indicates the distance (near to far) to the subject within the region, and is calculated from the average, maximum, minimum, median, or variance of the subject distance information obtained for each mesh frame. The dividing unit 116 holds the calculated distance information for each region and proceeds to step S15.

[0045] In step S15, the dividing unit 116 adds subject information to each region. The subject information is information indicating whether a subject in the region is the main subject, and is determined based on subject detection information, focus determination information, contrast information, user setting information, and the like. For example, in a determination method based on subject detection information, a detection target is set in advance, and whether or not a subject is detected in each region is stored as subject information. In FIG. 8(d), if buildings are set as detection targets, then subject detection will be detected in regions 82, 83, 84, and 85 of regions 81 to 86, and subject detection will not be detected in the other regions. Even when using information other than the subject detection result, it is possible to determine subject information based on information such as in-focus regions, regions other than low-contrast regions such as the sky, and regions where an autofocus frame has been set by the user. The dividing unit 116 stores the determined subject information for each region and proceeds to step S16.

[0046] In step S16, the determination unit 114 determines the correction strength based on the distance information. A method for determining the correction strength based on the distance information will be described. First, factors that cause fluctuations to change will be described. One of the factors that cause fluctuations to change is a change in the atmospheric refractive index, which changes depending on the wavelength of light, temperature, air pressure, humidity, carbon dioxide concentration, etc. Taking the above into consideration, the greater the distance to the subject, the greater the fluctuation tends to be. From the above, the correction strength is determined to be weak when the distance information for the specified area indicates "near" and strong when it indicates "far." By determining the correction strength as described above, it is possible to determine a correction strength appropriate for the degree of fluctuation.

[0047] More specifically, in each group or each divided region, the longer the distance between the imaging means and the subject, the stronger the correction strength determined by the determination unit 114. In other words, the shorter the distance between the imaging means and the subject, the weaker the correction strength determined by the determination unit 114. In other words, among the divided regions, the determined correction strengths are different in at least two regions having different subject distance information. Conversely, the determined correction strengths are the same in at least two regions having the same subject distance information, which is a feature of the image processing device according to this embodiment.

[0048] A method for determining the correction intensity taking the background into consideration will now be described. While it has been explained above that the greater the distance to the subject, the greater the fluctuation tends to be, some subjects are less susceptible to fluctuation. An example of a subject less susceptible to fluctuation is a flat area such as the sky (hereinafter referred to as the background). This is because the background has a small difference in pixel values ​​to begin with, and is therefore less susceptible to changes in pixel values ​​due to fluctuation. Therefore, the correction intensity is determined to be weaker when a predetermined region is the background. The background region can be identified by determining that the predetermined region is the background when it is "not the main subject" based on the retained subject information. Alternatively, the predetermined region may be identified as the background when the distance information for the predetermined region indicates "infinity" (i.e., the distance between the imaging unit and the subject is greater than or equal to a predetermined distance). These functions are executed by an identification unit (not shown) and are realized by the CPU 103 executing a program. The correction intensity for a region identified as the background by the identification unit is weaker than the correction intensity for regions other than the identified region, or is fixed to a predetermined value (e.g., a small value such as 0).

[0049] By determining the correction strength as described above, it is possible to prevent determining an unnecessarily strong correction strength when the predetermined area is the background. This makes it possible to reduce motion blur that occurs when a moving subject passes through the background. The determination unit 114 stores the coordinate information of the predetermined area and the determined correction strength in association with each other, and proceeds to step S17.

[0050] In step S17, the determination unit 114 determines whether or not the determination of the correction intensity has been completed for all regions divided by the division unit 116. If it is determined that the determination has been completed (Yes), the determination unit 114 outputs the coordinate information of all regions that it has stored and the corresponding correction intensity to the correction unit 115, and ends this flow. If it is determined that the determination has not been completed (No), the process proceeds to S16, and the process is repeated until the determination of the correction intensity has been completed for all regions.

[0051] Therefore, by using the correction strength determination flow in this embodiment, it is possible to determine an appropriate correction strength for each divided region, which makes it possible to reduce fluctuations by applying fluctuation correction to regions where fluctuations are occurring, while reducing blurring caused by moving objects in other regions.

[0052] In this embodiment, an example has been shown in which an input image is divided into multiple regions and grouped based on subject distance information, but the grouping step may be omitted. In that case, the correction strength of the correction process may be determined for each of the multiple divided regions. However, in consideration of the computational cost of the CPU 103, it is desirable to calculate the correction strength for each group, as in this embodiment. Alternatively, each of the multiple acquired subject distance information may be linked to the correction strength determined for each different subject distance information and stored as a table, and the correction strength of the correction process to be applied to the multiple divided regions may be referenced from the stored table. This reduces the computational cost of the CPU 103 compared to determining the correction strength for all of the multiple divided regions.

[0053] According to this embodiment, it is possible to correct fluctuations with an appropriate correction strength regardless of the subject being photographed or the degree of fluctuation.

[0054] <Embodiment 2> Hereinafter, a method for determining the correction intensity by the determination unit 114 in this embodiment will be described with reference to the flowchart in Fig. 9. Note that the same parts as those in the first embodiment are given the same reference numerals, and duplicated descriptions will be omitted.

[0055] In step S21, the dividing unit 116 adds fluctuation information for each region. The fluctuation information for each region is determined based on multiple pieces of fluctuation information acquired by the acquiring unit 111 for mesh frames included in the region. FIG. 10 shows the fluctuation information for each mesh frame, with larger values ​​indicating greater fluctuation. The fluctuation information for each region is calculated from the average, maximum, minimum, median, or variance of the fluctuation information of the mesh frames present in the region. Furthermore, it is not necessary to use all of the mesh frames within the region; it is also possible to use only mesh frames in edge regions, or only edge frames with high contrast. For example, if only the mesh frames of the edge portions are used for region 84, the fluctuation information is calculated to be 10 using only the mesh frames (15,3), (16,3), (17,3), (18,3), (19,3), (15,4), (19,4), (15,5), (16,5), (17,5), (18,5), and (19,5). Also, as in region 81, if the distance information in step S14 is infinity or if the subject information in step S15 indicates no main subject, it is not necessary to add fluctuation information for that region. Divider 116 holds the calculated fluctuation information for each region and proceeds to step S22.

[0056] In step S22, the determination unit 114 determines the correction intensity based on the distance information and the fluctuation information. First, a method for determining the correction intensity based on the fluctuation information will be described. The correction intensity is determined using the fluctuation information stored for each region so that the correction intensity is weak when the fluctuation information for a specific region indicates "small" and strong when the fluctuation information indicates "large." By determining the correction intensity as described above, it is possible to determine a correction intensity appropriate for the degree of fluctuation. However, if the image contains changes in pixel values ​​that are not due to fluctuation, there is a possibility that the change may be erroneously determined. In response to this, a method for determining the correction intensity based on the distance information and the fluctuation information will be described. The correction intensity is determined based on a value calculated by adding or multiplying the correction intensity determined based on the fluctuation information and the correction intensity determined in S16. By determining the correction intensity as described above, it is possible to determine a correction intensity appropriate for the degree of fluctuation with higher accuracy. The determination unit 114 associates the coordinate information of the specific region with the determined correction intensity and stores the association, and proceeds to step S17.

[0057] Therefore, according to the correction intensity determination flow in this embodiment, compared to embodiment 1, by additionally using fluctuation information to determine the correction intensity, it is possible to determine an appropriate correction intensity with higher accuracy for each divided area.

[0058] This makes it possible to reduce fluctuations in areas where fluctuations are occurring by performing fluctuation correction, while reducing motion blur in other areas.

[0059] According to this embodiment, it is possible to correct fluctuations with an appropriate correction strength regardless of the subject being photographed or the degree of fluctuation.

[0060] <Embodiment 3> Hereinafter, a method for determining the correction intensity by the determination unit 114 in this embodiment will be described with reference to the flowchart in Fig. 11. Note that the same parts as those in the first and second embodiments are given the same reference numerals, and redundant description will be omitted.

[0061] In step S31, the division unit 116 adds moving object information to each region. The moving object information indicates whether a moving object is present within the region and is determined for each region based on the moving object detection result. Moving object detection methods include methods using optical flow, background subtraction, etc., as well as a method using the acquisition unit 111 to perform detection based on temporal changes in pixel values ​​at specified coordinates within the image. Similar to FIG. 10, FIG. 12 shows fluctuation information for each mesh frame. However, while FIG. 10 shows fluctuation information for when no moving object is present, FIG. 12 shows fluctuation information for when a moving object is present in region 86. When a car is moving in region 86, the temporal change in pixel value at the specified coordinates is greater than the pixel value change due to fluctuation. This is because the amount of movement of a moving object per given time is greater than the amount of fluctuation. Therefore, by setting a predetermined threshold for the temporal change in pixel value, a pixel value change below the threshold can be considered to be due to fluctuation, and a pixel value change above the threshold can be considered to be due to a moving object, thereby determining the presence or absence of a moving object within the region. Furthermore, even if the difference between the minimum and maximum fluctuation amounts for each mesh frame over a given time period is large, it is considered that the pixel value change is not due to fluctuation but is caused by a moving object. This is because moving objects frequently enter and exit the region, and the pixel value change varies greatly depending on whether or not there is a moving object within the region. Therefore, the presence or absence of a moving object within a region may be determined based on whether or not there is a mesh frame in which the difference between the minimum and maximum fluctuation amounts over a given time period is equal to or greater than a given threshold. The dividing unit 116 retains the determined moving object information for each region and proceeds to step S32.

[0062] In step S32, the determination unit 114 determines the correction strength based on the distance information and fluctuation information. In step S16, the method for determining the correction strength based on the fluctuation information was described. However, if a moving subject is included in the predetermined region, the fluctuation information for the predetermined region indicates "large" relative to changes in pixel values ​​caused by the moving subject. Therefore, if the determination method described above is followed, the correction strength will be strong, resulting in motion blur, a drawback of motion correction. Therefore, the correction strength is determined using the moving object information stored for each region, so that it is weaker when the moving object information for the predetermined region indicates "moving object present." Note that, when the moving object information indicates "no moving object," the correction strength is determined based on the distance information and fluctuation information (step S22). As described above, by determining the correction strength taking moving objects into consideration, it is possible to determine an appropriate correction strength while reducing motion blur. The determination unit 114 associates the coordinate information of the predetermined region with the determined correction strength and stores the association, and proceeds to step S17.

[0063] Therefore, according to the correction intensity determination flow in this embodiment, by additionally using moving object information in determining the correction intensity compared to embodiment 2, it is possible to determine an appropriate correction intensity for each divided region. This makes it possible to reduce fluctuations by applying fluctuation correction to regions where fluctuations are occurring, while reducing motion blur in other regions.

[0064] According to this embodiment, it is possible to correct fluctuations with an appropriate correction strength regardless of the subject being photographed or the degree of fluctuation.

[0065] <Other embodiments> The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-described first embodiment. This program is supplied to a system or device via a network or a computer-readable non-volatile storage medium, and is read and executed by one or more processors in the computer of the system or device. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0066] 101 Imaging optical system 102 Image sensor 103 CPU 104 RAM 105 ROM Bus 106

Claims

1. a dividing means for dividing an input image into a plurality of regions; an acquisition means for acquiring information about a subject distance for each of the plurality of regions; a correction means for executing a correction process for correcting fluctuations in the input image for each of the plurality of regions; a determination unit that determines a correction intensity of the correction process for each of the plurality of regions based on information about the subject distance for each of the plurality of regions, The image processing device according to claim 1, wherein the determined correction strength is different in at least two of the plurality of regions for which the information relating to the subject distance is different.

2. 2. The image processing apparatus according to claim 1, wherein the dividing means groups the plurality of regions based on information relating to the subject distance.

3. 3. The image processing apparatus according to claim 2, wherein the information about the subject distance is information indicating a distance between an image capturing unit and the subject.

4. 4. The image processing apparatus according to claim 3, wherein the determined correction strength is stronger as the distance between the image capturing means and the subject increases in each of the plurality of regions.

5. 4. The image processing apparatus according to claim 3, wherein the determined correction strength is weaker as the distance between the image capturing means and the subject becomes shorter in each of the plurality of regions.

6. 3. The image processing apparatus according to claim 2, wherein the information about the subject distance is information indicating a defocus amount in the input image.

7. further comprising a specifying means for specifying an area corresponding to a background of the input image among the plurality of areas; 2. The image processing device according to claim 1, wherein the determining unit determines the correction strength for each of the plurality of regions so that the correction strength for the specified region is weaker than the correction strength for regions other than the specified region.

8. 8. The image processing apparatus according to claim 7, wherein the specifying unit specifies the area corresponding to the background based on information relating to the subject distance for each of the plurality of areas.

9. the specifying means determines whether or not the distance between the imaging means and the subject for each of the plurality of regions is greater than a predetermined distance based on information about the subject distance for each of the plurality of regions; 8. The image processing apparatus according to claim 7, wherein an area where the distance between said image capturing means and said subject is determined to be greater than a predetermined distance is specified as an area corresponding to the background.

10. the acquiring means acquires information about the fluctuation in the input image based on a difference between a pixel value of the input image and a pixel value of at least one image input before or after the input image; 3. The image processing apparatus according to claim 1, wherein the determining means determines the correction strength for each of the plurality of regions based on information about the fluctuation.

11. 2. The image processing apparatus according to claim 1, wherein the determined correction strength is the same for at least two of the plurality of regions for which the information relating to the subject distance is the same.

12. 3. The image processing apparatus according to claim 2, wherein the dividing means groups, into one group, those regions among the plurality of regions having substantially the same information about the subject distance.

13. 4. The image processing device according to claim 3, wherein the image processing means determines the correction intensity for each of the groups based on information about the subject distance corresponding to the group.

14. a segmentation step of segmenting an input image into a plurality of regions; an acquisition step of acquiring information about a subject distance for each of the plurality of regions; a correction step of executing a correction process for correcting fluctuations in the input image for each of the plurality of regions; a determination step of determining a correction intensity of the correction process for each of the plurality of regions based on information about the subject distance for each of the plurality of regions, An image processing method, wherein the determined correction strength differs in at least two of the plurality of regions where the information relating to the subject distance differs.

15. A program for causing a computer to execute the image processing method according to claim 14.

16. A computer-readable storage medium storing the program according to claim 15.

Images