Image processing apparatus, image processing method, and program

Abstract

This image processing apparatus comprises an image processing unit that performs area signal generation processing for generating an area signal which indicates, for each of the positions within a frame of an image signal, the results of determination of the stability and highness of a spatial frequency, and image processing of the image signal which is to be executed on the basis of the area signal.

Description

Image processing device, image processing method, and program

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to an image processing technology for dealing with moire that occurs in an image.

[0002] For example, when a subject with a fine pattern is photographed with a digital camera, such as when a performer is photographed with a display showing a video in the background, patterns that do not actually exist may appear. This is called moiré, and results in an unattractive image. Patent Document 1 listed below discloses a technique for detecting moiré from the difference between an original image and an image blurred by an optical low-pass filter or the like, and for suppressing moiré.

[0003] Japanese Patent Application Laid-Open No. 2018-207414

[0004] However, moiré reduction processing using optical low-pass filters and the like also reduces high-frequency components unrelated to moiré, so there is a trade-off between moiré reduction and image resolution; attempting to reduce moiré results in a decrease in the overall resolution of the image.

[0005] Therefore, the present disclosure proposes a technique that enables necessary processing to be performed on moiré regions without reducing the perceived resolution of the entire image as much as possible.

[0006] The image processing device according to the present technology includes an image processing unit that performs an area signal generation process that generates an area signal indicating a determination result of stability and high frequency of spatial frequency for each position in a frame of an image signal, and an image processing unit that performs image processing on the image signal based on the area signal. An area in a frame that is determined to have stable and high frequency spatial frequency can be evaluated as an area where moiré is likely to occur.

[0007] 1 is a block diagram of an imaging device equipped with an image processing device according to an embodiment of the present technology; FIG. 2 is an explanatory diagram of frequency bands targeted for moiré reduction processing in the embodiment; FIG. 3 is an explanatory diagram of an area signal and an image with moiré reduced according to the embodiment; FIG. 4 is a block diagram of a moiré reduction processing configuration in the image processing device according to the embodiment; FIG. 5 is a flowchart of a moiré reduction processing procedure in the embodiment; FIG. 6 is an explanatory diagram of upsampling in the embodiment; FIG. 7 is an explanatory diagram of peak extraction in the embodiment; FIG. 8 is an explanatory diagram of periodicity determination in the embodiment; FIG. 9 is an explanatory diagram of periodicity determination in the embodiment; FIG. 10 is an explanatory diagram of blending processing in the embodiment; FIG. 11 is an explanatory diagram of variable setting of blending ratios in the embodiment; FIG. 12 is an explanatory diagram of binary setting of blending ratios in the embodiment; FIG. 13 is a block diagram of a processing configuration in the image processing device according to the embodiment; FIG. 14 is a block diagram of an image reduction processing configuration in the image processing device according to the embodiment; FIG. 15 is a block diagram of another example of a moiré reduction processing configuration in the image processing device according to the embodiment;

[0008] Hereinafter, the embodiments will be described in the following order: <1. Imaging device and moire> <2. Moire reduction processing> <3. Various processing examples> <4. Summary and modified examples>

[0009] 1 shows the configuration of an imaging device 1 equipped with an image processing device 10 according to an embodiment. Note that installing the image processing device 10 in the imaging device 1 is just one example, and the image processing device 10 can be installed in a variety of devices.

[0010] The imaging device 1 in Fig. 1 is a so-called digital camera that can capture both still images and video by switching between different shooting modes. It may be a digital camera for professional use or a digital camera for general use. The imaging device 1 may be capable of capturing only video images or only still images. The imaging device 1 may have a separate main body and lens barrel, or the lens barrel may be integrated into the main body.

[0011] The imaging device 1 includes an image processing device 10 , an imaging element (image sensor) 12 , a recording control unit 14 , a display unit 15 , an output unit 16 , an operation unit 17 , a lens system 21 , a driver unit 22 , a camera control unit 30 , and a memory unit 31 .

[0012] The lens system 21 includes lenses such as a zoom lens and a focus lens, and an iris mechanism. The lens system 21 guides light (incident light) from a subject and focuses the light on the image sensor 12.

[0013] The image sensor 12 is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signal obtained by photoelectrically converting the received light, and further performs A / D (Analog / Digital) conversion processing. The captured image data is then output as digital data to the image processing device 10 or the camera control unit 30 in the downstream stage.

[0014] The image processing device 10 is configured with an arithmetic processor such as a DSP (Digital Signal Processor). The image processing device 10 performs various signal processing on captured image data from the image sensor 12. For example, the image processing device 10 performs development processing, resolution conversion processing depending on the output destination, metadata addition processing, formatting processing, etc. In the case of this embodiment, the image processing device 10 is particularly provided with a processing unit 60, which will be described later with reference to FIG. 4, etc., and can perform processing related to the occurrence of moiré, such as moiré reduction.

[0015] The recording control unit 14 performs processing to record image files such as still image data and video data, and metadata associated with the image files, onto a recording medium such as a non-volatile memory. The actual configuration of the recording control unit 14 can be various. For example, the recording control unit 14 may be a circuit that writes to and reads from a flash memory built into the imaging device 1, or may be in the form of a memory card (e.g., a portable flash memory) that can be attached to and detached from the imaging device 1 and a card recording / playback unit that accesses the memory card for recording / playback.

[0016] The display unit 15 is a display unit that displays various information to the user, and specifically refers to a display panel or viewfinder such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display provided on the back of the imaging device 1.

[0017] The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 30. For example, the display unit 15 displays a playback image of image data read from the recording medium by the recording control unit 14. The display unit 15 is also supplied with image data of captured images whose resolution has been converted for display by the camera signal processing unit 13. The display unit 15 displays a so-called through image (a monitoring image of the subject), which is an image captured during release standby, based on the image data of the captured image in response to instructions from the camera control unit 30. The display unit 15 also executes displays on the screen of various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the camera control unit 30.

[0018] The output unit 16 performs wired or wireless data communication and network communication with external devices. For example, it transmits and outputs image content to an external display device, recording device, playback device, etc. The output unit 16 may also be a network communication unit that performs communication via various networks such as the Internet, a home network, a LAN (Local Area Network), etc., and transmits and receives various data to and from servers, terminals, etc. on the network.

[0019] The operation unit 17 collectively represents input devices for the user to input various operations. Specifically, the operation unit 17 represents various operators (such as a shutter button) provided on the main body of the imaging device 1. The operation unit 17 detects user operations, and sends signals corresponding to the input operations to the camera control unit 30.

[0020] The operation unit 17 may be implemented not only by physical keys or other operators but also by a touch panel. For example, a touch panel may be formed on a display panel, and various operations may be performed by touch panel operations using icons, menus, etc. displayed on the display panel. Alternatively, the operation unit 17 may be configured to detect user tap operations, etc. using a touch pad or the like. Furthermore, the operation unit 17 may also be configured as a receiving unit for an external operation device such as a separate remote controller.

[0021] The camera control unit 30 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 31 stores information and the like used for processing by the camera control unit 30. The illustrated memory unit 31 collectively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 31 may be a memory area built into the microcomputer chip that constitutes the camera control unit 30, or may be configured by a separate memory chip.

[0022] The camera control unit 30 executes programs stored in the ROM, flash memory, etc. of the memory unit 31 to control the entire imaging device 1. For example, the camera control unit 30 controls the shutter speed of the image sensor 12, instructs the camera signal processing unit 13 to perform various signal processing, controls the imaging and recording operations in response to user operations, plays back recorded image files, and controls the operations of the lens system 21, such as zoom, focus, and aperture adjustment in the lens barrel, and controls the user interface operations. With regard to aperture adjustment, the camera control unit 30 controls the F-number variable in response to user operations and instructs the F-number for automatic control (auto iris).

[0023] The RAM in the memory unit 31 is used to temporarily store data, programs, etc. as a working area when the CPU of the camera control unit 30 processes various types of data. The ROM and flash memory (non-volatile memory) in the memory unit 31 are used to store the OS (Operating System) used by the CPU to control each unit, content files such as image files, application programs for various operations, firmware, etc.

[0024] The driver unit 22 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for an aperture mechanism motor, etc. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 30, and execute operations such as moving the focus lens or zoom lens and opening and closing the aperture blades of the aperture mechanism.

[0025] Note that the above configuration is an example, and does not show all of the configurations that are typically provided in an imaging device. Furthermore, the technology of this embodiment can be applied even to an imaging device that does not have some of the configurations shown in the drawings.

[0026] 2. Moire Reduction Processing Generally, moire occurring in images captured by a digital camera can be attributed to the following causes (C1) and (C2).

[0027] (C1) If the spatial frequency of the subject exceeds the Nyquist frequency of the image sensor, it cannot be sampled correctly and is displayed as a pattern with a low spatial frequency. (C2) Even if the spatial frequency of the subject is below the Nyquist frequency, interference with the sampling frequency causes a pattern that does not actually exist to be visible.

[0028] Image processing device 10 of this embodiment targets moiré caused by (C2) and performs reduction processing on it. For example, in many professional cameras, signals above the Nyquist frequency are cut off by an optical low-pass filter, and it is the moiré caused by (C2) that is most likely to be a problem in practice.

[0029] 2, the vertical axis represents amplitude and the horizontal axis represents frequency, with bands FA and FB shown with the Nyquist frequency FN as the boundary. Band FA is a frequency band lower than the Nyquist frequency FN, but moiré is more noticeable in the higher frequencies below the Nyquist frequency FN, such as the shaded band. Therefore, moiré reduction processing is performed on areas of the image whose spatial frequencies fall within the shaded band.

[0030] In particular, this moiré reduction process does not use computationally intensive processes such as Fourier transforms or machine learning, but estimates spatial frequencies using a computationally intensive algorithm. Regions where the spatial frequencies are stable and where the stable frequencies are evaluated as high frequencies are determined to be regions where moiré is likely to occur, and low-pass filter processing is applied to reduce the moiré. Note that the term "low-pass filter" is also abbreviated as "LPF."

[0031] 3 shows an original image before moiré reduction processing, an area signal corresponding to the original image, and an output image after moiré reduction processing. In the original image, moiré 100 appears as a streaky pattern in the background portion. For this original image, an area signal is generated that identifies areas where the spatial frequency is stable and high frequency. The area signal is information in which a moiré occurrence estimation value is set for each pixel in a frame. The illustrated area signal is information that identifies the background area in the image frame as an area where moiré occurrence is estimated.

[0032] For example, the area signal is information in which pixels where moiré is estimated to occur are represented as "1" (represented by white) and pixels where moiré is not estimated to occur are represented as "0" (represented by black), and is generated for each pixel in each frame of the captured image. Alternatively, the area signal may not be binary, but may be multi-valued, so that an image where moiré is estimated to occur takes on a value according to the accuracy of the estimated occurrence.

[0033] For example, based on such an area signal, LPF processing is performed on areas (pixels) where moiré is estimated to occur, thereby reducing moiré, and LPF processing is not performed on other areas (pixels).As a result, as shown in the output image, an image can be obtained in which moiré is reduced while the overall perceived resolution is not reduced.

[0034] 4 shows a specific configuration of the processing unit 60. For example, processing equivalent to this processing unit 60 is performed within the image processing device 10.

[0035] The processing unit 60 supplies the input image signal (the original image in FIG. 3) to the blending unit 54 , the LPF 53 , and the area signal generating unit 50 .

[0036] The area signal generation unit 50 performs frequency estimation of the spatial frequency. Then, for a region where the spatial frequency is high, for example, a region where the spatial frequency corresponds to the band of the shaded area in FIG. 2 , the stability of the spatial frequency is determined. Here, the stability of the spatial frequency means that there is a high degree of continuity of almost the same spatial frequency. It is not necessary for the spatial frequency to be completely the same; it is sufficient if a certain threshold value is set for the determination and the region can be evaluated as being relatively stable, as will be described later. The area signal generation unit 50 then generates an area signal indicating a region where the spatial frequency is determined to be high and stable, i.e., a region where moiré is estimated to occur (hereinafter referred to as an "estimated moiré region"), and provides the signal to the blending unit 54.

[0037] In the LPF 53, a predetermined cutoff frequency suitable for moiré reduction is set, and low-pass processing is performed on the input image signal, and the band-limited LPF image signal is supplied to the blending unit 54. The input image signal and the LPF image signal are input to the blending unit 54. The blending unit 54 blends the input image signal and the LPF image signal based on an area signal. For example, for each pixel, the blending process is performed by selecting the LPF image signal or increasing its blending ratio for pixels determined to be in a moiré estimated area, and selecting the input image signal or increasing its blending ratio for pixels not determined to be in a moiré estimated area. An output image is obtained as a result of this blending process. The output image is an image with reduced moiré, as shown in FIG. 3.

[0038] The generation of the area signal and the moire reduction processing executed by the processing unit 60 as described above will now be described in detail. Fig. 5 is a flowchart showing the processing procedure of the processing unit 60. In Fig. 5, steps S101 to S104 are processed by the area signal generation unit 50, and step S105 is processed by the blending unit 54. Such processing is executed for each frame of the input image signal.

[0039] In step S101, the processing unit 60 upsamples one frame of image supplied as an input image signal. The purpose of the upsampling is to obtain the peak position more accurately in the next step S102.

[0040] Figure 6 shows an image as a spatial waveform, with the vertical axis representing brightness values ​​and the horizontal axis representing pixel positions. This is also true for Figures 7 to 10. The pixel positions can be in either the horizontal or vertical direction. The solid line represents the waveform of the original image, and the white circles (○) represent the sampling results. The vertical dashed lines represent the sampling period. When moiré occurs, the waveform after sampling often does not sample the peaks of the original waveform, as shown in the top row of Figure 6. For example, the "○" is often located at a position that is not a peak of the waveform.

[0041] When upsampling is performed, interpolation is performed as shown in the bottom of Figure 6. The black circles (●) represent the interpolation results. This interpolation makes it possible to determine the position of the peak more accurately. This is because, according to the sampling theorem, signals below the Nyquist frequency can ideally be restored to their original frequency by upsampling.

[0042] In step S102 of Fig. 5, the processing unit 60 extracts peaks. When there are five consecutive sampling points a, b, c, d, and e, including interpolated points, the peak c is determined to have a magnitude relationship of a<b<c>d>e or a>b>c<d<e. Note that, for illustrative purposes, five consecutive points are used here, but this does not necessarily have to be five points.

[0043] The top row of Figure 7, like the bottom row of Figure 6, shows the upsampling results, and the bottom row of Figure 7 shows the results of actual peak detection. Internally, information can be stored as three values: "upward convex peak," "downward convex peak," and "not a peak." In the bottom row of Figure 7, upward convex peaks are indicated by diagonally shaded circles and are labeled as peaks Pu1, Pu2, Pu3, and Pu4, while downward convex peaks are indicated by diagonally shaded circles and are labeled as peaks Pb1, Pb2, and Pb3. Non-peak samples are indicated by double circles (◎).

[0044] In step S103 of Fig. 5, the processing unit 60 calculates the period, i.e., the distance between peaks. As shown in the upper part of Fig. 8, the distance between peaks that are convex upward and the distance between peaks that are convex downward are calculated. This corresponds to calculating the period of the waveform. For the sake of explanation, it is assumed that the distance between peaks Pu1 and Pu2 is calculated as "4", the distance between peaks Pu2 and Pu3 is calculated as "5", and so on, as shown in the figure.

[0045] If the distance between adjacent peaks on opposite sides is recorded as a period, the result will be as shown in the bottom of Figure 8. The numerical value of the period is shown in a circle. For example, the period of peak Pb1 is "4", which is the distance between peaks Pu1 and Pu2. The period of peak Pu2 is "5", which is the distance between peaks Pb1 and Pb2. The period of peak Pb2 is "5", which is the distance between peaks Pu2 and Pu3.

[0046] 5, the processing unit 60 determines, for each pixel in the frame, whether the period is equal to or less than a certain value, i.e., whether there is high frequency and whether the frequency around the pixel of interest is stable, based on the calculated period. For example, each pixel is set as a pixel of interest to be determined, and the following process is performed for each pixel.

[0047] An example will be given using the upper part of Figure 9. The pixel of peak Pb2, surrounded by a dashed line frame, is set as the pixel of interest. A predetermined range of pixels surrounding this pixel is set as the region of interest, and the total number of the most frequent and second most frequent periods is counted. For example, if peak Pb2 is set as the pixel of interest, a total of 11 pixels from peak Pb1 to peak Pb3 is set as the region of interest. The periods in this region of interest are "4," "5," "5," "5," and "5," and the total number of the most frequent and second most frequent periods is 5. For the sake of explanation, this total number of the most frequent and second most frequent periods is referred to as the "period judgment value." In the above case, the period judgment value is "5."

[0048] If the waveform does not have a stable period, the numerical values ​​will vary, and the period judgment value will be small. For example, in a pattern with a high frequency but a slightly irregular period, as shown in the lower part of Figure 9, if the pixel of interest is peak Pb12, the region of interest will be a total of 11 pixels from peak Pb11 to peak Pb13, but the periods in that region of interest are "5", "5", "2", "4", and "6", and the total number of periods with the most and second most frequent is 3, so the period judgment value will be "3".

[0049] In this way, the period judgment value functions as a value indicating the degree of period stability. The larger this period judgment value, the higher the possibility of moiré occurrence can be considered. For example, a threshold value can be set for the period judgment value, and if it is equal to or greater than the threshold, it can be determined that the region is a moiré estimation region, and if it is less than the threshold, it can be determined that the region is not a moiré estimation region. Note that if the period value of the pixel of interest is equal to or greater than a predetermined value, it can be determined that there is no high frequency, so it can be determined that the region is not a moiré estimation region without determining the stability of the period.

[0050] Within the processing unit 60, the area signal generation unit 50 supplies this period judgment value as area information to the blending unit 54 for each pixel of interest. For pixels that are not judged to have period stability because they do not have high frequency characteristics, the period judgment value may be set to 0. The area signal generation unit 50 may also supply the binary value resulting from comparing the period judgment value with a predetermined threshold to the blending unit 54 as area information.

[0051] The area signal described above serves as information for determining an area where the spatial frequency is equal to or higher than a specific frequency and where the period is relatively stable as an estimated moiré area. Note that in calculating the period determination value, it is possible to make a stricter determination based on the variation in period by assigning a negative weight to the number of periods other than the first and second most frequent.

[0052] 5, the processing unit 60 references the area signal and blends the original input image signal with the LPF image signal that has passed through the LPF 53. The input image signal, the LPF image signal, and the output image signal are shown in FIG. 10. The input image signal is blended at (100-α)%, and the LPF image signal is blended at α% to obtain the output image signal.

[0053] In this case, the processing unit 60 (blending unit 54) can determine the percentage of the LPF image signal to blend with the original input image signal by obtaining the period judgment value as the area signal. The degree of moiré reduction is determined by the relationship between the period judgment value and the blending ratio, but this can be freely changed by setting.

[0054] 11 and 12, the horizontal axis represents the periodicity determination value, and the vertical axis represents the blending ratio of the LPF image signal. For example, as shown in FIG. 11, it is conceivable to linearly increase the blending ratio of the LPF image signal with respect to the periodicity determination value.

[0055] It is also possible to blend the LPF image signal when a certain threshold is exceeded, as shown in Figure 12. In the case of Figure 12, the blending rate of the LPF image signal is set to 100% when a certain threshold is exceeded, and to 0% when the threshold is not exceeded, thereby selecting the LPF image signal for pixels in the moiré estimation area, and selecting the input image signal for pixels outside the moiré estimation area. In the case of Figure 12, it is also possible to always perform blending, such as setting the blending rate of the LPF image signal to 90% when a certain threshold is exceeded, and to set the blending rate of the LPF image signal to 10% when the threshold is not exceeded.

[0056] As described above, for each pixel, the input image signal and the LPF image signal are blended according to the area signal (period determination value) when that pixel is the pixel of interest. This increases the blending rate of the LPF image signal for pixels in areas with high frequencies and stable periods, and decreases the blending rate of the LPF image signal for pixels in other areas. This makes it possible to reduce moiré while maintaining the sense of resolution in other areas. In particular, the processing of this embodiment not only determines whether the spatial frequency is high but also whether the spatial frequency is constant within the area of ​​interest. This makes it possible to separate high-frequency areas where moiré is likely to occur from high-frequency areas where it is not. This makes it possible to reduce moiré while suppressing a decrease in the sense of resolution in areas unrelated to moiré.

[0057] Furthermore, the above processing does not require information from the previous frame. This is because the blending ratio is determined by determining the periodicity of the spatial frequency within the frame. This allows the processing to function even when the subject and imaging device are stationary. Furthermore, no frame delay occurs.

[0058] Furthermore, the process shown in Figure 5, which detects areas prone to moiré, uses not only high frequency but also spatial frequency stability as a criterion for judgment, resulting in high accuracy in moiré judgment and not requiring heavy processing such as Fourier transform to judge spatial frequency stability. As a result, the overall computational complexity is small, making real-time processing possible on edge devices. Another advantage is that the process does not require special optical systems and can be completed using only signal processing.

[0059] Furthermore, because the LPF is applied during signal processing, its characteristics can be easily changed compared to optical LPFs. Therefore, the degree of moiré reduction can also be easily changed. The LPF coefficients can also be freely set and can be varied according to the desired period.

[0060] 3. Various Processing Examples Various processing examples will be described as application examples. Fig. 13 shows an example in which a filter processing unit 55 is provided in place of the LPF 53 in Fig. 4 as a configuration of the processing unit 60. In this case, the filter processing unit 55 refers to a filter that performs some kind of image processing other than the LPF.

[0061] By providing a filter that performs specific brightness processing or color conversion processing as the filter processing unit 55, it is possible to realize a function of using the detected area signal to warn of the high possibility of moiré occurrence. The output of the filter processing unit 55 is a filtered image signal. This filtered image signal may be, for example, a highlight image in which the brightness value of the input image signal is increased, or an image signal in which the color has been changed. In other words, the input image signal is an image signal in which the brightness or color has been significantly changed. The blending unit 54 blends the input image signal and the filtered image signal.

[0062] In this case, the processing unit 60 generates an area signal in the area signal generation unit 50. This is the same processing as steps S101 to S104 in FIG. 5. The processing unit 60 then performs processing to generate a moiré warning image in accordance with the area signal in the blending unit 54. This can be realized as processing to blend the input image signal and the filtered image signal in accordance with the area signal. For example, when the area signal is the above-mentioned period determination value, the blend ratio of the filtered image signal is selected using the blend ratios shown in FIG. 11 or 12, and the blending is performed to obtain an output image signal.

[0063] This produces an output image signal in which the brightness and color of only the moiré-predicted region differs significantly from the original input image signal. This output image clearly shows the moiré-presumed region to the viewer, providing a moiré warning.

[0064] Next, an example can be considered in which detail processing (sharpness processing) is performed by the filter processing unit 55. Detail processing is processing that emphasizes the edge portions, contour portions, etc. of an image by superimposing an image signal that has been passed through a high-pass filter on the image signal, for example. In other words, the filtered image signal in this case is a detail processed signal.

[0065] In the processing unit 60, the area signal generation unit 50 generates an area signal. The blending unit 54 then blends the detail processing signal with the original input image signal in accordance with the area signal. For example, when the area signal is the period determination value described above, a blending ratio for the detail processing signal is selected at a ratio opposite to the blending ratio shown in Figure 11 or 12, and the signals are blended to produce an output image signal. In other words, the lower the period determination value and the lower the likelihood of moiré occurrence for a pixel, the higher the blending ratio of the detail processing signal is.

[0066] In image processing, moiré can be emphasized by detail processing that strengthens details. The above processing allows the detected area signal to be used to adjust the detail processing strength, thereby avoiding areas where moiré is likely to be noticeable and suppressing the emphasis of moiré, and obtaining an output image signal that has undergone detail processing.

[0067] Conversely, moiré can also be emphasized by using the detail processing signal. In the processing unit 60, the area signal generation unit 50 generates an area signal. Then, the blending unit 54 performs moiré emphasis processing in accordance with the area signal. Specifically, when the area signal is the above-mentioned periodicity determination value, the blending ratio of the detail processing signal is selected using the blending ratios shown in Figure 11 or 12. In other words, the higher the periodicity determination value, the higher the blending ratio of the detail processing signal.

[0068] This increases the ratio of detail processing signals in areas where moiré is suspected, resulting in an output image signal with emphasized moiré. Therefore, for example, moiré can be used as an image effect. Another possible use is to warn users that moiré is occurring by displaying it in an emphasized manner.

[0069] Next, consider the case of reducing an image. Reducing an image reduces the Nyquist frequency. For example, reducing an image by half reduces the Nyquist frequency by half. This can result in frequencies above or close to the Nyquist frequency, which can cause moiré.

[0070] In this case, to prevent moiré from occurring, a process such as using an LPF to limit the band and then thinning out pixels may be performed. In this case, the characteristics of the LPF pose a trade-off: whether to prioritize leaving the necessary frequencies and not remove frequency bands that may cause moiré, or to prioritize preventing moiré and remove frequency bands that you want to keep. Therefore, it is possible to adjust the blend ratio of the LPF image signal using the detected area signal.

[0071] 14, the processing unit 60 has a reduction processing unit 56 provided after the blending unit 54, which reduces the image by thinning out pixels to generate an output image signal. In this case, a parameter for setting the reduction ratio is provided to the LPF 53 and the reduction processing unit 56. The LPF 53 sets a cutoff frequency in accordance with the reduction ratio parameter. The reduction processing unit 56 sets the pixel thinning rate in accordance with the reduction ratio parameter.

[0072] The processing unit 60 then performs blending processing of the input image signal and the LPF image signal in the blending unit 54 according to the area signal. Specifically, when the area signal is the above-mentioned periodicity determination value, the blending ratio of the LPF image signal is selected at a blending ratio such as that shown in Figure 11 or 12. In other words, the higher the periodicity determination value, the higher the blending ratio of the LPF image signal. This makes it possible to obtain a reduced image with reduced moire as the output image signal.

[0073] It is also possible to improve the quality of the reduced image by using multiple LPFs with different cutoff frequencies depending on the area signal, without providing a blending unit 54, and inputting the output of the selected LPF to the reduction processing unit 56.

[0074] 15 shows an example of a processing unit 60 including an area signal generation unit 50 and an LPF 53. An input image signal is supplied to the area signal generation unit 50 and the LPF 53. The area signal generation unit 50 generates an area signal indicating an estimated moiré region as described above. The cutoff frequency of the LPF 53 is then changed in accordance with the area signal, and the output of the LPF 53 is used as an output image signal.

[0075] For example, for pixels that are determined not to be in the moiré estimation area by the area signal, a cutoff frequency of approximately the Nyquist frequency is used, and for pixels in the moiré estimation area, a lower cutoff frequency is used. By using this type of processing, it is possible to obtain an output image signal with reduced moiré.

[0076] 4. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0077] The image processing device 10 according to the embodiment includes a processing unit 60 that performs an area signal generation process for generating an area signal indicating the determination result of the stability and high frequency of the spatial frequency for each position within a frame of an image signal, and performs image processing on the image signal based on the area signal. An area within a frame that is determined to have a stable and high frequency spatial frequency can be evaluated as an area where moiré is likely to occur. In other words, the area signal can be used to indicate an area within a frame of an image signal where moiré is likely to occur, and by performing predetermined image processing on that area, image processing that distinguishes between areas where moiré occurs and areas where it does not occur can be realized.

[0078] In the embodiment, an example was given in which the image processing device 10 performs moiré reduction processing as image processing (see FIGS. 4 to 12 ). Since the area signal can determine the area where moiré is estimated to occur, the moiré reduction processing can be performed on that area. This makes it possible to reduce moiré while preventing the moiré reduction processing from affecting areas where moiré is not estimated to occur. Specifically, moiré reduction is possible in areas where moiré occurs without reducing resolution in areas where moiré does not occur. This makes it possible to reduce moiré while maintaining as much resolution as possible across the entire image. Furthermore, this method does not require computationally intensive processing, and it is also unnecessary for the user to specify the moiré range to reduce moiré in that area. This makes it suitable for real-time moiré reduction processing or moiré reduction processing on an edge device. Furthermore, because moiré reduction is performed through signal processing, the LPF characteristics and the degree of moiré reduction can be easily changed in real time. Furthermore, since the range of distance, angle, and focus at which moiré can be captured without worrying about moiré is expanded, incorporating the image processing device 10 into the imaging device 1 as shown in FIG. 1 improves the flexibility of imaging.

[0079] In the embodiment, the moiré reduction process involves blending an input image signal with an LPF image signal (band-limited image signal) obtained by band-limiting the input image signal using an LPF, based on an area signal. The LPF image signal band-limited by the LPF 53 is blended with the input image signal data by a blending unit 54. In this case, by using the area signal, the blending is performed so that the LPF image data ratio is high in the moiré estimation area and the input image data ratio is high in the other areas, thereby achieving moiré reduction while maintaining resolution sensitivity for the entire image.

[0080] In the embodiment, an example was given in which the area signal generation process extracts peaks from the spatial waveform of the input image signal, and generates an area signal containing information determining the stability and high frequency of the spatial frequency for each pixel using the period between surrounding peaks (see FIGS. 5 to 9 ). For example, the area signal contains a high-frequency period determination value for a pixel of interest that is considered to have a high spatial frequency, or the result of comparing the period determination value with a threshold. If the fluctuation in the period (peak-to-peak distance) between peaks in the spatial waveform of the input image signal is small, the spatial frequency can be evaluated as stable. Therefore, if the period fluctuation between surrounding peaks is small from the perspective of each pixel, it can be determined that the spatial frequency is stable, which is one of the conditions that makes moiré more likely to occur.

[0081] In the embodiment, an example has been given in which the area signal generation process extracts peaks after upsampling the spatial waveform of the input image signal (see FIGS. 5 to 9). By upsampling the spatial waveform of the input image signal, peak positions can be detected more accurately. This improves the accuracy of determining the stability of spatial frequencies.

[0082] In the embodiment, an example was given in which the area signal generation process extracts peaks from the spatial waveform of the input image signal, and generates an area signal indicating a period judgment value that indicates the stability of the period between surrounding peaks for pixels whose period of surrounding peaks is equal to or less than a predetermined value. If the periods of the peaks before and after the pixel of interest are greater than a predetermined value, it can be said that the spatial frequency is not high. Therefore, pixels whose period of surrounding peaks is equal to or less than a predetermined value can be said to be pixels in an area with a high spatial frequency. In this case, by calculating the period judgment value of the period of the surrounding peaks, it is possible to obtain area information that includes judgment information indicating that the spatial frequency within the frame is stable and high-frequency.

[0083] In the embodiment, an example of moiré reduction processing has been given in which the blending ratio of each pixel of the input image signal and the LPF image signal is set according to the information for each pixel contained in the area signal and blended. For example, as shown in Figure 11, this is processing in which the blending ratio of the LPF image signal is increased according to the period determination value. This makes it possible to perform blending according to the estimated degree of moiré occurrence.

[0084] In the embodiment, an example of moiré reduction processing has been given in which either the input image signal or the LPF image signal is selected for each pixel according to the information for each pixel contained in the area signal. For example, as shown in Figure 12, the period determination value is binarized to indicate whether or not moiré is estimated to occur. Then, a blending process is performed in which the LPF image signal is selected for pixels in areas where moiré is estimated to occur, and the input image data is selected for pixels in other areas. This also makes it possible to reduce moiré and maintain resolution.

[0085] In the embodiment, an example has been given in which a moiré warning image is synthesized with an image signal. For example, filtered image signals are blended and synthesized based on an area signal, and an output image signal is generated by synthesizing an image showing an area where moiré is estimated to occur (see FIG. 13 ). This makes it possible to clearly notify staff of the possibility of moiré occurrence. Note that the moiré warning is not limited to being generated by blending filtered image signals with input image signals; it is also possible to simply output a moiré warning message. For example, a moiré warning message may be synthesized with the input image signal.

[0086] In the embodiment, an example has been given in which detail processing is performed on an area other than an area determined to have a stable and high spatial frequency based on the area signal (see FIG. 13 ). For example, the detail processing signal is blended into an area where moiré is unlikely to occur based on the area signal. This makes it possible to apply detail processing only to areas where moiré is unlikely to occur.

[0087] In the embodiment, an example has been given in which detail processing is performed on an area determined to have a stable and high spatial frequency based on the area signal (see FIG. 13 ). For example, the detail processing signal is blended into an area where moiré is expected to occur based on the area signal. This makes it possible to output an effect image in which moiré is emphasized.

[0088] In the embodiment, an example was given in which an input image signal and an LPF image signal are blended based on an area signal, and the blended image signal is then reduced (see FIG. 14 ). This makes it possible to reduce the occurrence of moire when reducing an image, thereby enabling a high-quality reduced image to be output. In this case, it is preferable to set the cutoff frequency of the LPF 53 according to the reduction ratio, as this reduces moire and prevents unnecessary degradation of resolution.

[0089] In the embodiment, as an example of moiré reduction processing, processing for band-limiting the input image signal using an LPF that varies the cutoff frequency based on an area signal is given (see Fig. 15). This enables processing that sufficiently limits the band of the moiré estimation area using an LPF image and reduces the band-limiting in other areas, thereby realizing moiré reduction while maintaining resolution sensitivity for the entire image.

[0090] The image processing device 10 according to the embodiment, i.e., the image processing device equipped with the processing unit 60, can be incorporated not only into the imaging device 1 but also into an image editing device or the like. For example, it can be realized in any equipment that can acquire image signals and is equipped with a chip capable of image processing, such as a system camera, a camera adapter, a single-lens reflex camera, a compact digital camera, a cinema camera, a master monitor, a mobile phone, a CCU (camera control unit), a game console, etc. Note that the camera adapter referred to here refers to a camera peripheral device that is capable of image signal processing.

[0091] 4 to 15 is executed by, for example, a CPU, a DSP (digital signal processor), an AI processor, or an information processing device 70 including these. That is, the program of the embodiment is a program that generates an area signal indicating the determination result of the stability and high frequency of the spatial frequency for each position within the frame of the image signal, and causes the arithmetic processing device to execute a process of performing image processing based on the area signal.

[0092] By using such a program, the image processing device 10 according to the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other device capable of executing information processing.

[0093] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such a program can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0094] Furthermore, such a program is suitable for widely providing the image processing device 10 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, an imaging device, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can be equipped with the image processing device 10 of the present disclosure.

[0095] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0096] The present technology may also be configured as follows: (1) An image processing device including a processing unit that performs area signal generation processing to generate an area signal indicating a determination result of spatial frequency stability and high frequency for each position within a frame of an image signal, and image processing on the image signal based on the area signal. (2) The image processing device according to (1), wherein the image processing is moiré reduction processing. (3) The image processing device according to (2), wherein the moiré reduction processing is processing to blend the image signal and a band-limited image signal obtained by band-limiting the image signal using a low-pass filter, based on the area signal. (4) The image processing device according to any of (1) to (3), wherein the area signal generation processing extracts peaks from the spatial waveform of the image signal, and generates the area signal including information determining spatial frequency stability and high frequency for each pixel using a period between surrounding peaks. (5) The image processing device according to any one of (1) to (3), wherein the area signal generation process upsamples the spatial waveform of the image signal and extracts peaks, and generates the area signal including information determining the stability and high frequency of spatial frequency for each pixel using the period between surrounding peaks. (6) The image processing device according to any one of (1) to (3), wherein the area signal generation process extracts peaks from the spatial waveform of the image signal and generates the area signal including a period determination value indicating the stability of the period between surrounding peaks for pixels whose period is equal to or less than a predetermined value. (7) The image processing device according to (3), wherein the moiré reduction process is a process of blending the image signal and the band-limited image signal by setting a blending ratio for each pixel of the image signal and the band-limited image signal according to information for each pixel included in the area signal. (8) The image processing device according to (3), wherein the moiré reduction process is a process of selecting either the image signal or the band-limited image signal for each pixel according to information for each pixel included in the area signal. (9) The image processing device according to (2), wherein the image processing is processing for synthesizing a warning image for moire generation with the image signal.(10) The image processing device according to any one of (1) to (9), wherein the image processing comprises detail processing of an area of ​​the image signal that is not determined to have a stable and high spatial frequency based on the area signal. (11) The image processing device according to any one of (1) to (9), wherein the image processing comprises detail processing of an area of ​​the image signal that is determined to have a stable and high spatial frequency based on the area signal. (12) The image processing device according to any one of (1) to (11), wherein the image processing comprises blending the image signal with a band-limited image signal obtained by band-limiting the image signal using a low-pass filter based on the area signal, and reducing the blended image signal. (13) The image processing device according to (2), wherein the moiré reduction processing comprises band-limiting the image signal using a low-pass filter that varies its cutoff frequency based on the area signal. (14) An image processing method comprising generating an area signal indicative of a determination result of the stability and high frequency of the spatial frequency for each position within a frame of the image signal, and performing image processing based on the area signal. (15) A program for generating an area signal indicating the result of determining the stability and high frequency of spatial frequency for each position in a frame of an image signal, and causing a processor to execute image processing based on the area signal.

[0097] REFERENCE SIGNS LIST 1 Imaging device 10 Image processing device 51 Frequency estimation unit 52 Area determination unit 53 LPF 54 Blending unit 55 Filter processing unit 60 Processing unit

Claims

1. An image processing device having a processing unit that performs an area signal generation process that generates an area signal that indicates the determination result of the stability and high frequency of spatial frequency for each position within a frame of an image signal, and image processing on the image signal that is performed based on the area signal.

2. The image processing device according to claim 1, wherein the image processing is moire reduction processing.

3. The image processing device according to claim 2, wherein the moiré reduction processing is processing for blending the image signal and a band-limited image signal obtained by band-limiting the image signal using a low-pass filter, based on the area signal.

4. An image processing device as described in claim 1, wherein the area signal generation process extracts peaks from the spatial waveform of the image signal, and generates the area signal containing information determining the stability and high frequency of the spatial frequency for each pixel using the period between surrounding peaks.

5. An image processing device as described in claim 1, wherein the area signal generation process upsamples the spatial waveform of the image signal, extracts peaks, and generates the area signal containing information determining the stability and high frequency of the spatial frequency for each pixel using the period between surrounding peaks.

6. An image processing device as described in claim 1, wherein the area signal generation process extracts peaks from the spatial waveform of the image signal, and generates the area signal including a period judgment value indicating the stability of the period between surrounding peaks for pixels whose period of surrounding peaks is equal to or less than a predetermined value.

7. An image processing device according to claim 3, wherein the moiré reduction processing is a process of blending by setting a blending ratio for each pixel of the image signal and the band-limited image signal in accordance with information for each pixel contained in the area signal.

8. An image processing device according to claim 3, wherein the moiré reduction processing is processing for selecting either the image signal or the band-limited image signal for each pixel according to information for each pixel contained in the area signal.

9. An image processing device according to claim 2, wherein the image processing is a process of synthesizing a warning image of moire occurrence with the image signal.

10. An image processing device according to claim 1, wherein the image processing comprises performing detail processing on an area of ​​the image signal that is not determined to have a stable and high spatial frequency based on the area signal.

11. The image processing device according to claim 1, wherein the image processing comprises performing detail processing on an area of ​​the image signal that is determined to have a stable and high-frequency spatial frequency based on the area signal.

12. An image processing device according to claim 1, wherein the image processing comprises blending the image signal and a band-limited image signal obtained by band-limiting the image signal using a low-pass filter based on the area signal, and reducing the blended image signal.

13. The image processing device according to claim 2, wherein the moiré reduction processing is processing for band-limiting the image signal using a low-pass filter that varies the cutoff frequency based on the area signal.

14. An image processing method for generating an area signal indicating the result of determining the stability and high frequency of spatial frequency for each position within a frame of an image signal, and performing image processing based on the area signal.

15. A program that generates an area signal indicating the result of determining the stability and high frequency of spatial frequency for each position within a frame of an image signal, and causes a processing unit to execute image processing based on the area signal.

Images