Image processing device

Abstract

To make it possible to display an assist display showing the features equivalent to the features of an image after distortion correction processing using an image before the distortion correction processing (without using the image after the distortion correction processing).SOLUTION: An image processing device has acquisition means for acquiring the frequency of each pixel value from an input image, correction means for correcting the frequency acquired by the acquisition means on the basis of the distortion of the input image, and generation means for generating an assist image indicating the frequency of each pixel value on the basis of the frequency after correction by the correction means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, and more particularly to image processing for assist display. [Background technology]

[0002] There is a technology called xR (cross reality). xR is a technology that merges the real world with the virtual world, and includes VR (virtual reality), AR (augmented reality), and MR (mixed reality).

[0003] When capturing images for xR, wide-angle lenses that are subject to distortion aberration, such as fisheye lenses, are often used to achieve a high level of immersion. When a square subject is captured using a fisheye lens, an image (fisheye image) in which the square subject is distorted into a barrel shape is obtained. When the captured image is displayed on a display device such as VR goggles, image processing (distortion correction processing) such as perspective projection conversion processing is performed to reduce image distortion. Patent Document 1 discloses a technology for reducing distortion in fisheye images. Patent Document 1 also discloses a technology for detecting a person from a fisheye image.

[0004] Also, during imaging, assist displays such as WFM (waveform monitor) display or peaking display may be performed. In WFM display, a WFM image (waveform monitor image) showing the distribution and frequency of the brightness values ​​of the image is displayed. By performing WFM display, it is possible to check the presence or absence of whiteout or blackout. In peaking display, the high frequency components of the image are emphasized. By performing peaking display, it is possible to check the focus and make it easier to perform the desired focus adjustment. When multiple images (multiple image regions) are obtained using multiple lenses, it is also possible to check the difference in focus between the multiple images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-61260 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the characteristics of an image before distortion correction (distorted image) are different from the characteristics of an image after distortion correction (image with reduced distortion). Therefore, if an assist display is performed using the image before distortion correction (without using the image after distortion correction) while the image after distortion correction is being displayed, the assist display shows characteristics different from the characteristics of the image being displayed (image after distortion correction). As a result, the user cannot easily grasp the characteristics of the image being displayed, or may grasp them incorrectly. If an image after distortion correction is used, an assist display showing the characteristics of the image can be performed, but the distortion correction process takes time and a load, which causes a delay in the display of the subject image in live view.

[0007] An object of the present invention is to enable assist display showing characteristics equivalent to those of an image after distortion correction processing, using an image before distortion correction processing (without using an image after distortion correction processing). [Means for solving the problem]

[0008] The image processing device of the present invention includes: an acquisition means for acquiring a frequency of each pixel value from an input image; and a correction means for correcting the frequency acquired by the acquisition means based on a distortion of the input image. The image forming apparatus further comprises a generating unit that generates an assist image indicating the frequency of each pixel value based on the frequency after correction by the correcting unit. Effect of the Invention

[0009] According to the present invention, it is possible to provide an assist display that shows the same characteristics as those of an image after distortion correction processing, by using an image before distortion correction processing (without using an image after distortion correction processing). [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an imaging device. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a plurality of divided regions. [Diagram 3] 3A to 3C are schematic diagrams showing examples of a perspective projection image, a fisheye image, and an assist image. [Figure 4] 3A and 3B are schematic diagrams showing an example of a subject region and a fisheye image region. [Diagram 5] FIG. 13 is a schematic diagram for explaining a method for determining a parameter n. [Figure 6] 3A to 3C are schematic diagrams showing examples of a perspective projection image, a fisheye image, and an assist image. [Figure 7] 1A to 1C are schematic diagrams showing an example of a perspective projection image, a fisheye image, and an assist display. [Figure 8] FIG. 11 is a schematic diagram showing an example of a transition of luminance values ​​in a fisheye image. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of an assist generation unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (First embodiment) Hereinafter, a first embodiment of the present invention will be described. An example of an imaging device having an assist display function will be described, but image processing for the assist display may be performed by an image processing device (for example, a tablet terminal, a personal computer, or a server) different from the imaging device.

[0012] 1 is a block diagram showing an example of the configuration of an imaging device 100 according to the first embodiment. The imaging device 100 has a lens unit 101, an imaging unit 102, an image processing unit 103, a recording unit 104, an image dividing unit 105, an assist generating unit 106, an image synthesis unit 107, a display unit 108, an operation unit 109, a CPU 110, a ROM 111, and a RAM 112.

[0013] The lens unit 101 has an optical system that forms a subject image on the imaging surface of the imaging unit 102. The lens unit 101 also has a zoom function, a focus adjustment function, and an aperture adjustment function. Although the lens used in the lens unit 101 is not particularly limited, a fisheye lens is used in the first embodiment. The lens unit 101 may or may not be detachable from the imaging device 100.

[0014] The imaging unit 102 captures an image of a subject. For example, the imaging unit 102 has a plurality of photoelectric conversion elements, and converts an image of a subject (optical image) formed by the lens unit 101 into an image signal. The imaging unit 102 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charged Coupled Device) image sensor.

[0015] The image processing unit 103 performs various image processing on the image signal output from the imaging unit 102. For example, the image processing unit 103 performs A / D conversion processing to convert the image signal, which is an analog signal, into image data, which is a digital signal, and performs flaw correction processing to reduce flaws caused by the imaging unit 102. In the first embodiment, it is assumed that YCbCr data in which each pixel value is a YCbCr value is used as the image data. The YCbCr value indicates a Y value which is a luminance value, a Cb value which is a color difference value, and a Cr value which is a color difference value.

[0016] The image processing unit 103 corrects distortion of a fisheye image (a distorted image captured using a fisheye lens). It is also possible to perform a distortion reduction process to reduce the distortion. The distortion reduction process is not particularly limited, but in the first embodiment, it is a perspective projection conversion process. An image after the perspective projection conversion process is referred to as a perspective projection image. The image processing unit 103 can output a fisheye image (image data of a fisheye image) or a perspective projection image (image data of a perspective projection image). A fisheye image can be output before the perspective projection conversion process, but a perspective projection image cannot be output unless after the perspective projection conversion. In the first embodiment, it is assumed that the image processing unit 103 outputs a fisheye image to the image division unit 105 in order to reduce a processing delay.

[0017] The recording unit 104 records the image data from the image processing unit 103 in a storage medium (recording medium) such as an SD card. The image recorded in the storage medium may be a fisheye image, a perspective projection image, or both a fisheye image and a perspective projection image.

[0018] The image dividing unit 105 divides the fisheye image (input image) output from the image processing unit 103 into a plurality of divided regions (a plurality of divided images) arranged in a lattice pattern. The number and arrangement of the divided regions are not particularly limited, but for example, as shown in Fig. 2, the image dividing unit 105 divides the fisheye image into 35 divided regions 201 to 235 arranged in a matrix of 7 rows and 5 columns.

[0019] The assist generator 106 uses a fisheye image (without using a perspective projection image) to generate an assist image indicating the frequency (frequency) of each pixel value. In the first embodiment, the assist generator 106 generates a WFM image (waveform monitor image) indicating the distribution and frequency of the luminance values ​​of the image as the assist image.

[0020] FIG. 3(A) is a schematic diagram showing an example of a perspective projection image, and FIG. 3(B) is a schematic diagram showing an example of a WFM image generated using the perspective projection image of FIG. 3(A). The horizontal axis of the WFM image of FIG. 3(B) indicates the horizontal position (horizontal position) of the perspective projection image of FIG. 3(A), and the vertical axis of the WFM image of FIG. 3(B) indicates the luminance value (Y value). The WFM image of FIG. 3(B) indicates the frequency of each luminance value in the perspective projection image (horizontal position) of FIG. 3(A) by light and dark (shading). For example, an area 301 in the WFM image of FIG. 3(B) corresponds to a window area in the perspective projection image of FIG. 3(A), and is shown bright because the frequency of the luminance value is high.

[0021] FIG. 3(C) is a schematic diagram showing an example of a fisheye image that is the source of the perspective projection image of FIG. 3(A), and FIG. 3(D) is a schematic diagram showing an example of a WFM image generated using the fisheye image of FIG. 3(C). In FIG. 3(D), the invalid area (black area outside the valid area showing the subject) is not considered. In a fisheye image, the image distortion increases as it moves away from the center, and the edges are compressed compared to the central part. Therefore, the WFM image of FIG. 3(D), which shows the frequency of each luminance value in the fisheye image, differs from the WFM image of FIG. 3(B), which shows the frequency of each luminance value in the perspective projection image. For example, since there are fewer pixels at the edge of the fisheye image than at the edge of the perspective projection image, the area 302 in the WFM image of FIG. 3(D) (the area corresponding to the window area in the fisheye image of FIG. 3(C)) is shown darker than the area 301 in FIG. 3(B) (reduced frequency).

[0022] Therefore, in the first embodiment, the frequency of each pixel value is corrected based on the distortion of the fisheye image (degree of distortion, distortion rate), and an assist image (WFM image) is generated based on the corrected frequency. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to display an assist display (display of a WFM image) that shows the same characteristics as the image after distortion correction processing.

[0023] The assist generation unit 106 includes an acquisition unit 106-1, a correction unit 106-2, and a generation unit 106-3. The acquisition unit 106-1 acquires the frequency (frequency information) of each pixel value (each luminance value) from the fisheye image (input image). The correction unit 106-2 generates the frequency information of each pixel value (each luminance value) from the acquisition unit 106-1 based on the distortion of the fisheye image. In the first embodiment, each of the acquisition unit 106-1 and the correction unit 106-2 performs processing on each of the multiple divided regions obtained by the image division unit 105. For example, the acquisition unit 106-1 acquires the frequency of each pixel value from the image in each of the multiple divided regions. The correction unit 106-2 corrects the frequency of each pixel value in each of the multiple divided regions based on the distortion of the image in the divided region. The generation unit 106-3 generates a WFM image based on the frequency after correction by the correction unit 106-2.

[0024] The image synthesis unit 107 synthesizes (superimposes) the assist image output from the assist generation unit 106 on the perspective projection image output from the imaging unit 102 to generate a synthesized image.

[0025] The display unit 108 displays the composite image output from the image composition unit 107. The display unit 108 is, for example, an EVF (Electronic View Finder) or a rear monitor. Note that the image to be displayed may be switchable between a perspective projection image and an assist image.

[0026] The operation unit 109 accepts various operations performed by the user on the imaging device 100. For example, the user performs various operations such as an operation to change the ISO sensitivity, an operation to change the shutter speed, an operation to switch between enabling and disabling the assist display, and an operation to change the type of assist display.

[0027] The CPU 110 is a control unit that controls the entire imaging device 100. The ROM 111 is a non-volatile memory that non-temporarily stores various data (information) such as image data, audio data, other data, and various programs for the operation of the CPU 110. The RAM 112 is a volatile memory that temporarily stores various data. For example, the CPU 110 controls each unit of the imaging device 100 according to the programs stored in the ROM 111, using the RAM 112 as a work memory.

[0028] When the CPU 110 receives an operation signal corresponding to an operation by a user (user operation) from the operation unit 109, the CPU 110 controls each unit of the imaging device 100 according to the operation signal. The CPU 110 also acquires information on the lens used to capture the fisheye image (input image). For example, the CPU 110 acquires information on the type of lens obtained from the lens unit 101. The CPU 110 then determines the distortion of the fisheye image from the acquired information and controls the processing of the assist generation unit 106 (correction unit 106-2). Note that if the lens used for imaging is fixed, the information on the distortion of the fisheye image can be determined in advance, so the CPU 110 may acquire the information on the distortion of the fisheye image without acquiring the lens information. The CPU 110 may then control the processing of the assist generation unit 106 (correction unit 106-2) based on the acquired information (information on the distortion of the fisheye image).

[0029] An operation example according to the first embodiment will be described with reference to FIGS.

[0030] 4(A) is a schematic diagram showing an example of a region of a subject (subject region). Each of the regions 401 to 403 is a region centered on a point 400, and the radius of the region 401, the distance from the outer edge of the region 401 to the outer edge of the region 402, and the distance from the outer edge of the region 402 to the outer edge of the region 403 are the same. The point 404 is a point on the outer edge of the region 401, the point 405 is a point on the outer edge of the region 402, and the point 406 is a point on the outer edge of the region 403.

[0031] 4B is a schematic diagram showing an example of a region (fisheye image region) of a fisheye image obtained by capturing the subject region of FIG. 4(A), and the point 410 corresponds to the point 400 in Fig. 4(A), and the regions 411 to 413 correspond to the regions 401 to 403 in Fig. 4(B), respectively. Due to image distortion, the distance from the outer edge of the region 411 to the outer edge of the region 412 is shorter than the radius of the region 411, and the distance from the outer edge of the region 412 to the outer edge of the region 413 is shorter than the distance from the outer edge of the region 411 to the outer edge of the region 412. The point 414 is a point on the outer edge of the region 411, the point 415 is a point on the outer edge of the region 412, and the point 416 is a point on the outer edge of the region 413.

[0032] Fig. 5 is a schematic diagram for explaining a method for determining a parameter n (magnification) for correcting the frequency of each pixel value (each luminance value). Point 500 on plane 510 corresponds to point 400 in Fig. 4(A) and point 410 in Fig. 4(B). Points 501 to 503 on plane 510 correspond to points 404 to 406 in Fig. 4(A), respectively, and points 504 to 506 on plane 510 correspond to points 414 to 416 in Fig. 4(B), respectively. Fig. 5 shows an example in which a fisheye lens 520 is used.

[0033] For example, CPU 110 obtains information about the lens used to capture the fisheye image and determines the radius f from the information. Then, CPU 110 calculates the angle θ between the optical axis of fisheye lens 520 and the imaging direction of point 501 (the direction passing through the center of curvature of fisheye lens 520 and point 501) from the radius f of fisheye lens 520 and the distance a from point 500 to point 501 using the following formula 1. θ=tan -1 (a / f)...(Formula 1)

[0034] The CPU 110 calculates the distance b from the point 500 to the point 504 using the following equation 2. b = f × sinθ = f × sin(tan -1 (a / f) ...(Formula 2)

[0035] Using the following equation 3, CPU 110 calculates angle δ between the optical axis of fisheye lens 520 and the imaging direction of point 502 (the direction passing through the center of curvature of fisheye lens 520 and point 502) from radius f of fisheye lens 520 and distance 2a from point 500 to point 502. δ=tan -1 (2a / f)...(Formula 3)

[0036] The CPU 110 calculates the distance c from the point 500 to the point 505 using the following equation 4. c = f × sin δ = f × sin(tan -1 (2a / f) ...(Formula 4)

[0037] The CPU 110 calculates the distance d from the point 504 to the point 505 using the following equation 5. d=cb = f × sin(tan -1 (2a / f) -f × sin(tan -1 (a / f) ...(Formula 5)

[0038] Then, CPU 110 calculates parameter n, which indicates the image distortion in area 412 in Fig. 4B, using the following equation 6. The larger the parameter n, the greater the image distortion. n=b / d =(f×sin(tan -1 (a / f))) / (f × sin(tan -1 (2a / f) -f × sin(tan -1 (a / f))) ...(Formula 6)

[0039] Similarly, CPU 110 calculates distance e from point 505 to point 506, and calculates parameter n=b / e indicating image distortion in region 413 in Fig. 4(B). CPU 110 determines parameter n=1 as parameter n indicating image distortion in region 411 in Fig. 4(B). CPU 110 determines parameter n=0 as parameter n corresponding to the invalid region of the fisheye image. The invalid region can be determined, for example, from information about the lens used to capture the fisheye image.

[0040] The number of regions for which the parameter n is calculated is not particularly limited, and may be more than 4. The regions for which the parameter n is calculated may be the same as or different from the divided regions obtained by the image dividing unit 105. Although an example in which a distance ratio is calculated as the parameter n has been described, an area ratio may be calculated as the parameter n.

[0041] The CPU 110 sets the determined multiple parameters n in the assist generation unit 106 (correction unit 106-2). For each of the multiple divided regions, the correction unit 106-2 multiplies the representative value (e.g., maximum value, minimum value, or average value) of the parameter n corresponding to the divided region by the frequency of each pixel value in the divided region. This increases the frequency of each pixel value in a region where image distortion is determined to be large (e.g., a region where the parameter n is greater than a threshold value). In the case of FIG. 2, for example, the frequency of the divided region 201 (region corresponding to parameter n=1) is not corrected, and the frequency of the divided regions 202 to 217 (regions corresponding to parameter n greater than 0 and less than 1) is increased based on the parameter n. Then, the frequency of the divided regions 226 to 235 (regions corresponding to parameter n=0) is corrected to 0. By using the frequency corrected in this way, a WFM image equivalent to the WFM image in FIG. 3(B) can be obtained from the fisheye image in FIG. 3(C). Note that the frequency of each pixel value may be reduced in an area where the image distortion is judged to be small (for example, an area where the parameter n is smaller than a threshold value). Either the process of increasing the frequency or the process of decreasing the frequency may be performed, or both may be performed.

[0042] As described above, according to the first embodiment, the frequency of each pixel value is corrected based on the distortion (degree of distortion, distortion rate) of the fisheye image, and an assist image (WFM image) is generated based on the corrected frequency. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to perform assist display (display of a WFM image) that shows characteristics equivalent to those of the image after distortion correction processing.

[0043] Although the example in which the input image is a fisheye image has been described, the input image may be another distorted image. For example, the input image may be an image captured using a lens other than a fisheye lens, an equirectangular image, or an equidistant projection image. Although the example in which the multiple divided regions are arranged in a lattice pattern has been described, the multiple divided regions may be a distorted image. The region may be a plurality of other regions. For example, the plurality of divided regions may be a plurality of regions arranged in a concentric circle. Although an example in which the assist image is a WFM image has been described, the assist image may be another image showing the frequency of each pixel value. For example, the assist image may be a histogram image showing a histogram of gradation values ​​(pixel values) such as luminance values.

[0044] Second embodiment A second embodiment of the present invention will be described below. In the first embodiment, a WFM image is generated as an assist image. In the second embodiment, a vectorscope image showing the distribution and frequency of combinations of hue and saturation is generated as an assist image. Note that, in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0045] FIG. 6(A) is a schematic diagram showing an example of a perspective projection image, and FIG. 6(B) is a schematic diagram showing an example of a vectorscope image generated using the perspective projection image of FIG. 6(A). The horizontal axis of the vectorscope image of FIG. 6(B) indicates the Cb value, and the vertical axis of the vectorscope image of FIG. 6(B) indicates the Cr value. The direction (axis) along the periphery of the vectorscope image (circular image) indicates the hue. When the position in the vectorscope image changes around the center of the vectorscope image in the direction along the periphery of the vectorscope image, the hue corresponding to the position changes. The distance from the center of the vectorscope image indicates the saturation, and the longer the distance, the higher the saturation. The vectorscope image of FIG. 6(B) indicates the frequency of each pixel value (each combination of Cb value and Cr value, each combination of hue and saturation) in the perspective projection image of FIG. 6(A) by lightness (shading). In Fig. 6B, plot 601 corresponds to red, plot 602 corresponds to magenta, plot 603 corresponds to yellow, plot 604 corresponds to blue, plot 605 corresponds to green, and plot 606 corresponds to cyan. In the perspective projection image in Fig. 6A, the size of the red area, the size of the magenta area, the size of the yellow area, the size of the blue area, the size of the green area, and the size of the cyan area are the same. Therefore, in the vectorscope image in Fig. 6B, plots 601 to 606 are shown with the same brightness.

[0046] FIG. 6(C) is a schematic diagram showing an example of a fisheye image that is the source of the perspective projection image of FIG. 6(A), and FIG. 6(D) is a schematic diagram showing an example of a vectorscope image generated using the fisheye image of FIG. 6(C). In FIG. 6(D), the invalid area is not considered. Plot 611 in FIG. 6(D) corresponds to red, plot 612 corresponds to magenta, plot 613 corresponds to yellow, plot 614 corresponds to blue, plot 615 corresponds to green, and plot 616 corresponds to cyan. In a fisheye image, the image distortion increases as it moves away from the center, and the edges are compressed compared to the center. Therefore, the vectorscope image of FIG. 6(D), which shows the frequency of each pixel value in a fisheye image, differs from the vectorscope image of FIG. 6(B), which shows the frequency of each pixel value in a perspective projection image. For example, in the vectorscope image of FIG. 6(B), plots 601 to 606 are shown with the same brightness, but in the vectorscope image of FIG. 6(D), plots 611 to 616 vary in brightness.

[0047] Therefore, in the second embodiment, the frequency of each pixel value is corrected based on the distortion (degree of distortion, distortion rate) of the fisheye image, and an assist image (vectorscope image) is generated based on the corrected frequency. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to perform an assist display (display of a vectorscope image) that shows the same characteristics as those of the image after distortion correction processing.

[0048] An operation example according to the second embodiment will be described. The CPU 110 determines a plurality of parameters n corresponding to a plurality of regions of the fisheye image, respectively, in the same manner as in the first embodiment, and sets the plurality of parameters n in the assist generation unit 106 (correction unit 106-2). The frequency of each pixel value is corrected in the same manner as in the first embodiment. By using the frequency corrected in this manner, a vectorscope image equivalent to the vectorscope image in Fig. 6(B) can be obtained from the fisheye image in Fig. 6(C).

[0049] As described above, according to the second embodiment, the frequency of each pixel value is corrected based on the distortion (degree of distortion, distortion rate) of the fisheye image, and an assist image (vectorscope image) is generated based on the corrected frequency. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to provide an assist display (display of a vectorscope image) that shows the same characteristics as those of the image after distortion correction processing.

[0050] Although the example in which the assist image is a vectorscope image has been described, the assist image may be another image showing the frequency of each pixel value. For example, the assist image may be a chromaticity diagram image showing the distribution and frequency of chromaticity (for example, an xy chromaticity diagram image showing an xy chromaticity diagram).

[0051] Third embodiment A third embodiment of the present invention will be described below. In the first embodiment (and the second embodiment), an assist image is displayed as the assist display. In the third embodiment, a peaking display that emphasizes high-frequency components of an image is performed as the assist display. In the peaking display, for example, the contour of a subject in focus is emphasized with a color. In the peaking display, the color applied to the high-frequency components may be different between a plurality of partial ranges that constitute the range of spatial frequencies corresponding to the high-frequency components. In the peaking display, the high-frequency components may be emphasized to a higher degree as the spatial frequency is higher. Note that, hereinafter, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and a description will be given of the points that are different from the first embodiment.

[0052] The assist generation unit 106 detects high-frequency components, which are frequency components higher than a threshold, from the fisheye image (for example, the pixel value (brightness value) of each pixel), and generates a high-frequency component image showing the detected high-frequency components. The high-frequency component image may be interpreted as an assist image. For example, the assist generation unit 106 detects high-frequency components using an HPF (high-pass filter). The detection condition for the high-frequency components (threshold defining the high-frequency components) may be changeable in response to a user operation. For example, the filter coefficient of the HPF may be changeable in response to a user operation. If the filter coefficient of the HPF changes, the cutoff frequency of the HPF (threshold defining the high-frequency components) also changes. In addition, the method of emphasizing high-frequency components in the peaking display may be changeable in response to a user operation. For example, the color applied to the high-frequency components may be changeable in response to a user operation.

[0053] FIG. 7(A) is a schematic diagram showing an example of a perspective projection image, and FIG. 7(B) is a schematic diagram showing an example of a peaking display using the perspective projection image of FIG. 7(A). FIG. 7(C) is a schematic diagram showing an example of a fisheye image that is the source of the perspective projection image of FIG. 7(A), and FIG. 7(D) is a schematic diagram showing an example of a peaking display using the fisheye image of FIG. 7(C). FIG. 8(A) and 8(B) are schematic diagrams showing an example of a transition of the brightness value of a fisheye image in the horizontal direction. The horizontal axis of FIG. 8(A) and 8(B) indicates the horizontal position, and the vertical axis of FIG. 8(A) and 8(B) indicates the brightness value (Y value). FIG. 8(A) shows a transition at the center of the fisheye image, and FIG. 8(B) shows a transition at the edge of the fisheye image. It is assumed that the transition in FIG. 8(A) and the transition in FIG. 8(B) are the same transition in a perspective projection image. However, in a fisheye image, the image becomes more distorted as it moves away from the center, and the edge is compressed compared to the center. Therefore, as shown in Figures 8(A) and 8(B), the transition of pixel values ​​is steeper at the edges than at the center, making it easier to detect high-frequency components. As a result, the subject (part) that is emphasized differs between the peaking display of Figure 7(D) using a fisheye image and the peaking display of Figure 7(B) using a perspective projection image. For example, in the peaking display of Figure 7(D), the subject (part) is emphasized at the edges of the fisheye image, whereas in the peaking display of Figure 7(B), the subject (part) is emphasized at the edges of the fisheye image. The non-emphasized subjects (the window frame and part of the chair) are emphasized.

[0054] Therefore, in the third embodiment, the detection conditions for high-frequency components (thresholds defining high-frequency components) are corrected based on the distortion of the fisheye image (degree of distortion, distortion rate), and the high-frequency components are detected based on the corrected detection conditions. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to perform assist display (peaking display) that shows characteristics equivalent to those of the image after distortion correction processing.

[0055] FIG. 9 is a block diagram showing an example of the configuration of the assist generating unit 106 according to the third embodiment. The assist generating unit 106 according to the third embodiment includes a detection unit 901, a correction unit 902, and a generation unit 903. The detection unit 901 detects high-frequency components, which are frequency components higher than a threshold, from a fisheye image (input image). For example, the detection unit 901 detects high-frequency components using an HPF. The correction unit 902 corrects detection conditions for high-frequency components (thresholds defining high-frequency components) based on distortion of the fisheye image. For example, the correction unit 902 corrects a filter coefficient of the HPF. In the third embodiment, each of the detection unit 901 and the correction unit 902 performs processing for each of a plurality of divided regions obtained by the image dividing unit 105. For example, the detection unit 901 detects high-frequency components from each of a plurality of divided regions. The correction unit 902 corrects detection conditions for each of a plurality of divided regions based on image distortion in the divided region. The generating unit 903 generates a high frequency component image indicating the high frequency components detected by the detecting unit 901 .

[0056] In the third embodiment, the image synthesis unit 107 synthesizes (superimposes) the high-frequency component image output from the assist generation unit 106 on the fisheye image output from the imaging unit 102 to generate a synthetic image, which is a fisheye image with emphasized high-frequency components. Note that image processing for emphasizing high-frequency components is not limited to synthesis of high-frequency component images. For example, the high-frequency components of the fisheye image may be emphasized by performing image processing for changing pixel values ​​on the fisheye image.

[0057] An operation example according to the third embodiment will be described. The CPU 110 determines a plurality of parameters n corresponding to a plurality of regions of a fisheye image, respectively, in the same manner as in the first embodiment, and sets the plurality of parameters n in the assist generation unit 106 (correction unit 902). The correction unit 902 corrects the detection condition (threshold defining the high frequency component) of the high frequency component in each of the plurality of divided regions based on a representative value (for example, a maximum value, a minimum value, or an average value) of the parameter n corresponding to the divided region. For example, the correction unit 902 raises the threshold in a region where the image distortion is determined to be large to make the detection condition in the region stricter, or lowers the threshold in a region where the image distortion is determined to be small to make the detection condition in the region relaxed. By using the detection condition corrected in this way, the peaking display in FIG. 7(E) can be performed using the fisheye image in FIG. 7(C). The subject (part) to be emphasized is the same in the peaking display in FIG. 7(E) using a fisheye image and the peaking display in FIG. 7(B) using a perspective projection image. It is possible to perform only one of the process of increasing the threshold value and the process of decreasing the threshold value, or to perform both.

[0058] As described above, according to the third embodiment, the detection conditions (thresholds defining high frequency components) are corrected based on the distortion of the fisheye image (degree of distortion, distortion rate), and the high frequency components are detected based on the corrected detection conditions. This makes it possible to use the image before distortion correction processing (without using the image after distortion correction processing) to perform assist display (peaking display) that shows characteristics equivalent to those of the image after distortion correction processing.

[0059] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment. .

[0060] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0061] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring a frequency of each pixel value from an input image; a correction means for correcting the frequency acquired by the acquisition means based on the distortion of the input image; A generating means for generating an assist image indicating the frequency of each pixel value based on the frequency after correction by the correcting means; 13. An image processing device comprising: (Configuration 2) The correction means increases the frequency of each pixel value in the area where the distortion is determined to be large. 2. The image processing device according to claim 1, (Configuration 3) The correction means reduces the frequency of each pixel value in the area where the distortion is determined to be small. 3. The image processing device according to configuration 1 or 2. (Configuration 4) The input image is an image captured using a fisheye lens. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) The input image is an equirectangular image. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 6) The input image is an equidistant projection image. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 7) The correction means corrects the frequency of each pixel value in each of a plurality of regions of the input image based on the image distortion in that region. 7. The image processing device according to any one of configurations 1 to 6. (Configuration 8) The plurality of regions are arranged in a lattice pattern. 8. The image processing device according to configuration 7. (Configuration 9) The plurality of regions are arranged concentrically. 8. The image processing device according to configuration 7. (Configuration 10) The camera further includes an acquisition unit for acquiring information about a lens used to capture the input image, The correction means corrects the frequency acquired by the acquisition means based on the distortion of the input image determined from the information. 10. The image processing device according to any one of configurations 1 to 9. (Configuration 11) The generating means generates the assist image by setting the frequency of each pixel value in an invalid area of ​​the input image to 0. 11. The image processing device according to any one of configurations 1 to 10. (Configuration 12) The assist image is a waveform monitor image showing the distribution and frequency of luminance values. 12. The image processing device according to any one of configurations 1 to 11. (Configuration 13) The assist image is a histogram image. 12. The image processing device according to any one of configurations 1 to 11. (Configuration 14) The assist image is a vectorscope image showing the distribution and frequency of combinations of hue and saturation. 12. The image processing device according to any one of configurations 1 to 11. (Configuration 15) The assist image is a chromaticity diagram image showing the distribution and frequency of chromaticity. 12. The image processing device according to any one of configurations 1 to 11. (Configuration 16) a detection means for detecting frequency components higher than a threshold from an input image; a correction means for correcting the threshold value based on a distortion of the input image; a processing means for performing image processing on the input image to enhance the high frequency components detected by the detection means; 13. An image processing device comprising: (Configuration 17) The correction means increases a threshold value in an area where the distortion is determined to be large. 17. The image processing device according to claim 16, (Configuration 18) The correction means reduces the threshold value in the area where the distortion is determined to be small. 18. The image processing device according to claim 16 or 17, (Configuration 19) The input image is an image captured using a fisheye lens. 19. The image processing device according to any one of configurations 16 to 18. (Configuration 20) The input image is an equirectangular image. 19. The image processing device according to any one of configurations 16 to 18. (Configuration 21) The input image is an equidistant projection image. 19. The image processing device according to any one of configurations 16 to 18. (Configuration 22) The correction means corrects a threshold value in each of a plurality of regions of the input image based on image distortion in that region. 22. The image processing device according to any one of configurations 16 to 21, (Configuration 23) The plurality of regions are arranged in a lattice pattern. 23. The image processing device according to claim 22, (Configuration 24) The plurality of regions are arranged concentrically. 23. The image processing device according to claim 22, (Configuration 25) The camera further includes an acquisition unit for acquiring information about a lens used to capture the input image, The correction means corrects the threshold value based on the distortion of the input image determined from the information. 25. The image processing device according to any one of configurations 16 to 24. (Method 1) An acquisition step of acquiring a frequency of each pixel value from an input image; a correction step of correcting the frequency acquired in the acquisition step based on the distortion of the input image; generating an assist image indicating the frequency of each pixel value based on the frequency after the correction in the correction step; 13. An image processing method comprising: (Method 2) a detection step of detecting frequency components higher than a threshold from the input image; a correction step of correcting the threshold value based on a distortion of the input image; a processing step of performing image processing on the input image to enhance the high frequency components detected in the detection step; 13. An image processing method comprising: (program) A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 25. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 25. [Explanation of symbols]

[0062] 100: Imaging device 106: Assist generation unit 107: Image synthesis unit 106-1: Acquisition unit 106-2: Correction unit 106-3: Generation unit 901: Detection unit 902: Correction unit 903: Generation unit

Claims

1. An acquisition means for acquiring a frequency of each pixel value from an input image; a correction means for correcting the frequency acquired by the acquisition means based on the distortion of the input image; A generating means for generating an assist image indicating the frequency of each pixel value based on the frequency after correction by the correcting means; 13. An image processing device comprising:

2. The correction means increases the frequency of each pixel value in the area where the distortion is determined to be large.

2. The image processing device according to claim 1,

3. The correction means reduces the frequency of each pixel value in the area where the distortion is determined to be small.

2. The image processing device according to claim 1,

4. The input image is an image captured using a fisheye lens.

2. The image processing device according to claim 1,

5. The input image is an equirectangular image.

2. The image processing device according to claim 1,

6. The input image is an equidistant projection image.

2. The image processing device according to claim 1,

7. The correction means corrects the frequency of each pixel value in each of a plurality of regions of the input image based on the image distortion in that region.

2. The image processing device according to claim 1,

8. The plurality of regions are arranged in a lattice pattern.

8. The image processing device according to claim 7,

9. The plurality of regions are arranged concentrically.

8. The image processing device according to claim 7,

10. The camera further includes an acquisition unit for acquiring information about a lens used to capture the input image, The correction means corrects the frequency acquired by the acquisition means based on the distortion of the input image determined from the information.

2. The image processing device according to claim 1,

11. The generating means generates the assist image by setting the frequency of each pixel value in an invalid area of ​​the input image to 0.

2. The image processing device according to claim 1,

12. The assist image is a waveform monitor image showing the distribution and frequency of luminance values.

2. The image processing device according to claim 1,

13. The assist image is a histogram image.

2. The image processing device according to claim 1,

14. The assist image is a vectorscope image showing the distribution and frequency of combinations of hue and saturation. It is an image 2. The image processing device according to claim 1,

15. The assist image is a chromaticity diagram image showing the distribution and frequency of chromaticity.

2. The image processing device according to claim 1,

16. a detection means for detecting frequency components higher than a threshold from an input image; a correction means for correcting the threshold value based on a distortion of the input image; a processing means for performing image processing on the input image to enhance the high frequency components detected by the detection means; 13. An image processing device comprising:

17. The correction means increases a threshold value in an area where the distortion is determined to be large.

17. The image processing device according to claim 16,

18. The correction means reduces the threshold value in the area where the distortion is determined to be small.

17. The image processing device according to claim 16,

19. The input image is an image captured using a fisheye lens.

17. The image processing device according to claim 16,

20. The input image is an equirectangular image.

17. The image processing device according to claim 16,

21. The input image is an equidistant projection image.

17. The image processing device according to claim 16,

22. The correction means corrects a threshold value in each of a plurality of regions of the input image based on image distortion in that region.

17. The image processing device according to claim 16,

23. The plurality of regions are arranged in a lattice pattern.

23. The image processing device according to claim 22.

24. The plurality of regions are arranged concentrically.

23. The image processing device according to claim 22.

25. The camera further includes an acquisition unit for acquiring information about a lens used to capture the input image, The correction means corrects the threshold value based on the distortion of the input image determined from the information.

17. The image processing device according to claim 16,

26. An acquisition step of acquiring a frequency of each pixel value from an input image; a correction step of correcting the frequency acquired in the acquisition step based on the distortion of the input image; generating an assist image indicating the frequency of each pixel value based on the frequency after the correction in the correction step; 13. An image processing method comprising:

27. a detection step of detecting frequency components higher than a threshold from the input image; a correction step of correcting the threshold value based on a distortion of the input image; a processing step of performing image processing on the input image to enhance the high frequency components detected in the detection step; 13. An image processing method comprising:

28. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 25.

29. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 25.

Images