Image processing apparatus, method for controlling the same, and program

Abstract

To improve visibility in displaying brightness information of an image.SOLUTION: A system control unit 117 of an imaging apparatus 110 uses an image pickup device 111 to acquire an image signal of an optical image formed by a lens barrel 200 and acquire design information of the lens barrel 200, detects brightness information from the image signal, generates a histogram on the basis of the design information of the lens barrel 200 by using brightness information of pixels arranged in a first region where the optical image is formed on an imaging surface of the image pickup device 111, and displays the generated histogram on a display 114.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, a control method thereof, and a program. [Background technology]

[0002] Some imaging devices equipped with a display device such as a liquid crystal panel can display distribution information of luminance values of an imaging signal together with an image (captured image) obtained by an imaging element on the display device in order to change the white balance and exposure conditions during imaging. For example, Patent Document 1 describes a technique in which luminance information of an imaging signal is converted into a histogram and superimposed on the captured image for display on a display device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 7-38801 Summary of the Invention [Problem to be solved by the invention]

[0004] When luminance information of an imaging signal is displayed on a display device, such as with the technology disclosed in Patent Document 1, the distribution of luminance values of all pixels of an imaging element is generally displayed. In this case, there is no problem as long as the imaging surface of the imaging element is included in the optical image formed by the imaging optical system.

[0005] In contrast, there are cases where an optical image formed by an imaging optical system configured as a full-circumference (circumferential) fisheye lens is included within the imaging plane of an image sensor when viewed from the optical axis direction to capture an image of a subject. In this case, the luminance values of pixels located outside the optical image on the imaging plane become small, resulting in a condition known as blackout or vignetting. In such a situation, if luminance information is acquired over the entire imaging plane of the image sensor as in the past to generate a histogram, a high peak will appear on the low-luminance side, resulting in an extremely low luminance peak in the optical image. As a result, it becomes difficult to confirm the distribution of luminance values in the optical image, which can lead to problems such as making it difficult to set the exposure, for example.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide an image processing device that improves visibility when displaying luminance information of an image. [Means for solving the problem]

[0007] The image processing device of the present invention comprises an acquisition means for acquiring an image signal of an optical image formed by an imaging optical system and captured by an imaging element, and design information of the imaging optical system, a detection means for detecting luminance information from the image signal, and a control means for generating a histogram of the luminance information and displaying it on a display device, wherein the control means determines a first region in which the optical image is formed on the imaging surface of the imaging element based on the design information, and generates the histogram using luminance information of pixels in the first region. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the visibility when displaying the luminance information of an image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging system according to a first embodiment. [Figure 2] 2 is a cross-sectional view showing a schematic configuration of a lens barrel that constitutes the imaging system of FIG. 1. [Figure 3] 2 is a flowchart illustrating an operation of the imaging system of FIG. 1 during imaging. [Figure 4] 10A and 10B are diagrams showing examples of a display target image and a brightness histogram in the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating an example of a display target image and a brightness histogram according to a second embodiment. [Figure 6] 10A and 10B are diagrams illustrating an example of a display target image and a brightness histogram according to a third embodiment. [Figure 7] 10A and 10B are diagrams showing display examples of an image to be displayed and a brightness histogram in a fourth embodiment. [Figure 8] 10A and 10B are diagrams showing a display example of a display target image and a brightness histogram according to a conventional example. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described in detail with reference to the drawings, taking an imaging device as an example of an image processing device according to the present invention. In the following description, the imaging device refers to the main body of the imaging device equipped with an image sensor. However, for convenience, a device in which an imaging lens is configured integrally with the device main body, such as a compact digital camera, is also included in the imaging device. In the case of a single-lens reflex camera or a mirrorless single-lens camera in which a lens barrel (interchangeable lens) can be attached to or detached from the device main body, the configuration in which the lens barrel is attached to the imaging device will be referred to as an "imaging system."

[0011] First Embodiment 1 is a block diagram showing a schematic configuration of an image capturing system 100 according to the first embodiment. The image capturing system 100 is generally composed of an image capturing device 110 and a lens barrel 200.

[0012] The imaging device 110 includes an imaging element 111, an A / D conversion unit 112, an image processing unit 113, a display unit 114, an operation unit 115, a recording unit 116, a system control unit 117, a memory unit 118, an attitude detection unit 119, and a camera side mount (lens mount) 122.

[0013] The lens barrel 200 includes two imaging optical systems, a right-eye optical system 201R and a left-eye optical system 201L, a lens mount (camera mount) 202, a lens control unit 203, a storage unit 204, a temperature detection unit 205, and a focus detection unit 206.

[0014] The imaging device 110 is a so-called digital camera. The camera-side mount 122 is a component that is attached to and detached from a lens-side mount 202 provided on the lens barrel 200 using a bayonet engagement structure or the like. An incident light beam (light reflected from a subject) that passes through the lens barrel 200 forms an image on the imaging surface of the imaging element 111. The imaging element 111 is, for example, a CMOS sensor, and converts the optical image of the subject formed on the imaging surface into an image signal composed of an analog electrical signal and outputs the image signal to the A / D conversion unit 112. The A / D conversion unit 112 converts the analog electrical signal transmitted from the imaging element 111 into an image signal composed of a digital electrical signal and outputs the generated image signal to the image processing unit 113. The image processing unit 113 performs various image processing (so-called development processing) on the image signal transmitted from the A / D conversion unit 112 to generate image data. The system control unit 117 is a microcomputer composed of a CPU and memories such as ROM and RAM, and generally controls the operation of the imaging system 100.

[0015] The display unit 114 includes an electronic viewfinder and a rear monitor, and displays various types of information. Note that the "rear monitor" refers to a vari-angle, tilt, or fixed liquid crystal display device or organic EL display device provided on the rear of the image capture device 110. The operation unit 115 is a user interface such as switches and buttons that allow the user to give instructions to the image capture system 100, and a touch panel provided on the rear monitor. In addition, digital electrical signals that have been A / D converted by the A / D conversion unit 112 and stored in a memory (not shown) are converted into analog electrical signals in a D / A converter (not shown), and the signals are sequentially transferred to the display unit 114 or the like for display, thereby enabling a live view display (LV).

[0016] The recording unit 116 is, for example, a memory card or the like, and saves (records) various data including image files after image processing by the image processing unit 113. The attitude detection unit 119 is composed of an acceleration sensor, a gyro sensor, etc., and detects the attitude and movement of the imaging system 100. Based on the information detected by the attitude detection unit 119 about the attitude and movement of the imaging system 100, it is possible to estimate the change in attitude of the imaging system 100 during imaging in a time series manner.

[0017] The storage unit 118 stores identification information of the imaging device 110 (hereinafter referred to as "camera identification information") that was written during the manufacturing process of the imaging device 110. The camera identification information includes individual identification information of the imaging device 110 (model information, serial number, information about the number of pixels of the imaging element, information about the physical size of the imaging element, etc.) and manufacturing error information (information about individual differences such as the color and brightness of the imaging element 111, etc.). When the imaging system 100 is started up, the system control unit 117 reads out the camera identification information stored in the storage unit 118, and also obtains lens identification information of the lens barrel 200 (details will be described later) from the lens barrel 200 and sends it to the image processing unit 113. The image processing unit 113 generates an image file by adding the camera identification information and lens identification information to the image data of the captured image, and stores it in the recording unit 116. Note that the captured image refers to a captured still image or video.

[0018] When the lens barrel 200 is attached to the imaging device 110 by fitting the lens side mount 202 into the camera side mount 122, the system control unit 117 and lens control unit 203 are electrically connected. This allows power to be supplied from the imaging device 110 to the lens barrel 200, and enables mutual communication between the system control unit 117 and the lens control unit 203. The lens control unit 203 is a microcomputer composed of a CPU and memories such as ROM and RAM, and performs overall operational control of the lens barrel 200 under the control of the system control unit 117.

[0019] The right-eye optical system 201R and the left-eye optical system 201L are arranged so that the right-eye image and the left-eye image formed through each optical system are formed side by side along the long side of the image sensor 111. Note that the detailed configuration of the right-eye optical system 201R and the left-eye optical system 201L will be described later with reference to Fig. 2, and the imaging state of the optical image relative to the image sensor 111 will be described later with reference to Fig. 4 etc.

[0020] The temperature detection unit 205 detects the temperature around the lens barrel 200. The focus detection unit 206 has a magnetic, optical, or resistive position sensor, and detects focus information (focus lens position) of each of the right eye optical system 201R and the left eye optical system 201L.

[0021] The memory unit 204 stores lens identification information for the lens barrel 200. The lens identification information includes individual identification information and manufacturing error information. The individual identification information for the lens barrel 200 includes, for example, model information, a serial number, and optical design information (aperture value range, focal length, distortion, etc.). The manufacturing error information for the lens barrel 200 includes, for example, information about the tilt of the camera-side mount 122 relative to the image sensor 111.

[0022] When the lens control unit 203 is connected to the system control unit 117 so as to be able to communicate with it, the lens control unit 203 receives a command from the system control unit 117 and transmits the lens identification information stored in the storage unit 204 to the system control unit 117. The lens control unit 203 also transmits temperature information detected by a temperature detection unit 205 and focus information detected by a focus detection unit 206 to the system control unit 117 as appropriate.

[0023] 2 is a cross-sectional view showing a schematic configuration of the lens barrel 200, showing the cross-sectional configuration on a plane including the imaging optical axes of the right-eye optical system 201R and the left-eye optical system 201L. When the imaging system 100 is in a normal posture in the horizontal direction, the right-eye optical system 201R and the left-eye optical system 201L are provided in the lens barrel 200 separated by a predetermined distance.

[0024] The optical image formed by the right-eye optical system 201R can be recorded (saved) as a moving image or still image for the right eye, and the optical image formed by the left-eye optical system 201L can be recorded as a moving image or still image for the left eye. For example, when a moving image captured by the imaging system 100 is played back on a known 3D display or VR goggles so that an image for the right eye is projected on the viewer's right eye and an image for the left eye is projected on the viewer's left eye, images with parallax are projected onto the right and left eyes due to the base length D1 of the lens barrel 200. This allows the viewer to view a moving image with a three-dimensional effect. Note that the method for playing back captured images is not directly related to the present invention, so a description thereof will be omitted.

[0025] The right-eye optical system 201R and the left-eye optical system 201L each function as a full-circle fisheye lens, enabling imaging with an angle of view of 180 degrees or more. Since the right-eye optical system 201R and the left-eye optical system 201L have the same configuration, only the right-eye optical system 201R will be described here.

[0026] The photographing optical axis of the right-eye optical system 201R is formed by, in order from the subject side, a first optical axis OA1R, a second optical axis OA2R that is approximately perpendicular to the first optical axis OA1R, and a third optical axis OA3R that is parallel to the first optical axis OA1R. The light beam of the first optical axis OA1R is refracted by a first prism 220R and guided to the second optical axis OA2R, and the light beam of the second optical axis OA2R is refracted by a second prism 230R and guided to the third optical axis OA3. A first-group lens 211R having a convex surface facing the subject is arranged on the first optical axis OA1R, a second-group lens 221R is arranged on the second optical axis OA2R, and third-group lenses 231R and 232R are arranged on the third optical axis OA3R. The first-group lens 211R, the second-group lens 221R, and the third-group lenses 231R and 232R form a full-circle fisheye lens.

[0027] 2, the components of the left-eye optical system 201L are indicated by replacing the "R" added to the end of the components of the right-eye optical system 201R with an "L." The holding members and other components that hold the optical components such as lenses and prisms that make up the lens barrel 200 are not directly related to the present invention, and therefore will not be described here.

[0028] Next, a flow of operations performed by the imaging system 100 when capturing an image will be described. Fig. 3 is a flowchart illustrating operations performed by the imaging system 100 when capturing an image. Each process (step) indicated by an S number in Fig. 3 is realized by the CPU of the system control unit 117 loading a predetermined program stored in its own ROM into RAM, and controlling each unit constituting the imaging system 100 in an integrated manner in cooperation with the lens control unit 203. When the user turns on the power of the imaging device 110, the system control unit 117 and the lens control unit 203 communicate with each other to start the processing of S301.

[0029] In S301, the system control unit 117 acquires focus information and temperature information of the lens barrel 200 from the lens control unit 203.

[0030] As described above, the right-eye optical system 201R and the left-eye optical system 201L are configured as full-circle fisheye lenses. Full-circle fisheye lenses have a short focal length and a large depth of field, resulting in a configuration similar to a pan-focus system that does not require frequent focusing. However, when used with a small aperture value or due to manufacturing errors in the installation position of the image sensor 111 in the imaging device 110, focus adjustment may be desirable because of the possibility of out-of-focus. Furthermore, aberrations such as distortion may vary depending on the focal length between the right-eye optical system 201R and the left-eye optical system 201L. By acquiring distortion according to the focus position, for example, more accurate processing can be achieved when later converting a captured image into a VR image. For these purposes, focus information is acquired. Temperature information is acquired, for example, to correct manufacturing error information for the lens barrel 200 when later converting a captured image into a VR image using the manufacturing error information.

[0031] In S302 , the system control unit 117 acquires the lens identification information of the lens barrel 200 stored in the storage unit 204 from the lens control unit 203 .

[0032] The individual identification information of the lens identification information includes optical design information of the lens barrel 200. For example, the right eye optical system 201R and the left eye optical system 201L are arranged in the lens barrel 200 at a distance close to the interpupillary distance of a human being so that an appropriate parallax can be obtained when a captured image is converted into a VR image for viewing. For example, since the average interpupillary distance of a human being is said to be about 65 mm, the distance (baseline length (first optical axis distance) D1) between the forefront first group lenses 211R and 211L is designed to be 65 mm, and this information is included in the individual identification information.

[0033] Furthermore, the right-eye optical system 201R and the left-eye optical system 201L each have their optical paths refracted by first prisms 220R and 220L and second prisms 230R and 230L to form an optical image on one image sensor 111. Therefore, the distance between the third group lenses 231R and 232R and the third group lenses 231L and 232L (third optical axis distance D2) that form an image on the image sensor 111 is different from the distance between the forefront first group lenses 211R and 211L (first optical axis distance D1). If the image sensor 111 is a full-size sensor with a width of 36 mm, two approximately circular optical images must be projected onto it. Therefore, the third optical axis distance D2, which is the center-to-center distance between the two optical images, is generally set to approximately 18 mm, half the width of the image sensor 111, and this information is included in the individual identification information.

[0034] Because the first inter-optical axis distance D1 and the third inter-optical axis distance D2 are significantly different in this way, when converting a captured image into a VR image, it is necessary to appropriately determine the optical design values of lens barrel 200 in accordance with the captured image. Therefore, the lens identification information stored in storage unit 204 is notified to system control unit 117 and attached to the image data of the captured image.

[0035] Furthermore, because each individual lens barrel 200 contains manufacturing errors, the lens identification information includes manufacturing error information acquired during the manufacturing process. For example, in the lens barrel 200, the right-eye optical system 201R and the left-eye optical system 201L are ideally arranged parallel to each other, but due to component tolerances, assembly accuracy, and the like, there is a possibility that they may be slightly deviated from the ideal parallel state. Furthermore, there is a possibility that the focal length and distortion rate may differ considerably between the right-eye optical system 201R and the left-eye optical system 201L due to assembly errors in lens position. Such error information within the lens barrel 200 that occurs during the manufacturing process is stored in the storage unit 204 as manufacturing error information.

[0036] Furthermore, since manufacturing errors vary depending on the ambient temperature when the imaging system 100 is in use, the temperature dependency of the manufacturing errors of the lens barrel 200 is determined during manufacturing and stored in the storage unit 204. Therefore, temperature information of the lens barrel 200 is acquired in S301, and manufacturing error information of the lens barrel 200 is acquired in S302, and these are attached to the image data of the captured image. This makes it possible to correct the manufacturing errors when the captured image is later converted into a VR image. Note that, because the data indicating the correlation between the manufacturing errors and temperature is used when converting the captured image into a VR image, it does not necessarily need to be stored in the storage unit 204; it is sufficient if it can be acquired when converting the captured image into a VR image.

[0037] In S303, the system control unit 117 reads out the camera identification information stored in the storage unit 118.

[0038] In S304, the system control unit 117 checks the shooting mode. Note that the processes of S301 to S304 do not necessarily have to be performed in this order, but may be performed before the process of S305 starts.

[0039] In S305, the system control unit 117 acquires the orientation information of the imaging system 100 (imaging device 110) from the orientation detection unit 119. The orientation information of the imaging system 100 is attached to the image data of the captured image, and is later used when converting the captured image into a VR image.

[0040] In S306, the system control unit 117 controls the acquisition and display of a live view image, and displays the live view image on the display unit 114. If the system control unit 117 determines that there is no input from the operation unit 115 even after a certain time has elapsed since the start of acquisition of a live view image, it ends control of the acquisition and display of the live view image and enters standby mode. Return from standby mode is performed by a known method.

[0041] In S307, the system control unit 117 determines whether an instruction to start shooting has been received from the operation unit 115. In the case of still image shooting, it determines whether a release button belonging to the operation unit 115 of the imaging device 110 has been pressed, and in the case of video shooting, it determines whether a video recording button belonging to the operation unit 115 of the imaging device 110 has been pressed. After video shooting has started, the video recording button is used as an operating member that ends video shooting by being pressed again. If the system control unit 117 determines that an instruction to start shooting has been received (YES in S307), it executes the processing of S308, and if it determines that an instruction to start shooting has not been received (NO in S307), it continues to execute the processing of S306.

[0042] In S308, the system control unit 117 executes shooting. In the case of still image shooting, the still image data is stored in the recording unit 116 together with the information acquired in S301 to S305. In the case of still image shooting, the processing of S306 is actually executed after S308, but this route is not shown in FIG. 3. In the case of video shooting, the frame image data is stored in the recording unit 116 together with the information acquired in S301 to S305 until an instruction to end shooting is given (the video record button is pressed again). Note that in the case of video shooting, the processing of S301 and S305 is executed at regular time intervals while the processing of S308 is being executed.

[0043] In S309, the system control unit 117 determines whether or not an instruction to end shooting has been received from the operation unit 115. An instruction to end shooting essentially means turning off the power to the imaging system 100 in the case of still image shooting, and means pressing the video recording button again in the case of video shooting.

[0044] If the system control unit 117 determines that an instruction to end shooting has been received during still image shooting (YES in S309), it ends shooting, and if it determines that an instruction to end shooting has not been received (NO in S309), it executes the processing of S306.

[0045] If the system control unit 117 determines that an instruction to end video shooting has not been received (NO in S309), it continues to execute the process of S308 (not shown). Also, if the system control unit 117 determines that an instruction to end video shooting has been received (YES in S309), it executes the process of S306 (not shown), and when it receives an instruction to turn off the power of the imaging system 100, it ends this flow.

[0046] Next, we will explain how images are displayed on the display unit 114. Here, it is assumed that a histogram showing brightness information of an image to be displayed (hereinafter referred to as a "brightness histogram") is displayed on the display unit 114 together with a captured image or a live view video (hereinafter referred to as a "display target image") according to a pre-set setting.

[0047] First, a conventional display example will be described. Fig. 8(a) is a diagram showing a conventional display example of a display target image and a brightness histogram on the display screen of the display unit 114. It is assumed that the display screen of the display unit 114 is the display screen of a rear monitor provided on the rear side of the imaging device 110, and that the photographer is observing the display screen from the rear side of the imaging device 110.

[0048] As described above, the right-eye optical system 201R and the left-eye optical system 201L are both configured as full-circle fisheye lenses, and therefore, each optical system forms two substantially circular optical images (hereinafter referred to as "image circles") side by side on the left and right (longitudinal direction) of the image sensor 111. Here, when the optical image formed on the image sensor 111 is displayed on the display unit 114, the image processing unit 113 generally performs processing to rotate the entire optical image on the imaging surface by 180 degrees (inversion processing in the vertical and horizontal directions). Therefore, in FIG. 8(a), the display screen displays a right-eye image 801R representing the image circle formed by the right-eye optical system 201R on the left side, and a left-eye image 801L representing the image circle formed by the left-eye optical system 201L on the right side, side by side.

[0049] Note that image processing can be used to display the right-eye image 801R on the right side and the left-eye image 801L on the left side. However, in this embodiment, the arrangement of the right-eye image 801R and the left-eye image 801L does not matter, so it is assumed here that they are arranged as described above. Also, although the right-eye image 801R and the left-eye image 801L are described as still images, they may be live view images or moving images that are being recorded or played back.

[0050] As shown in FIG. 8(a), the entire area of the right-eye image 801R and the left-eye image 801L when they are displayed on the display screen is referred to as the "first area 801." At this time, low-brightness areas are generated at the four corners and above and below the center of the display screen where incident light from the lens barrel 200 is not irradiated and no optical image is displayed; these areas are hereinafter referred to as the "second area 802." Note that, on the actual display screen of the imaging device 110, the second area 802 is generally displayed as blacked-out (black), but in FIG. 8(a), the second area 802 is shown as colorless (the background color) because it is sufficient to be able to distinguish between the first area 801 and the second area 802.

[0051] A luminance histogram 803 is displayed on a portion of the display screen. FIG. 8(b) is an enlarged view of the luminance histogram 803. The luminance information indicates the distribution of luminance values of all pixels of the image sensor 111. Therefore, the luminance histogram 803 represents the distribution of luminance values that combines the luminance information of a region corresponding to a first region 801 (hereinafter referred to as the "first pixel region") on the imaging surface of the image sensor 111 and the luminance information of a region corresponding to a second region 802 (hereinafter referred to as the "second pixel region"). Ideally, the light beam that passes through the lens barrel 200 does not form an image (is not irradiated with light) on the second pixel region, resulting in a low luminance value. As a result, as shown in FIG. 8(b), a large peak appears on the low luminance value side, making it difficult to read the distribution of luminance values of the first region 801 corresponding to the right eye image 801R and the left eye image 801L. Therefore, in the first embodiment, this problem is solved, and it becomes possible to display a luminance histogram that represents the luminance information of the first pixel region (first region 801).

[0052] Fig. 4(a) is a diagram showing a display example according to the first embodiment of an image to be displayed and a brightness histogram on the display screen of the display unit 114. Note that the left eye image 401L, the right eye image 401R, the first region 401, and the second region 402 are the same as the right eye image 801R, the left eye image 801L, the first region 801, and the second region 802 in Fig. 8(a), respectively. As in Fig. 8, the second region 402 in Fig. 4 is displayed colorless (the color of the background).

[0053] The area and number of effective pixels of the imaging surface of the imaging element 111 are stored as camera identification information (individual identification information) in the storage unit 118 of the imaging device 110. In addition, the areas of two image circles (first pixel regions) that form images on the imaging surface of the imaging element 111 can be calculated from the lens identification information (optical design information and manufacturing error information) of the lens barrel 200. Therefore, the area ratio of the first pixel region to the imaging surface of the imaging element 111 can be calculated, and from this, the area ratio of the second pixel region to the imaging surface can be calculated. Then, the number of pixels in the second pixel region can be calculated by multiplying the number of effective pixels of the imaging element 111 by the area ratio of the second pixel region.

[0054] Therefore, the system control unit 117 first generates the luminance histogram 803 shown in Fig. 8. That is, the system control unit 117 collectively generates a histogram of the luminance information of the first pixel region and the second pixel region corresponding to the first region 401 and the second region 402, respectively. Next, the system control unit 117 deletes the luminance information for the number of pixels in the second pixel region from the luminance histogram 803, sequentially from the low luminance value side of the luminance histogram 803. As a result, only the luminance information of the pixels in the first pixel region remains, thereby obtaining the luminance histogram 403 that represents the distribution of luminance values in the first region 401, and displaying it on the display screen of the display unit 114.

[0055] 4(b) is an enlarged view of the luminance histogram 403. The luminance histogram 403 does not include luminance information of the low-luminance second region 402, and therefore does not have a large peak on the low-luminance side, making it possible to more clearly express the distribution of luminance values on the high-luminance side than the luminance histogram 803. This makes it easier for the user to recognize the distribution of luminance values in the first region 401, and thus makes it easier to adjust exposure, etc.

[0056] Furthermore, because signals are normally read from pixels of the image sensor 111 in the row direction (from top to bottom), read control becomes complicated when attempting to read only signals from pixels in the first pixel region. In contrast, with the above method, after signals from all pixels of the image sensor 111 are read using a normal read method, luminance information for the number of pixels in the second pixel region is deleted in order from the lowest luminance value side, thereby making it possible to reduce the computational load on the system control unit 117.

[0057] It is desirable that as much of the brightness histogram 403 as possible be displayed in the second region 402 so as not to obscure the first region 401, but in this case, it is more desirable that at least one of the left eye image 401L and the right eye image 401R be displayed so as not to obscure it. Therefore, in Fig. 4(a), the brightness histogram 403 is arranged so that the right eye image 401R is not obscured by the brightness histogram 403 and so that most of it is displayed in the second region 402.

[0058] A brightness histogram can also be displayed during live view image display, depending on the settings in the imaging device 110. In this case, the distribution of brightness values in the first region 401 can be displayed in the same manner as in the above embodiment, except that the brightness histogram 403 is updated based on frame images obtained at regular time intervals. The process for displaying a brightness histogram during video shooting is similar to the method for displaying a brightness histogram when an image displayed as live view is acquired. Note that the brightness histogram 403 is not limited to being updated at regular time intervals. For example, the brightness histogram 403 may be updated when a moving object in the subject scene is detected and the movement of the moving object is detected.

[0059] Second Embodiment In the second embodiment, the imaging system is configured by attaching a lens barrel (not shown) equipped with one imaging optical system configured with a full-circle fisheye lens to the imaging device 110. Therefore, one substantially circular optical image is formed on the imaging element 111.

[0060] 5 is a diagram showing a display example of a display target image and a brightness histogram according to the second embodiment on the display screen of the display unit 114. A first area 501 (display target image) corresponding to an image circle is displayed approximately in the center of the display screen of the display unit 114, and its periphery is a second area 502 (area not irradiated with incident light), which is displayed in black, for example. However, in FIG. 5, the second area 502 is displayed in a colorless manner, as in FIG. 4.

[0061] In the second embodiment, the number of pixels in the second pixel region is determined using a method similar to that of the first embodiment, but an area slightly smaller (for example, 85% to 95%) than the area of the image circle determined from the lens identification information is used. In other words, the number of pixels in the second pixel region is estimated slightly higher to create the brightness histogram 503. This makes it possible to prevent brightness information of pixels in the second pixel region from being reflected in the brightness histogram due to, for example, a relative positional shift between the imaging device 110 and the lens barrel, and allows the user to accurately know brightness information of the imaging range (field).

[0062] <Third embodiment> In the first and second embodiments, the number of pixels corresponding to the second region on the display screen is calculated, and the luminance information for the calculated number of pixels is deleted in order from the lowest luminance value side to generate a luminance histogram representing the luminance information of pixels corresponding to the first region on the display screen. In contrast, in the third embodiment, a predetermined rectangular region is set in the image circle formed on the imaging surface of the image sensor 111, and a luminance histogram is generated using the luminance information of the pixels within the set rectangular region.

[0063] 6 is a diagram showing a display example according to the third embodiment of an image to be displayed and a brightness histogram on the display screen of display unit 114. In the third embodiment, similar to the second embodiment, it is assumed that imaging device 110 is equipped with a lens barrel (not shown) equipped with one imaging optical system constituted by a full-circle fisheye lens, and one substantially circular optical image is formed on imaging element 111.

[0064] A first area 601 (image to be displayed) representing an image circle is displayed approximately at the center of the display screen of the display unit 114, and its periphery is a second area 602 (area not irradiated with incident light), which is displayed in black, for example. However, in Fig. 6, the second area 602 is displayed in a colorless manner, as in Fig. 5.

[0065] The size (diameter) of the image circle formed on the imaging surface of the image sensor 111 is assumed to be clear from the lens identification information. Furthermore, the center of the imaging surface of the image sensor 111 and the center of the image circle formed on the imaging surface by the imaging optical system generally coincide with each other. Therefore, the system control unit 117 sets, for example, a rectangular area that is similar in shape to the imaging surface and inscribed in the image circle or has the largest area within the image circle, reads signals from pixels within the set rectangular area, and generates and displays a brightness histogram 603. This allows the user to know the approximate brightness distribution in the first area 601.

[0066] At this time, a rectangular region 605 from which pixel signals have been read out may be displayed superimposed on the first region 601 on the display screen of the display unit 114. At this time, it is desirable to adjust the transmittance of the rectangular region 605 so that the image to be displayed in the first region 601 can be roughly confirmed through the rectangular region 605. This allows the user to confirm the image to be displayed in the first region 601 and easily understand which region of the first region 601 has its brightness information reflected in the brightness histogram 603.

[0067] A CMOS sensor is a typical example of the image sensor 111, and pixel signals in a CMOS sensor are generally read out in the row direction. Therefore, by making the area for reading out pixel signals within the image circle a rectangle consisting of two pairs of opposing sides of the CMOS sensor and two pairs of parallel sides, the calculation load on the system control unit 117 can be reduced.

[0068] The rectangular area inscribed in the image circle does not necessarily have to be similar in shape to the imaging surface of the image sensor 111, and can be, for example, a rectangle with the largest area inscribed in the image circle. Specifically, if the shape of the image circle can be considered to be a perfect circle, the rectangular shape with the largest area inscribed in the image circle will be a square. In this case, the proportion of the square area within the image circle is approximately 64% (2 / π=0.636...). Furthermore, the rectangle does not necessarily have to be the largest area as described above, as long as it has a sufficient area to determine the approximate luminance distribution within the image circle.

[0069] <Fourth embodiment> In the fourth embodiment, a modified example of the display example (FIG. 4) in the first embodiment will be described. FIG. 7(a) is a diagram showing a display example according to the fourth embodiment of an image to be displayed on the display screen of the display unit 114 and a brightness histogram. The left eye image 701L, right eye image 701R, first region 701, and second region 702 shown in FIG. 7(a) are the same as the right eye image 401R, left eye image 401L, first region 401, and second region 402 in FIG. 4(a), respectively.

[0070] One cause of a misalignment between the imaging optical axis of the lens barrel 200 and the center of the imaging surface of the image sensor 111 is mechanical looseness that occurs in the connection between the camera-side mount 122 and the lens-side mount 202. When the positional relationship between the lens barrel 200 and the image sensor 111 deviates from an ideal state, the pixel information of pixels located near the periphery outside the image circle is reflected in the brightness histogram instead of pixels located near the periphery inside the image circle corresponding to the first region 401. Furthermore, even if the positional relationship between the lens barrel 200 and the image sensor 111 is ideal, pixels located near the outer edge of the image circle outside the image circle corresponding to the first region 401 may not necessarily have low brightness values due to the influence of incident light. Furthermore, in the case of an image to be displayed in which so-called black crush occurs within the image circle due to underexposure or the like, the brightness values of pixels in the first pixel region corresponding to the first region 401 may be lower than the brightness values of pixels in the second pixel region corresponding to the second region 402.

[0071] In addition, in the brightness histograms in the first and second embodiments, brightness information on the low brightness value side is mechanically (automatically) deleted by the number of pixels corresponding to the second region 402 in the image sensor 111, so it is desirable to indicate to the user that such processing has been performed.

[0072] Therefore, in the fourth embodiment, the system control unit 117 generates a brightness histogram 703 that shows the area where brightness information that has been mechanically deleted in order from the low brightness value side would normally be displayed, in a manner that distinguishes it from the area showing the distribution of remaining brightness values, and displays it on the display unit 114.

[0073] 7(b) is an enlarged view of the luminance histogram 703. In the luminance histogram 703, the region that would normally display luminance information that has been mechanically deleted in order from the luminance value side is displayed as a subtraction region 704, distinguished from the region showing the distribution of remaining luminance values. The subtraction region 704 can be displayed, for example, by hatching or filling in with a transparent color.

[0074] Naturally, the subtraction region 704 is displayed on the low brightness value side of the brightness histogram 703. The user can know that the brightness information of the subtraction region 704 has been subtracted from the overall brightness information of the image sensor 111. This allows the user to recognize, for example, that although the user visually recognizes that there is an obviously dark region in the first region 701, the brightness value of that region may not be reflected.

[0075] Fifth Embodiment In the first to fourth embodiments, it is assumed that the number of pixels corresponding to the second region on the display screen can be obtained using the camera identification information and the lens identification information. However, there may be cases where the number of pixels in the second pixel region cannot be identified because the lens identification information cannot be obtained or the first pixel region cannot be determined from the lens identification information.

[0076] In such a case, if the fisheye lens is monocular as in the second embodiment, a luminance histogram is generated using the camera identification information and the number of pixels in the second pixel region calculated by regarding the circle with the largest diameter that contacts the long side of the imaging surface as the first pixel region. Also, if the fisheye lens is binocular as in the first embodiment, a luminance histogram is generated using the camera identification information and the number of pixels in the second pixel region calculated by regarding two circles with the same size and largest diameter that can be contained within the imaging surface as the first pixel region. In this way, a luminance histogram that roughly reflects the luminance information of the subject can be generated.

[0077] While the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0078] For example, the system control unit 117 may be held in a housing separate from the image sensor 111. Furthermore, in the above embodiment, the present invention has been described as being applied to an image sensor. This is because fisheye lenses have long been widely used as interchangeable lenses for image sensors. However, in recent years, clip-on fisheye lenses that can be attached to and detached from smartphone cameras have become commercially available. In light of this situation, it is clear that the present invention can be widely applied to electronic devices such as smartphones and tablet PCs that have an image sensor-based imaging function. When the first pixel area on the imaging surface of the image sensor 111 cannot be determined due to the optical specifications of the clip-on fisheye lens, the method of the fifth embodiment can be used.

[0079] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An image processing device comprising: an acquisition means for acquiring an image signal of an optical image formed by an imaging optical system and captured by an imaging element, and design information of the imaging optical system; a detection means for detecting luminance information from the image signal; and a control means for generating a histogram of the luminance information and displaying it on a display device, wherein the control means determines a first region in which the optical image is formed on the imaging surface of the imaging element based on the design information, and generates the histogram using luminance information of pixels in the first region. (Configuration 2) The image processing device described in Configuration 1, characterized in that the control means determines a second region on the imaging surface of the imaging element where the optical image is not formed based on the design information, calculates the number of pixels in the second region, and creates a histogram of the brightness values of all pixels of the imaging element, and then deletes brightness information for the number of pixels in the second region in order from the lowest brightness value side, thereby generating a histogram consisting of brightness information of pixels in the first region. (Configuration 3) The image processing device according to Configuration 2, wherein the control means displays on the histogram an area in which the luminance information deleted in order from the low luminance value side in the histogram would normally be displayed, in a manner that distinguishes it from an area showing the distribution of remaining luminance values. (Configuration 4) The image processing device according to configuration 3, wherein the area where the deleted luminance information would normally be displayed is displayed by hatching or filling in. (Configuration 5) An image processing device described in any one of configurations 1 to 4, characterized in that the first area is one or two approximately circular areas contained within the imaging surface when viewed from a direction perpendicular to the imaging surface. (Configuration 6) The image processing device described in Configuration 1, characterized in that the first region is approximately circular when viewed from a direction perpendicular to the imaging surface, and the control means generates the histogram using brightness information of pixels in a rectangular region provided within the first region. (Configuration 7) The image processing device according to configuration 6, wherein the rectangular area has a shape similar to the imaging surface of the imaging element or is square. (Configuration 8) The image processing device according to configuration 6 or 7, wherein the control means superimposes the rectangular area on the image captured by the imaging element and displays the image on the display device. (Configuration 9) The image processing device according to any one of configurations 1 to 8, wherein the control means superimposes the histogram on the image captured by the imaging element and displays the image on the display device. (Configuration 10) A program that causes a computer to function as each of the means of the image processing device according to any one of configurations 1 to 9. (Configuration 11) An imaging device comprising an imaging element that converts an optical image formed by an imaging optical system into an image signal, an acquisition means that acquires design information of the imaging optical system, a detection means that detects luminance information from the image signal, and a control means that generates a histogram of the luminance information and displays it on a display device, wherein the control means determines a first region in which the optical image is formed on the imaging surface of the imaging element based on the design information, and generates the histogram using luminance information of pixels in the first region. (Structure 12) An imaging system having an imaging device and a lens barrel detachable from the imaging device, wherein the lens barrel has an imaging optical system that guides incident light from a subject to the imaging device and a memory means that stores design information of the imaging optical system, and the imaging device has an imaging element that converts an optical image formed by the imaging optical system into an image signal, a display device, an acquisition means that acquires the design information from the lens barrel, a detection means that detects luminance information from the image signal, and a control means that generates a histogram of the luminance information and displays it on the display device, wherein the control means determines a first region where the optical image is formed on the imaging surface of the imaging element based on the design information, and generates the histogram using luminance information of pixels in the first region. (Configuration 13) The imaging system according to Configuration 13, wherein the lens barrel is a full-circle fisheye lens. (Method 1) A control method for an image processing device, comprising the steps of: acquiring an image signal of an optical image formed by an imaging optical system using an imaging element; acquiring design information of the imaging optical system; detecting luminance information from the image signal; determining a first region in which the optical image is formed on the imaging surface of the imaging element based on the design information; generating a histogram using luminance information of pixels in the first region; and displaying the histogram on a display device.

[0080] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0081] 100 Imaging System 110 Imaging device 111 Image sensor 114 Display section 117 System Control Unit 118 Storage section 200 lens barrel 201L Left eye optical system 201R Right eye optical system 203 Lens control unit 204 Storage section 401,501,601,701 First Area 402,502,602,702 Second Area 403,503,603,703 Luminance Histogram 605 Rectangular area 704 Subtraction Area

Claims

1. an acquisition means for acquiring an image signal of an optical image formed by an imaging optical system and captured by an imaging element, and design information of the imaging optical system; a detection means for detecting luminance information from the image signal; a control means for generating a histogram of the luminance information and displaying it on a display device; The image processing device is characterized in that the control means determines a first region in which the optical image is formed on the imaging surface of the imaging element based on the design information, and generates the histogram using luminance information of pixels in the first region.

2. The image processing device according to claim 1, characterized in that the control means determines a second region on the imaging surface of the image sensor where the optical image is not formed based on the design information, calculates the number of pixels in the second region, and creates a histogram of the brightness values of all pixels of the image sensor, and then deletes brightness information for the number of pixels in the second region in order from the lowest brightness value side, thereby generating a histogram consisting of brightness information of pixels in the first region.

3. The image processing device according to claim 2, characterized in that the control means displays on the histogram an area in which the luminance information deleted from the low luminance value side in the histogram would normally be displayed, in a manner that distinguishes it from an area showing the distribution of remaining luminance values.

4. 4. The image processing device according to claim 3, wherein the area where the deleted luminance information is originally displayed is displayed by hatching or filling.

5. 3. The image processing device according to claim 1, wherein the first area is one or two substantially circular areas that are included within the imaging surface when viewed from a direction perpendicular to the imaging surface.

6. the first region has a substantially circular shape when viewed from a direction perpendicular to the imaging surface, 2. The image processing apparatus according to claim 1, wherein the control means generates the histogram using luminance information of pixels in a rectangular area provided within the first area.

7. 7. The image processing device according to claim 6, wherein the rectangular area has a shape similar to the imaging surface of the imaging element or is square.

8. 8. The image processing apparatus according to claim 6, wherein the control means superimposes the rectangular area on the image captured by the image sensor and displays the image on the display device.

9. 3. The image processing apparatus according to claim 1, wherein the control means displays the histogram on the display device by superimposing it on the image captured by the image sensor.

10. acquiring an image signal of an optical image formed by an imaging optical system using an imaging element; acquiring design information of the imaging optical system; detecting luminance information from the image signal; determining a first region where the optical image is formed on an imaging surface of the imaging element based on the design information; generating a histogram using luminance information of pixels in the first region; and displaying the histogram on a display device.

11. A program that causes a computer to function as each of the means of the image processing apparatus according to claim 1 or 2.

12. an imaging element that converts an optical image formed by the imaging optical system into an image signal; an acquisition means for acquiring design information of the imaging optical system; a detection means for detecting luminance information from the image signal; a control means for generating a histogram of the luminance information and displaying it on a display device; the control means determines a first region in which the optical image is formed on the imaging surface of the image sensor based on the design information, and generates the histogram using luminance information of pixels in the first region.

13. An imaging device; an imaging system having a lens barrel detachable from the imaging device, The lens barrel comprises: an imaging optical system that guides incident light from a subject to the imaging device; a storage means for storing design information of the imaging optical system, The imaging device is an imaging element that converts an optical image formed by the imaging optical system into an image signal; a display device; an acquisition means for acquiring the design information from the lens barrel; a detection means for detecting luminance information from the image signal; a control means for generating a histogram of the luminance information and displaying it on the display device; the control means determines a first region in which the optical image is formed on the imaging surface of the image sensor based on the design information, and generates the histogram using luminance information of pixels in the first region.

14. 14. The imaging system according to claim 13, wherein the lens barrel is a full-circle fisheye lens.

Images