Image processing apparatus and image processing method

Abstract

In an image processing device that controls an imaging means for capturing images of a subject that needs to be directly facing the subject, the simultaneous display of a UI indicating facing information and a UI indicating focus information on the display screen significantly reduces the visibility of the screen, preventing the user from correctly obtaining the information indicated by each UI. [Solution] The system is characterized by comprising: a first acquisition means for acquiring distance information corresponding to an image captured by an imaging means; a second acquisition means for acquiring focus information in a specified area of ​​the image; a first generation means for generating first information indicating whether the imaging direction of the imaging means is directly aligned with the subject based on the distance information; a second generation means for generating second information indicating the focus information of the subject; and a display control means for exclusively displaying the first and second information on a display means.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method that provide a user interface for assisting shooting by an imaging device.

Background Art

[0002] In recent years, it is expected that the number of facilities over 50 years old since construction will increase rapidly, and the importance of proper maintenance of buildings has been re-recognized. In the inspection of structures such as bridges and tunnels, the soundness is determined using deformation information such as concrete cracks and is utilized for maintenance. As a technique attracting attention for detecting cracks, there is a technique using AI (Artificial Intelligence). This is a technique in which an inspection target is photographed with high definition using a camera such as a single-lens reflex camera and a drone, the angle and tilt of the photographed image are corrected by image processing, and the cracks are detected by inputting the image into a crack detection network learned by deep learning. At this time, if the imaging direction of the camera is not directly facing the inspection target, cracks may not be detected, the length of the cracks may be detected as short, or the sufficient effect may not be exhibited. Patent Document 1 discloses that by displaying正对 information for making the imaging surface of the camera parallel to the subject as a UI (User Interface) on the display surface based on the result obtained by distance measurement, the direction for correcting the tilt is notified to the photographer in an easy-to-understand manner.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above-mentioned Patent Document 1, when shooting in the camera's autofocus mode, focus information indicating the degree of focus is displayed as a UI on the display surface. As a result, there is a problem that the visibility of the screen is significantly reduced because the UI indicating the orientation information (hereinafter referred to as the orientation UI) and the UI indicating the focus information (hereinafter referred to as the focus UI) are displayed simultaneously. Furthermore, there is a problem that the user cannot correctly obtain the information indicated by each UI because the focus UI and the orientation UI are displayed overlapping.

[0005] Therefore, the present invention aims to enable users to accurately obtain the information displayed by each UI while maintaining screen visibility by switching the display of the focus UI and the front-facing UI at an appropriate timing. [Means for solving the problem]

[0006] To solve the above problems, the present invention provides: The system is characterized by comprising: a first acquisition means for acquiring distance information corresponding to an image captured by an imaging means; a second acquisition means for acquiring focus information in a specified area of ​​the image; a first generation means for generating first information indicating whether the imaging direction of the imaging means is directly aligned with the subject based on the distance information; a second generation means for generating second information indicating the focus information of the subject; and a display control means for exclusively displaying the first and second information on a display means. [Effects of the Invention]

[0007] According to the present invention, by switching the display of the focus UI and the front-facing UI at an appropriate timing, the user can accurately obtain the information shown by each UI while maintaining screen visibility. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic block diagram showing an example of the hardware configuration of an image processing device, as an example of a display system. [Figure 2] This is a diagram showing a portion of the light-receiving surface of an image sensor. [Figure 3] This is a flowchart diagram for calculating distance information. [Figure 4] This figure shows the correlation calculation when the correlation between a pair of image signal sequences is high in an ideal state where no noise is present. [Figure 5] This figure shows the correlation calculation in a small block where noise is present. [Figure 6] This is a flowchart of the UI display timing control process in single autofocus mode in the first embodiment. [Figure 7] This figure shows an example of a display screen with the focus UI shown. [Figure 8] This figure shows an example of a display screen with a front-facing UI. [Figure 9] This figure shows an example of an assist display. [Figure 10] This is a flowchart illustrating the UI display timing control process in single autofocus mode in the second embodiment. [Figure 11] This is a flowchart illustrating the UI display timing control process in the continuous autofocus mode and video recording mode in the third embodiment. [Modes for carrying out the invention]

[0009] Specific examples of the present invention will be described in detail below with reference to the drawings.

[0010] First, let's describe the system that forms the basis of the present invention. Figure 1 is an internal configuration diagram of an image processing device 100, which is an example of an image processing device for carrying out the present invention. In Figure 1, the image processing device 100 is a device capable of image input, output, and even recording.

[0011] As shown in Figure 1, each component is connected to the internal bus 101, and each component can exchange data with each other via the internal bus 101.

[0012] The lens unit 106 is an element composed of a lens group including a zoom lens and a focus lens, an aperture mechanism, and a drive motor.

[0013] The optical image formed by the light beam passing through the lens unit 106 is received (light-received) by the imaging unit 107. The imaging unit 107 uses a CCD, a CMOS sensor, etc., and replaces an optical signal with an electrical signal. Since the electrical signal obtained here is an analog value, it also has an A / D conversion function for converting it into a digital value. In the present embodiment, the imaging unit 107 is a sensor equipped with an image plane phase difference function, and the details thereof will be described later.

[0014] The CPU 102 controls each part of the image processing apparatus 100 according to the program stored in the ROM 103, using the RAM 104 as a work memory.

[0015] The ROM 103 is a non-volatile recording element, and stores programs for operating the CPU 102, various adjustment parameters, etc.

[0016] The RAM 104 is a volatile memory using semiconductor elements. Generally, compared with the frame memory 111, those with lower speed and lower capacity are used.

[0017] The frame memory 111 is an element that can temporarily store an image signal and read it out when necessary. Since the image signal has a huge amount of data, those with high bandwidth and large capacity are required. In recent years, DDR4-SDRAM (Dual Data Rate 4-Synchronous Dynamic RAM), etc. are often used. By using this frame memory 111, for example, it becomes possible to perform processes such as synthesizing images that are different in time, or cutting out only necessary areas.

[0018] The image processing unit 105 performs various image processing operations on data from the imaging unit 107 or on image data stored in the frame memory 111 or recording medium 112, based on the control of the CPU 102. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, resizing, noise reduction, and color conversion of image data. The image processing unit 105 also corrects variations in the performance of pixels in the imaging unit 107, corrects defective pixels, corrects white balance, corrects brightness, and corrects distortion and vignetting caused by lens characteristics. In this embodiment, the image processing unit 105 also performs distance information generation, the details of which will be described later. At least a portion of the image processing unit 105 may be composed of dedicated circuit blocks for performing specific image processing. Furthermore, depending on the type of image processing, the CPU 102 may perform image processing according to a program without using the image processing unit 105.

[0019] Based on the calculation results obtained by the image processing unit 105, the CPU 102 controls the lens unit 106, allowing for optical adjustments such as image magnification, focal length, and aperture for adjusting light intensity. Furthermore, by moving a portion of the lens group along a plane perpendicular to the optical axis, image stabilization can also be performed.

[0020] As an interface with the outside of the device, there is first an operation unit 113 that accepts user input. The operation unit 113 uses mechanical elements such as buttons and switches, and consists of a power switch, a mode selector switch, etc.

[0021] The display unit 114 is a display device that can display images. For example, it can display images processed by the image processing unit 105, setting menus for operating various programs provided in the image processing device, and the operating status of the image processing device 100 can also be checked. The display unit 114 can utilize small, low-power devices such as LCDs (Liquid Crystal Displays) or organic ELs (Electroluminescence). Furthermore, in this embodiment, it is also equipped with a resistive or capacitive thin-film element called a touch panel, which can be used as part of the operation unit 113.

[0022] The CPU 102 generates strings to inform the user of the settings status of the image processing device 100, as well as menus for configuring the image processing device 100, and performs display control to display them on the display unit 114. Depending on the settings, each menu may be superimposed on the image processed by the image processing unit 105. In addition to text information, it is also possible to display (including superimposed on the image) shooting assist displays such as histograms, vectorscopes, waveform monitors, zebras, peaking, and false color according to the settings.

[0023] Other interfaces include a video terminal 109. In this embodiment, the system is configured to output image data and other data to an external monitor, etc., by providing interfaces such as SDI (Serial Digital Interface), HDMI (High Definition Multimedia Interface), and DisplayPort (registered trademark). The interfaces are not limited to those listed above and various other interfaces can be applied.

[0024] Furthermore, it is equipped with a network module 108 that can transmit not only images but also control signals. The network module 108 is an interface for inputting and outputting image and audio signals. The network module 108 can also communicate with external devices via the internet or other means to send and receive various data such as files and commands. The network module 108 may be wireless or wired.

[0025] The image processing device 100 not only outputs images to an external source but also has the function of recording them internally. The recording medium 112 can record image data and various setting data, and a large-capacity storage element is used. For example, an HDD (Hard Disk Drive) or SSD (Solid State Drive) is used and is mounted on the recording medium I / F 110.

[0026] The object detection unit 115 is a block for detecting objects using artificial intelligence, such as deep learning using a neural network. Taking deep learning-based object detection as an example, the CPU 102 transmits the processing program stored in the ROM 103, as well as network structures such as SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once), and weight parameters, to the object detection unit 115. Based on the various parameters obtained from the CPU 102, the object detection unit 115 performs processing to detect objects from the image signal and loads the processing results into the RAM 104.

[0027] The attitude detection unit 115 detects the attitude state of the image processing device 100 using signals obtained from, for example, a gyro sensor or an accelerometer. This makes it possible to detect whether the camera is tilted or shaking.

[0028] Figure 2 shows a portion of the light-receiving surface of the imaging unit 107.

[0029] The imaging unit 107 arranges pixel units in an array, each holding two photodiodes, which are light-receiving units, as photoelectric conversion means for a single microlens, in order to enable autofocus by detecting a phase difference on the imaging surface. This makes it possible for each pixel unit to receive the light beam divided from the exit pupil of the lens unit 106.

[0030] Figure 2(A) is a schematic diagram of a portion of the surface of an image sensor with an example of a Bayer array for red (R), blue (B), and green (Gb, Gr). Figure 2(B) is an example of a pixel section that holds two photodiodes as photoelectric conversion means for a single microlens, corresponding to the color filter array in Figure 2(A).

[0031] An image sensor with this configuration can output two signals for phase difference detection (hereinafter also referred to as the A image signal and the B image signal) from each pixel. It can also output an imaging signal (A image signal + B image signal) which is the sum of the signals from the two photodiodes. In the case of this summed signal, the output is equivalent to the output of the image sensor in the Bayer array example schematicly explained in Figure 2(A).

[0032] The imaging unit 107 can output a phase difference detection signal for each pixel, but it can also output a value obtained by averaging the phase difference detection signals of multiple adjacent pixels. Outputting an averaged value can shorten the time it takes to read signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.

[0033] The CPU 102 performs a correlation calculation on two phase-differential image signals (A image signal and B image signal) from the imaging unit 107 to calculate information such as the amount of defocus, parallax information, and various reliability values. The amount of defocus on the image plane is calculated based on the difference between the A image signal and the B image signal. The amount of defocus has positive and negative values, and whether the amount of defocus is positive or negative indicates whether the image is front-focused or back-focused. Furthermore, the absolute value of the amount of defocus indicates the degree to which the image is in focus, and if the amount of defocus is 0, the image is in focus. In this embodiment, the image is considered to be in focus when the amount of defocus is within a predetermined range. That is, the CPU 102 calculates information on whether the image is front-focused or back-focused based on the positive or negative value of the amount of defocus. Furthermore, it calculates focus degree information, which is the degree of focus (degree of focus deviation), based on the absolute value of the amount of defocus. Information on whether the image is front-focused or back-focused is output when the amount of defocus exceeds a predetermined value, and information indicating that the image is in focus is output when the absolute value of the amount of defocus is within the predetermined value.

[0034] Furthermore, the CPU 102 calculates the distance to the subject using the principle of triangulation based on the parallax information and the lens information of the lens unit 106.

[0035] Furthermore, Figure 2 shows an example in which pixel sections, each holding two photodiodes as photoelectric conversion means, are arranged in an array for a single microlens. However, it is also possible to arrange pixel sections, each holding three or more photodiodes as photoelectric conversion means, in an array for a single microlens. Alternatively, the microlens may have multiple pixel sections with different aperture positions for the light-receiving part. In other words, it is sufficient as long as two phase-difference detection signals, such as the A image signal and the B image signal, which can be detected by phase difference detection, are obtained as a result.

[0036] Next, the distance information generation process performed by the image processing unit 105 will be explained using Figures 3 to 5.

[0037] In step S301, the image processing unit 105 calculates the B-image signal for phase difference detection by finding the difference between the two signals output from the imaging unit 107: the (A-image signal + B-image signal) for imaging and the A-image signal for phase difference detection. Here, we will explain the method in which the (A-image signal + B-image signal) for imaging and the A-image signal for phase difference detection are output, but this method is not limited to this, and the imaging unit 107 may output the A-image signal and the B-image signal separately. In this case, the (A-image signal + B-image signal) for imaging can be calculated by adding the A-image signal and the B-image signal. Also, in the case of a stereo camera equipped with two sensors, the image signals from the output of each sensor may be used as the A-image signal and the B-image signal.

[0038] Next, in step S302, shading due to optical factors is corrected for both the A image signal and the B image signal for phase difference detection.

[0039] Next, in step S303, filtering is performed on both the A image signal and the B image signal for phase difference detection. The filtering is, for example, a high-pass filter composed of FIR (Finite Impulse Response). Here, we will explain using the A image signal and B image signal for phase difference detection that have passed through a high-pass filter, but this is not limited to this; signals may also be generated by passing them through band-pass filters or low-pass filters with different filter coefficients. Then, the correlation calculation process described later may be performed using the A image signal and B image signal for phase difference detection that have been generated.

[0040] Next, in step S304, the A image signal and B image signal for phase difference detection, which were filtered in step S303, are divided into minute blocks and a correlation calculation is performed. There are no restrictions on the size or shape of the minute blocks, and adjacent blocks may overlap in region.

[0041] The following describes the correlation calculation for a pair of images, image A and image B. The signal sequence of image A at the pixel of interest is denoted as E(1)~E(m), and the signal sequence of image B at the same pixel of interest is denoted as F(1)~F(m). The correlation amount C(k) between the two signal sequences is calculated using equation (1) while relatively shifting the signal sequence F(1)~F(m) of image B relative to the signal sequence E(1)~E(m) of image A. C(k)=Σ|E(n)-F(n+k)|···(1)

[0042] In equation (1), the Σ operation means calculating the sum over n. In the Σ operation, the range of n and n+k is limited to the range from 1 to m. The displacement amount k is an integer value and is the relative pixel displacement amount with the detection pitch of the pair of data as the unit. Figure 4 shows the result of the calculation of equation (1) when the correlation between the pair of image signal sequences is high in an ideal state where there is no noise. As shown in Figure 4, the correlation amount C(k) is minimized at the displacement amount where the correlation between the pair of image signal sequences is high (k=kj=0). Hereafter, the k at which the discrete correlation amount C(k) is minimized will be denoted as kj. The three-point interpolation process shown in equations (2) to (4) calculates x, which gives the minimum value C(x) for the continuous correlation amount. The pixel displacement amount x is a real value with the unit of pixels.

[0043]

number

[0044]

number

[0045] In equation (4), SLOP represents the smallest and most localized correlation and the slope of change in its adjacent correlations. Figure 4 shows a specific example: C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1) =1830 Let's assume that in this example, kj = 0. From equations (2) to (4), SLOP=830 x = -0.078 pixels This is the result.

[0046] When the image is in focus, the pixel displacement x between the signal trains of image A and image B is ideally 0.00.

[0047] On the other hand, Figure 5 shows the calculation results when equation (1) is applied to a small block containing noise. As shown in Figure 5, the correlation between the signal sequence of image A and the signal sequence of image B decreases due to the influence of randomly distributed noise. The minimum value of the correlation amount C(k) becomes larger than the minimum value shown in Figure 4, and the correlation curve takes on an overall flat shape (the absolute difference between the maximum and minimum values ​​is small).

[0048] In Figure 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1) = 1800 Let's assume that in this example, kj = 0. From equations (2) to (4), SLOP=500 x = -0.32 pixels This is the result.

[0049] Compared to the calculation results in the noise-free state shown in Figure 4, the pixel displacement x deviates from the ideal value.

[0050] When the correlation between a pair of image signal sequences is low, the change in the correlation amount C(k) becomes small, and the correlation curve becomes generally flat, resulting in a small SLOP value. Similarly, when the subject image has low contrast, the correlation between a pair of image signal sequences also becomes low, and the correlation curve becomes flat. Based on this property, the reliability of the calculated pixel shift amount x can be judged by the SLOP value. That is, if the SLOP value is large, it can be judged that the correlation between the pair of image signal sequences is high, and if the SLOP value is small, it can be judged that no significant correlation was obtained between the pair of image signal sequences. In this embodiment, since equation (1) was used for the correlation calculation, the correlation amount C(k) is smallest and local minimum at the shift amount where the correlation between the pair of image signal sequences is highest. Alternatively, a correlation calculation method may be used in which the correlation amount C(k) is maximum and local maximum at the shift amount where the correlation between the pair of image signal sequences is highest.

[0051] Next, in step S305, the confidence level is calculated. As mentioned above, this can be defined by the C(kj) value, which represents the degree of agreement between the two images calculated in step S304, and the SLOP value.

[0052] Next, interpolation is performed in step S306. Although correlation calculation was performed in step S304, the pixel displacement amount calculated in step S305 may not be usable due to low reliability. In such cases, it is necessary to interpolate from the pixel displacement amount calculated in the surrounding area. Interpolation methods include applying a median filter, or reducing the pixel displacement amount data and then re-enlarging it. Alternatively, color data may be extracted from the (A image signal + B image signal) used for imaging, and the pixel displacement amount may be interpolated using the color data.

[0053] Next, in step S307, the defocus amount is calculated by referring to the amount x calculated in step S304. Specifically, the defocus amount (denoted as DEF) can be calculated using the following formula (5). DEF = P·x ···(5)

[0054] In equation (5), P is a conversion coefficient determined by the detection pitch (pixel arrangement pitch) and the distance between the projection centers of the left and right viewpoints in a pair of disparity images, and its unit is mm / pixel.

[0055] Next, in step S308, the distance is calculated from the defocus amount calculated in step S307. When Da is the distance to the subject, Db is the focal position, and F is the focal length, the following equation holds approximately.

[0056]

number

[0057]

number

[0058] If we let Db0 be the value of Db when DEF=0, then equation 7 is

[0059]

number

[0060] The relative distance is Da-Da', so from equations 7 and 8...

[0061]

number

[0062] Following the flowchart in Figure 3 described above, the pixel displacement, defocus amount, and distance to the subject can be calculated from the A image signal and B image signal used for phase difference detection. In this embodiment, this information related to phase difference is collectively referred to as distance information.

[0063] <First Embodiment> The following describes a first embodiment of the present invention. First, the UI display timing control process in the single autofocus mode performed by the image processing device 100 will be described using Figures 6 to 9. Here, the single autofocus mode is a mode in which, after the shutter release is half-pressed to focus on the subject, the focus is kept fixed even if the subject moves. In this embodiment, as an example, we assume a case where one side of a flat subject such as a concrete wall is captured within the camera's field of view.

[0064] Figure 6 is a flowchart illustrating the single autofocus mode performed by the image processing device 100. This flowchart is executed by each unit at the direction of the CPU 102 or the CPU 102 in live view mode, where the image processing unit 105 processes the image signals acquired by the imaging unit 107 sequentially capturing images, and the processed image data is sequentially displayed on the display unit 114. In live view mode, a focus UI, such as a frame positioned by the user or by subject detection processing by the image processing unit 105 or CPU 102, is superimposed on the image displayed on the display unit 114. The focus UI changes its display mode to show the position (area) and focus information, such as whether the subject in that position is in focus or how much it is out of focus. In step S601, the CPU 102 determines whether the release button, one of the operation units 113, has been half-pressed. If it is determined that the release button has been half-pressed, the process proceeds to step S602. On the other hand, if it is determined that the release button has not been half-pressed, the process returns to step S601. Furthermore, in the following steps, if the half-press of the shutter release button is released, the process returns to step S601.

[0065] Next, in step S602, the CPU 102 determines whether the area where the focus UI is displayed is in focus. If it is determined to be in focus, the process proceeds to step S603. On the other hand, if it is determined not to be in focus, the process proceeds to step S605. Next, in step S603, the CPU 102 displays a focus UI on the display unit 114 indicating that it is in focus. Figure 7(a) is an example of the display unit 114 with a focus UI 700 indicating that it is in focus displayed on it. Next, in step S604, the CPU 102 hides the focus UI displayed on the display unit 114. Here, it is hidden, but the size of the UI can be changed, the color can be changed, or it can be made transparent. On the other hand, in step S605, the CPU 102 displays a focus UI on the display unit 114 indicating that it is not in focus. The display position of the focus UI does not need to be in the center; it can be determined, for example, by touch operations or menu settings on the display unit 114 by the user.

[0066] In step S606, the CPU 102 controls the lens unit 106 and adjusts the position of the focus lens to achieve focus. Then, the process returns to step S602.

[0067] Next, in step S607, the CPU 102 determines whether the imaging direction of the imaging unit 107 of the image processing device 100 is directly aligned with the subject. If it is determined that they are directly aligned, the process proceeds to step S608. On the other hand, if it is determined that they are not directly aligned, the process proceeds to step S609. The determination of whether they are directly aligned can be made using various pieces of information. For example, distance information such as the amount of pixel shift, the amount of defocus, or the distance to the subject may be used. The distance information may be calculated based on the flow shown in Figure 3, or an externally mounted distance measuring sensor may be used.

[0068] Next, in step S608, the CPU 102 displays a face-to-face UI on the display unit 114 indicating that the imaging direction of the imaging unit 107 and the subject to be imaged are directly aligned. Figure 8(a) is an example of the display unit 114 with a face-to-face UI 800 displayed, indicating that it is directly aligned with the display unit 114. In this embodiment, the face-to-face UI 800 is displayed with four UIs arranged at the four corners, but this is not limited to this. For example, the number of UIs to be displayed is determined by touch operations or menu settings on the operation unit 113 and the display unit 114 by the user. On the other hand, in step S609, the CPU 102 displays a face-to-face UI on the display unit 114 indicating that it is not directly aligned. Figure 8(b) is an example of the display unit 114 with a face-to-face UI 801 displayed, indicating that it is not directly aligned with the display unit 114. In step S610, the CPU 102 displays an assist display on the display unit 114 to help the image processing device 100 and the subject to be directly aligned.

[0069] Figure 9 shows an example of the display unit 114 with assist displays 900 and 901 displayed. For example, assist display 900 shows a triangular icon that is split into two at the top and overlapping at the bottom, indicating that the image is in front focus relative to the focus plane. Assist display 901 shows a triangular icon that is split into two at the bottom and overlapping at the top, indicating that the image is in back focus relative to the focus plane. This allows the user to know how to move the image processing device 100 to face the subject directly. When the image is in focus, the assist display takes the form of overlapping triangular icons at the top, as shown in the four corners of Figure 8(a), so the user only needs to move the image processing device to face the subject directly until the assist display for the in-focus state is shown. In this embodiment, an assist display that represents the degree of focus in the form of a displayed icon was given as an example, but the display method is not limited to this, and pixel misalignment (difference in degree of focus) may be displayed as a heat map. The process then proceeds to step S611.

[0070] Next, in step S611, the CPU 102 determines whether the focus position relative to the focus plane deviates significantly from the threshold compared to the focus state determined in step S602. If it determines that it does not deviate significantly from the threshold, the process proceeds to step S613. On the other hand, if it determines that it deviates significantly from the threshold, the process proceeds to step S612. In this embodiment, steps S611 and S612 describe the process of readjusting the focus if the focus position deviates significantly from the threshold. However, the process is not limited to this; a warning indicating that the focus adjustment should be readjusted may be displayed on the display unit 114 and the process proceeds to step S613, or this step may be ignored and the process proceeds to step S613. In step S612, the CPU 102 hides the frontal UI displayed on the display unit 114. Here, it is hidden, but the UI may be made less conspicuous by changing its size, color, or transparency. After that, the process returns to step S602. Next, in step S613, the CPU 102 determines whether the release button, one of the operation units 113, has been fully pressed. If it determines that the release button has been fully pressed, the process proceeds to step S614. On the other hand, if it determines that the release button has not been fully pressed, the process returns to step S607.

[0071] In step S614, the CPU 102 hides all UI elements displayed on the display unit 114. It is not necessary to hide all UI elements; some may be hidden, or this step may be skipped and the process proceeds to the next step. While the UI elements are hidden here, their size, color, or transparency may also be changed. The process then terminates. Subsequent steps operate based on a well-known imaging process. Specifically, the image captured by the imaging unit 107 with imaging parameters set in response to the full press of the release button is subjected to various image processing steps by the image processing unit 105 and displayed on the display unit 114 and / or recorded on the recording medium 112.

[0072] As described above, in this embodiment, by switching the display of the focus UI and the facing UI at an appropriate timing and displaying them exclusively, the user can accurately obtain the information shown by each UI while ensuring screen visibility.

[0073] <Second Embodiment> The second embodiment of the present invention will now be described. In the first embodiment, an example in which the focus UI is displayed first and then the facing UI is displayed was explained using Figure 6, but in this embodiment, an example in which the facing UI is displayed first and then the focus UI is displayed will be described. First, the UI display timing control process in single autofocus mode performed by the image processing device 100 will be explained using Figure 10. The flow in Figure 10 is also executed in each part in live view mode by the CPU 102 or by instruction from the CPU 102, similar to Figure 6. In this embodiment, as an example, we assume a case in which a flat subject such as a concrete wall is photographed with one side within the camera's field of view. In this embodiment, the same reference numerals are used for the same flow as in the first embodiment, and redundant explanations are omitted.

[0074] In step S601, the CPU 102 determines whether the release button, one of the control units 113, has been half-pressed. If it determines that the release button has been half-pressed, the process proceeds to step S1001. On the other hand, if it determines that the release button has not been half-pressed, the process proceeds to step S607. Although not shown in the flowchart, if the half-press of the release button is released in the following steps, the process returns to step S601. In step S607, the CPU 102 determines whether the imaging direction of the imaging unit 107 of the image processing device 100 is directly aligned with the subject. If it determines that they are directly aligned, the process proceeds to step S608. On the other hand, if it determines that they are not directly aligned, the process proceeds to step S609. In step S608, the CPU 102 displays a "Direct Alignment UI" on the display unit 114 to indicate that they are directly aligned. After that, the process returns to step S601.

[0075] Meanwhile, in step S609, the CPU 102 displays a "Facing UI" on the display unit 114 indicating that the imaging direction of the imaging unit 107 of the image processing device 100 is not aligned with the subject. In step S610, the CPU 102 displays an assist display on the display unit 114 to align the image processing device 100 with the subject. Then, the process returns to step S601. Next, in step S1001, the CPU 102 changes the "Facing UI" displayed on the display unit 114 to hidden. If the "Facing UI" to be hidden was the "Facing UI" displayed in step S609, the process will proceed to the next step without the subject and the image processing device 100 being aligned, so a warning indicating that they are not aligned may be displayed on the display unit 114. Next, in step S602, the CPU 102 determines whether the area where the focus UI is displayed is in focus. If it is determined to be in focus, the process proceeds to step S603. On the other hand, if it is determined not to be in focus, the process proceeds to step S605.

[0076] Next, in step S603, the CPU 102 displays a focus UI on the display unit 114 indicating that the system is in focus. On the other hand, in step S605, the CPU 102 displays a focus UI on the display unit 114 indicating that the system is not in focus. In step S606, if the focus mode is set to AF (Auto Focus) mode, the CPU 102 automatically controls the lens unit 106 and adjusts the position of the focus lens to achieve focus. If it is in MF (Manual Focus) mode, the user manually adjusts the position of the focus lens so that the focus UI indicating focus is displayed. After that, the process returns to step S602.

[0077] Next, in step S1002, the CPU 102 determines whether the degree of alignment of the image processing device 100 with respect to the subject is significantly different from the degree of alignment determined in step S607. If it is determined that there is no difference, the process proceeds to step S613. On the other hand, if it is determined that there is a difference, the process proceeds to step S604. Here, the process of re-determining alignment when the alignment position is misaligned has been explained, but the process is not limited to this; a warning indicating that the alignment determination will be re-determined may be displayed on the display unit 114 and the process proceeds to step S613, or this step may be ignored and the process proceeds to step S613. In step S604, the CPU 102 hides the focus UI displayed on the display unit 114. After that, the process returns to step S601. Next, in step S613, the CPU 102 determines whether the release button has been fully pressed. If it is determined that the release button has been fully pressed, the process proceeds to step S614. On the other hand, if it is determined that the release button has not been fully pressed, the process returns to step S602. Next, in step S614, the CPU 102 hides all UI displayed on the display unit 114. Subsequent steps operate based on the well-known imaging process. That is, the image captured by the imaging unit 107 with imaging parameters set in response to the full pressing of the release button is subjected to various image processing by the image processing unit 105 and displayed on the display unit 114 and / or recorded on the recording medium 112.

[0078] As explained above, by switching between the focus UI and the front UI at the appropriate timing and displaying them exclusively, users can accurately obtain the information shown by each UI while maintaining screen visibility.

[0079] <Third Embodiment> The following describes a third embodiment of the present invention. First, the UI display timing control process in the servo autofocus mode and continuous autofocus mode performed by the image processing device 100 will be described using Figure 11. The flow in Figure 11 is the same as in Figure 6, and is executed in each part in live view mode by the CPU 102 or by instructions from the CPU 102. Here, the servo autofocus mode is a mode in which focusing is repeated while the shutter release is half-pressed. The continuous autofocus mode is a mode in which focusing is repeated at all times regardless of the shutter release state. In this embodiment, as an example, we assume a case where one side of a flat subject such as a concrete wall is captured within the camera's field of view. In this embodiment, the same reference numerals are used for the same flow as in the first embodiment, and redundant explanations are omitted.

[0080] In the flowchart of Figure 11, first, in step S601, the CPU 102 determines whether the shutter release button is half-pressed. If it determines that the shutter release button is half-pressed, the process proceeds to step S602. On the other hand, if it determines that the shutter release button is not half-pressed, the process returns to step S601. Although not shown in the flowchart, in the following steps, if the shutter release button is released, the process returns to step S601. In continuous autofocus mode, this step is ignored.

[0081] Next, in step S602, the CPU 102 determines whether the area where the focus UI is displayed is in focus. If it determines that it is in focus, the process proceeds to step S603. On the other hand, if it determines that it is not in focus, the process proceeds to step S605. Next, in step S603, the CPU 102 displays a focus UI on the display unit 114 indicating that it is in focus. On the other hand, in step S605, the CPU 102 displays a focus UI on the display unit 114 indicating that it is not in focus. In step S606, the CPU 102 controls the lens unit 106 and adjusts the position of the focus lens so that it is in focus. After that, the process proceeds to step S1101. Next, in step S1101, the CPU 102 determines whether to display the focus UI. If it determines to display it, the process proceeds to step S607. On the other hand, if it determines not to display it, the process proceeds to step S612. Various determination methods can be applied to this determination. For example, the gyroscope or accelerometer of the attitude detection unit 115 may be used to determine whether there is an attitude change in the image processing device 100, or the amount of change in the degree of alignment within the screen obtained from the information used in step S607 may be used as a trigger. Alternatively, the process may proceed to step S607 without making this determination and always display the alignment UI. Depending on this determination, there may be cases where the focus UI and the alignment UI are displayed on the screen simultaneously, and the focus UI and the alignment UI may overlap. In such cases, the user can decide which UI to prioritize by setting menus on the operation unit 113 and the display unit 114. Prioritization is not limited to this method, and the determination result in step S1101 may also be used. Depending on the determination result in this step and the determination result in step S607, this flow processing may be terminated and the process may proceed to a general imaging process. In step S612, the CPU 102 hides the alignment UI displayed on the display unit 114. Then, the process proceeds to step S613. Next, in step S607, the CPU 102 determines whether the image processing device 100 and the subject are facing each other directly. If it is determined that the objects are facing each other directly, proceed to step S608. On the other hand, if it is determined that the objects are not facing each other directly, proceed to step S609.Next, in step S608, the CPU 102 displays a face-to-face UI on the display unit 114 indicating that the camera is facing the subject. On the other hand, in step S609, the CPU 102 displays a face-to-face UI on the display unit 114 indicating that the camera is not facing the subject. In step S610, the CPU 102 displays an assist display on the display unit 114 to help the image processing device 100 face the subject. Then, the process proceeds to step S613. Next, in step S613, the CPU 102 determines whether the release button has been fully pressed. If it is determined that the release button has been fully pressed, the process proceeds to step S614. On the other hand, if it is determined that the release button has not been fully pressed, the process returns to step S601. This step is ignored in continuous autofocus mode. Next, in step S614, the CPU 102 hides all UIs displayed on the display unit 114. Subsequent steps operate based on the well-known imaging process. In other words, the image captured by the imaging unit 107 with imaging parameters set in response to the full pressing of the release button is subjected to various image processing by the image processing unit 105 and displayed on the display unit 114 and / or recorded on the recording medium 112.

[0082] As explained above, by displaying the focused UI and the directly facing UI at the appropriate timing, users can accurately obtain the information shown by each UI while maintaining screen visibility.

[0083] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0084] Furthermore, the present invention also includes cases in which a software program that realizes the functions of the above-described embodiment is supplied directly from a recording medium or via wired / wireless communication to a system or device having a computer capable of executing the program, and the program is executed.

[0085] Therefore, in order to implement the functional processing of the present invention on a computer, the program code supplied to and installed on the computer itself also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0086] In that case, the form of the program is irrelevant, as long as it possesses the functionality of a program, including object code, programs executed by an interpreter, and script data supplied to the OS.

[0087] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0088] Another possible method for supplying the program is to store the computer program forming the present invention on a server on a computer network, and then have connected client computers download and run the computer program. [Explanation of symbols]

[0089] 100 Image Processing Devices 101 Internal Bus 102 CPU 103 ROM 104 RAM 105 Image Processing Unit 106 Lens Unit 107 Imaging Unit 108 Network Modules 109 Video terminals 110 Recording medium I / F 111 frame memory 112 Recording media 113 Operation section 114 Display section 115 Object detection unit 116 Posture detection unit 701 Focus UI indicating that focus is not achieved. 702 Focus UI indicating that the focus is achieved. 801 Direct orientation UI indicating that the user is not facing the object directly. 802 A UI that indicates a direct orientation. 901 Guide indicating a shift to the left 902 Guide indicating shift to the right

Claims

1. A first acquisition means for acquiring distance information corresponding to an image captured by an imaging means, A second acquisition means for acquiring focus information in a specified region of the aforementioned image, A first generating means that generates first information indicating whether the imaging direction of the imaging means is directly aligned with the subject based on the distance information, A second generation means for generating second information indicating the focus information of the subject, An image processing apparatus characterized by having a display control means for exclusively displaying the first and second information on a display means.

2. It has an operating section, The image processing apparatus according to claim 1, characterized in that the display control means switches the display of the first and second information in response to an operation on the operation unit and displays them on the display means.

3. The image processing apparatus according to claim 1, characterized in that the distance information is either the amount of defocusing or the distance to the subject.

4. The image processing apparatus according to claim 1, wherein the second information is generated based on the amount of defocus in the specified region, and is generated as information indicating a focused state when the amount of defocus is within a predetermined range.

5. The image processing apparatus according to claim 2, characterized in that the display of the first and second information is switched in response to an operation on the operation unit, and the imaging parameters of the imaging means are locked.

6. The image processing apparatus according to claim 1, characterized in that the first information is information indicating the degree of defocus for each region based on distance information of each region in the image.

7. A first acquisition step involves acquiring distance information corresponding to an image captured by an imaging means, A second acquisition step of acquiring focus information in a specified region of the aforementioned image, A first generation step of generating first information indicating whether the imaging direction of the imaging means is directly aligned with the subject based on the distance information, A second generation step of generating second information indicating the focus information of the subject, An image processing method characterized by comprising a display control step of exclusively displaying the first and second information on a display means.

Images