Information processing apparatus and imaging apparatus
The information processing device improves image sensor alignment accuracy and efficiency by determining a subject area and generating alignment guidance, addressing the challenges of partial subject views and manual adjustments in existing imaging technologies.
Patent Information
- Application Number
- JP2023085676
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-12-05
AI Technical Summary
Existing imaging devices struggle with accurately aligning the image sensor with the subject, especially when the subject is only partially within the field of view, leading to decreased accuracy and increased time in manual adjustments.
An information processing device that determines a subject area within a captured image, acquires first information on the parallelism of the image sensor to the subject, and generates second information as a guide for the user to adjust the alignment, improving the accuracy and efficiency of sensor alignment.
Enhances the precision and speed of aligning the image sensor with the subject by providing real-time guidance, thereby improving the overall imaging process.
Smart Images

Figure 2024168751000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing apparatus that generates an image in which information serving as a UI (User Interface) is superimposed on an image obtained by imaging. [Background technology]
[0002] In the maintenance inspection of structures such as bridges and tunnels, there is a method that uses AI (Artificial Intelligence) to detect cracks in concrete. Specifically, the object to be inspected is photographed with an imaging device such as a digital camera, and the image obtained by the imaging is input to a crack detection network trained by deep learning to detect cracks. In this case, if the imaging device does not capture the image while the imaging element is facing the object to be inspected, cracks may not be detected or the length of the crack may not be detected accurately.
[0003] Patent Document 1 discloses a camera that displays, as a UI, facing information for a user to make an image sensor parallel to a subject based on a distance measurement result for the subject, and informs the user of the direction in which to correct the tilt of the image sensor. Patent Document 2 discloses an imaging device that evaluates the degree to which the image sensor faces a region specified by a user in a captured image, and controls the tilt angle of the image sensor based on the result. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Japanese Patent Application Publication No. 08-186757 [Patent Document 2] Patent No. 6679779 Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the imaging device of Patent Document 1, when the subject is included in only a part of the imaging angle of view, the imaging device displays facing information in an area that is not the subject. This reduces the accuracy of facing the imaging element to the subject. In addition, in the imaging device of Patent Document 2, the user must manually set the area for evaluating the facing degree every time imaging is performed, which takes time to capture an image.
[0006] The present invention provides an information processing device and an imaging device that can improve the accuracy of facing an imaging element to a subject. [Means for solving the problem]
[0007] An information processing device according to one aspect of the present invention is characterized by having a determination means for determining a subject area including the subject in a first image obtained by imaging the subject using an optical system and an imaging element, an acquisition means for acquiring first information that changes according to the parallelism of the imaging element to the subject at multiple locations in the subject area, and a control means for generating second information that serves as a guide for a user to adjust the parallelism according to the first information, and outputting a second image including an image of the subject area and the second information. Note that an imaging device including the above information processing device also constitutes another aspect of the present invention.
[0008] Also, an information processing method according to another aspect of the present invention includes the steps of: determining an object region including the object in a first image obtained by imaging the object using an optical system and an image sensor; acquiring first information at a plurality of locations in the object region, the first information varying according to the parallelism of the image sensor to the object; generating second information serving as a guide for a user to adjust the parallelism according to the first information; and outputting a second image including an image of the object region and the second information. Note that a program for causing a computer to execute processing according to the information processing method also constitutes another aspect of the present invention. Effect of the Invention
[0009] According to the present invention, the first information is acquired and the second information is generated from a limited area of the subject, thereby making it possible to improve the accuracy of facing the imaging element to the subject. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a part of a light receiving surface of an image sensor. [Diagram 3] 11 is a flowchart showing a process for calculating distance information in the embodiment. [Figure 4] FIG. [Diagram 5] FIG. [Figure 6] 6 is a flowchart showing a process of displaying facing information during autofocus in the first embodiment. [Figure 7] 5 is a flowchart showing a process of determining a subject area in the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of highlighting of a subject region in the first embodiment. [Figure 9] 11 is a flowchart showing a process of determining an arrangement of facing information in the first embodiment. [Figure 10] FIG. 4 is a diagram for explaining the arrangement of facing information in the first embodiment. [Figure 11] FIG. 4 is a diagram showing a display example of facing information in the first embodiment. [Figure 12] 13 is a flowchart showing a process of displaying facing information during autofocus in the second embodiment. [Figure 13] 13 is a flowchart showing a process of displaying a sub-window according to the third embodiment. [Figure 14] 13 is a flowchart showing a process for determining a sub-window display area in the third embodiment. [Figure 15] FIG. 13 is a diagram showing an example of a sub-window display in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, an image capturing device (digital camera) equipped with an information processing device that executes the processes of specific embodiments 1 to 3 described later will be described. Fig. 1 shows the configuration of the image capturing device 100. The image capturing device 100 captures images, processes the images, and records the images.
[0012] Connected to the internal bus 101 are a CPU 102, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, a video terminal 109, a recording medium I / F 110, a display unit 114, and an object detection unit 115. The units connected to the internal bus 101 can exchange data with each other via the internal bus 101.
[0013] The lens unit 106 has an optical system including a zoom lens, a focus lens, and an aperture mechanism, and a motor for driving the optical system. An optical image formed by the lens unit 106 is converted into an image signal as an analog electrical signal (i.e., imaged) by an image sensor such as a CCD sensor or a CMOS sensor included in the imaging unit 107. The imaging unit 107 also includes an A / D conversion unit that converts the image signal output from the image sensor into a digital signal (image data). The image sensor is an image plane phase difference sensor, and its details will be described later.
[0014] A CPU 102 as a computer operates according to a program stored in a ROM 103 using a RAM 104 as a work memory, thereby controlling each unit of the imaging device 100 .
[0015] The ROM 103 is a non-volatile memory, and stores programs for operating the CPU 102, various parameters, and the like. The RAM 104 is a volatile memory. The frame memory 111 is a memory for temporarily storing image data, and the image data can be read out when necessary. By using the frame memory 111, it is possible to perform processing such as combining images obtained by imaging at different times and cutting out a specific area from an image obtained by imaging.
[0016] The image processing unit 105 as an information processing device performs various image processing on image data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112 under the control of the CPU 102. The image processing performed by the image processing unit 105 includes pixel interpolation, encoding, compression, decoding, enlargement / reduction (resizing), noise reduction, color conversion, and the like of the image data. The image processing unit 105 also corrects performance variations among a plurality of pixels of the imaging element, corrects defective pixels, corrects white balance, corrects brightness, and corrects distortion and peripheral light loss caused by lens characteristics. The image processing unit 105 also performs processing to generate distance information, which will be described later. The image processing unit 105 may be configured as a dedicated image processing computer for performing image processing, or the CPU 102 may have the functions of the image processing unit 105.
[0017] The CPU 102 performs optical zoom, focusing, and aperture adjustment by controlling each motor of the lens unit 106 based on the calculation results for lens control obtained from the image processing unit 105. The CPU 102 may also perform image blur correction by moving a lens that constitutes a part of the optical system in a direction perpendicular to the optical axis according to camera shake detected by an attitude detection unit 116, which will be described later.
[0018] The operation unit 113 accepts an operation by a user and outputs an instruction corresponding to the operation to the CPU 102 and the image processing unit 105. The operation unit 113 has operation members such as a switch, a dial, and a button. The operation members include a power switch, a mode switching dial, and a release button. When the user presses the release button halfway, the release button instructs the CPU 102 to start an image capture preparation operation such as autofocus (AF) and exposure control (AE). When the user presses the release button all the way, the release button causes the CPU 102 to start an image capture operation for acquiring image data for recording. The operation unit 113 may be a touch sensor provided on the display unit 114.
[0019] The display unit 114 as a display means displays images processed by the image processing unit 105, setting menus, etc. The display unit 114 has a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display. The display unit 114 may also be provided with the above-mentioned touch sensor.
[0020] The CPU 102 generates text information and icons to inform the user of the setting status of the imaging device 100, as well as images of a setting menu for making various settings of the imaging device 100, and displays these on the display unit 114 by superimposing them on the image processed by the image processing unit 105.
[0021] The video terminal 109 is a video interface terminal such as SDI (Serial Digital Interface), HDMI (High Definition Multimedia Interface) (registered trademark), DisplayPort (registered trademark), etc. The CPU 102 can output video (image data) to an external monitor or the like via the video terminal 109.
[0022] The network module 108 is an interface for inputting and outputting image data and audio signals. The CPU 102 communicates with external devices via the network module 108 over the Internet or the like, and can transmit and receive not only image data and audio signals, but also control signals such as commands. The network module 108 may be of a wireless communication type or a wired communication type.
[0023] The recording medium 112 is a hard disk drive (HDD) or a solid state drive (SSD) capable of recording image data and various setting data, and is detachably attached to the recording medium I / F 110.
[0024] The object detection unit 115 detects an object using artificial intelligence, such as deep learning using a neural network. In object detection using deep learning, the CPU 102 transmits a program, a network structure, weight parameters, and the like stored in the ROM 103 to the object detection unit 115. Examples of the network structure include SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once). The object detection unit 115 performs a process of detecting an object from image data based on various parameters obtained from the CPU 102, and transmits the process result to the RAM 104.
[0025] The attitude detection unit 116 detects the attitude and shake of the imaging device 100 using an angular velocity sensor, an acceleration sensor, etc. This makes it possible to detect the tilt of the imaging device 100 and camera shake due to hand shake or the like.
[0026] FIG. 2A is a schematic diagram of a part of an image sensor in which red (R), blue (B), and green (Gb, Gr) pixels are arranged in a Bayer array. As shown in FIG. 2B, each pixel has two photodiodes A and B as photoelectric conversion elements, and the two photodiodes A and B are provided with one microlens (not shown). With this configuration, the two photodiodes A and B receive light from two different regions of the exit pupil of the lens unit 106 and output A and B signals. The imaging unit 107 generates a pair of image signals (A and B image signals) for phase difference detection by synthesizing the A and B signals output from each of a plurality of pixels on the image sensor. In addition, an imaging signal for obtaining image data is generated by adding the A and B image signals.
[0027] In the imaging unit 107, in addition to outputting the A and B signals from each pixel, it is also possible to output a value obtained by averaging the A and B signals from multiple pixels that are close to each other. By outputting the averaging value, it is possible to shorten the time required to read out signals from the imaging unit 107 and reduce the bandwidth of the internal bus 101.
[0028] The CPU 102 performs correlation calculations on the pair of image signals to calculate the phase difference between them, and calculates information on the defocus amount from the phase difference. The defocus amount indicates the degree of defocus with respect to the in-focus state of the lens unit 106. The CPU 102 determines that the in-focus state is reached if the defocus amount is within a predetermined range including 0, and determines that the state is front-focused or back-focused if the defocus amount is positive or negative, respectively. The CPU 102 performs image plane phase difference AF by controlling a motor that drives a focus lens in the lens unit 106 according to the positive or negative defocus amount. The CPU 102 also calculates the reliability of the calculated phase difference. The CPU 102 further calculates the distance to the subject from the parallax between the pair of image signals and the optical information of the lens unit 106 using the principle of triangulation.
[0029] Although Fig. 2B shows a case where two photodiodes are provided for each pixel, the number of photodiodes for each pixel may be three or more. Also, the image sensor may be provided with a plurality of pixels having different aperture positions, and the plurality of pixels may be treated as one pixel for focus detection. In this way, the image sensor may be configured to obtain a pair of image signals for phase difference detection.
[0030] Next, the information generation process performed by the image processing unit 105 will be described with reference to the flowcharts of Figures 3 to 5. The image processing unit 105 functions as a determination unit, a generation unit, and a control unit.
[0031] The flowchart in Fig. 3 shows distance calculation processing that the image processing unit 105 executes according to a program in response to the start of imaging in the imaging unit 107. First, in step S301, the image processing unit 105 calculates (acquires) a B image signal by determining the difference between the A image signal and the imaging signal (A image signal + B image signal) output from the imaging unit 107. Note that in this step, the A image signal and the B image signal may each be acquired from the imaging unit 107. In this case, a signal obtained by adding the A image signal and the B image signal can be acquired as the imaging signal. Furthermore, in the case of a stereo camera having two lens units and two imaging elements, the A image signal may be acquired from one of the two imaging elements, and the B image signal may be acquired from the other.
[0032] Next, in step S302, the image processing unit 105 performs shading correction due to optical factors on the A and B image signals.
[0033] Next, in step S303, the image processing unit 105 performs filtering on each of the A and B image signals. The filtering is, for example, high-pass filtering using a finite impulse response (FIR) filter. Below, a case will be described in which the A and B image signals that have been subjected to high-pass filtering are used, but filtering may also be performed using a band-pass filter or low-pass filter with different filter coefficients.
[0034] Next, in step S304, the image processing unit 105 divides each of the A and B image signals that have been subjected to the filter processing in step S303 into minute blocks, and performs correlation calculation for each divided block. There are no particular restrictions on the size or shape of the minute blocks, and adjacent blocks may partially overlap each other.
[0035] The correlation calculation between the A and B image signals will be explained below. The signal sequence of the A image signal at a pixel of interest is assumed to be the A image signal sequence E(1) to E(m), and the signal sequence of the B image signal at a pixel of interest is assumed to be the B image signal sequence F(1) to F(m). While shifting the B image signal sequence F(1) to F(m) relatively with respect to the A image signal sequence E(1) to E(m), the correlation amount C(k) at the shift amount k between these paired image signal sequences is calculated using the following equation (1).
[0036]
number
[0037] In equation (1), the Σ calculation refers to an operation to calculate the sum for n. In the Σ calculation, the range of n and n+k is limited to the range from 1 to m. The amount of shift k is an integer value, and is the relative pixel shift amount in units of the detection pitch (pixel arrangement pitch) of the paired image signal strings. Figure 4 shows the calculation results of equation (1) when the correlation between the paired image signal strings is high in an ideal state without noise.
[0038] As shown in Fig. 4, at a shift amount where the correlation between paired image signal sequences is high (k = kj = 0), the correlation amount C(k) is minimum. The k at which the discrete correlation amount C(k) is minimum is denoted as kj. The pixel shift amount (phase difference) x that gives the minimum value C(x) for continuous correlation amounts is calculated by the three-point interpolation process shown in equations (2) to (4). The pixel shift amount x is a real value, and its unit is pixel.
[0039]
number
[0040]
number
[0041]
number
[0042] The SLOP in equation (4) represents the slope of the change in the correlation amount that is the smallest and local minimum and the correlation amount adjacent to it. C(kj) = C(0) = 1000 C(kj-1)=C(-1)=1700 C(kj+1)=C(1)=1830 In this example, kj=0. From equations (2) to (4), SLOP=830 x=-0.078pixel In the focused state, the pixel shift amount x between the paired image signal rows is ideally 0.00.
[0043] On the other hand, the calculation results when formula (1) is applied to a small block where noise is present are shown in Fig. 5. As shown in Fig. 5, the correlation between paired image signal sequences decreases due to the effect of randomly distributed noise. The minimum value of the correlation amount C(k) becomes larger than the minimum value shown in Fig. 4, and the correlation amount curve becomes generally flat (the absolute value of the difference between the maximum and minimum values becomes smaller).
[0044] In FIG. 5, as a specific example, C(kj) = C(0) = 1300 C(kj-1)=C(-1)=1480 C(kj+1)=C(1)=1800 In this example, kj=0. From equations (2) to (4), SLOP=500 x=-0.32pixel It becomes.
[0045] Compared to the calculation result in a noise-free state shown in FIG. 4, the pixel shift amount x is farther from the ideal value.
[0046] When the correlation between the pair of image signal strings is low, the change in the correlation amount C(k) is small and the correlation amount curve is generally flat, so the SLOP value is small. Similarly, when the subject image has low contrast, the correlation between the pair of image signal strings is low and the correlation amount curve is flat. Based on this property, the reliability of the calculated pixel shift amount x can be determined by the SLOP value. That is, when the SLOP value is large, it can be determined that the correlation between the pair of image signal strings is high, and when the SLOP value is small, it can be determined that no significant correlation has been obtained between the pair of image signal strings. In this embodiment, since formula (1) is used for the correlation calculation, the correlation amount C(k) is minimum and minimal at the shift amount k where the correlation between the pair of image signal strings is highest. Alternatively, a correlation calculation method in which the correlation amount C(k) is maximum and maximal at the shift amount k where the correlation between the pair of image signal strings is highest may be used.
[0047] Next, in step S305, the image processing unit 105 calculates the reliability of the calculated pixel shift amount x. The reliability can be defined by C(kj), which indicates the degree of match between the pair of image signals calculated in step S304, and the value of SLOP.
[0048] Next, in step S306, the image processing unit 105 performs an interpolation process. Although the correlation calculation could be performed in step S304, there are cases where the pixel shift amount x of the pixel of interest cannot be adopted due to low reliability calculated in step S305. In such cases, it is necessary to obtain the pixel shift amount of the pixel of interest by interpolation using the pixel shift amounts calculated for the pixels surrounding the pixel of interest. As a method of interpolation, a median filter may be applied to the pixel shift amount calculated for the surrounding pixels, or a method of reducing the pixel shift amount data and then enlarging it again may be used. Also, color data may be extracted from the imaging signal, and the pixel shift amount may be interpolated using the color data.
[0049] Next, in step S307, the image processing unit 105 calculates the defocus amount by referring to the pixel shift amount x calculated in step S304 or acquired by interpolation in step S306. The defocus amount (DEF) can be calculated by the following formula (5).
[0050]
number
[0051] In equation (5), P is a conversion coefficient determined by the detection pitch and the distance between the projection centers of the two viewpoints in the pair of parallax images (image signal A and image signal B) (the distance between the center of gravity of the image sensor generating image signal A and the center of gravity of the image sensor generating image signal B). The unit of the conversion coefficient is mm / pixel.
[0052] Next, in step S308, the image processing unit 105 calculates the distance from the defocus amount calculated in step S307. When the distance to the subject is Da, the focal position of the lens unit 106 is Db, and the focal length is F, the following formula (6) approximately holds.
[0053]
number
[0054] Therefore, the distance to the subject, Da, is
[0055]
number
[0056] It becomes.
[0057] If Db when DEF=0 is Db0, then equation (7) becomes:
[0058]
number
[0059] The absolute distance Da' to the subject can be calculated. The relative distance Da-Da' is calculated from equations (7) and (8) as follows:
[0060]
number
[0061] It can be found by:
[0062] By the above processing shown in FIG. 3, the pixel shift amount, defocus amount, and distance to the subject can be obtained from the A and B image signals.
[0063] The information on the focus state of the optical system thus obtained (pixel shift amount, defocus amount, etc.) or the information on the distance from the imaging element to the subject (the above-mentioned relative distance, etc.) is obtained as first information at multiple locations on the subject in Examples 1 to 3 described below. The first information obtained at multiple locations changes depending on the degree of parallelism of the imaging section (imaging surface of the imaging element) 107 with respect to the subject. The first information is used to generate facing information as second information described below. In the following description, information on the distance from the imaging element to the subject is referred to as distance information. Furthermore, the information on the focus state or distance is not limited to the focus state or distance itself, but may be information that can be converted into or associated with the focus state or distance. EXAMPLES
[0064] As a first embodiment, a facing information display process executed by the imaging device 100 (CPU 102 and image processing unit 105) during servo AF and continuous AF will be described. Servo AF is a function that repeats AF while the user half-presses the release button. Continuous AF is a function that constantly repeats AF regardless of the operation of the release button.
[0065] The flowchart in Fig. 6 shows a process for superimposing and displaying the facing information on an input image (first image) acquired by imaging performed by the imaging device 100 and displayed on the display unit 114. The facing information will be described later. The CPU 102 and the image processing unit 105 execute this process according to a program. This process is executed when the user sets the servo AF mode or the continuous AF mode as the AF mode through the setting menu and the imaging unit 107 starts imaging.
[0066] First, in step S601, the CPU 102 determines whether the lens unit 106 is in focus in a predetermined AF area within the imaging range (entire imaging element), and if so, performs processing in step S602, and if not, ends this processing. The AF area may be set by the CPU 102 or the user.
[0067] Next, in step S602, the image processing unit 105 determines a subject area, which is an area in which a subject appears, from the captured image generated by imaging. The determination of the subject area will be described with reference to the flowchart of FIG.
[0068] In step S701, the image processing unit 105 determines the subject area included in the input image. The captured image is input to the image processing unit 105 in a raster direction with the upper left pixel as the origin. In this embodiment, the subject area is determined using distance information within the effective image area of the captured image input to the image processing unit 105. The effective image area is the area of the signal (image) output from the imaging element as the effective pixels of the captured image. However, the image area for determining the subject area may be a size different from the effective image area. For example, when an image area obtained by cutting out a part of the effective image area is enlarged and displayed on the display unit 114, the subject area may be determined within the cut-out image area. In the following description, the image area for determining the subject area is referred to as a determination image area.
[0069] The distance information is acquired by the process shown in Fig. 3, and a distance in a direction away from the imaging unit 107 is shown as a positive distance. Note that the distance information may be acquired not only by the process shown in Fig. 3, but also by using a distance measuring device that is provided in the imaging device 100 separately from the imaging element and that is capable of measuring the distance to the subject. Examples of distance measuring devices include radar and LiDAR (Light Detection and Ranging).
[0070] Next, in step S702, the image processing unit 105 obtains the distance df to the subject (focus position) on which the lens unit 106 is in focus, from the distance information obtained in step S701, and stores it in the RAM 104.
[0071] Next, in step S703, the image processing unit 105 initializes the coordinates (x, y) of the pixel for which distance information is to be referenced to (0, 0). Specifically, the coordinates of the upper left pixel of the determination image area are set to (0, 0). In the coordinates (x, y), x indicates the horizontal coordinate of the determination image area, and y indicates the vertical coordinate of the determination image area.
[0072] Next, in step S704, the image processing unit 105 obtains the distance d(x, y) of the pixel at the coordinates (x, y).
[0073] Next, in step S705, the image processing unit 105 obtains the difference between the distance df to the focal position and the distance d(x, y) of the pixel at the coordinates (x, y), and judges whether the difference is equal to or smaller than a threshold. If the difference is equal to or smaller than the threshold, the process of step S706 is performed. If the difference exceeds the threshold, the process of step S707 is performed. As described above, the direction of the distance away from the imaging unit 107 is regarded as a positive direction, so when the distance d(x, y) is closer to the imaging unit 107 than the focal position, the difference is negative. Conversely, when the distance d(x, y) is farther from the imaging unit 107 than the focal position, the difference is positive. The threshold is a value used to judge the subject region. Since the difference in distance can be positive or negative, two threshold values, positive and negative, are prepared. The absolute values of these two thresholds may be equal to each other or different from each other. The threshold may be a fixed value set in advance, or may be arbitrarily set by the user through a setting menu.
[0074] In step S706, the image processing unit 105 determines that the pixel at the coordinates (x, y) is a pixel within the object region, and sets a determination flag for that pixel to 1 indicating that the pixel is within the object region (hereinafter referred to as an object region flag).
[0075] In step S707, the image processing unit 105 determines that the pixel at the coordinates (x, y) is outside the subject area, and sets the inside subject area flag for that pixel to 0. Note that in this embodiment, the inside subject area flag is 1 or 0 to indicate whether or not the pixel is inside the subject area, but as long as it can indicate whether or not the pixel is inside the subject area, the flag may be expressed by a value other than 1 or 0, or the inside subject area flag and the outside subject area flag may be provided separately.
[0076] Next, in step S708, the image processing unit 105 determines whether the determination of whether all pixels in the determination image area are within the object area or not is completed. If completed, this process ends, and if not completed, the process of step S709 is performed.
[0077] In step S709, image processing unit 105 changes the coordinates of the pixel for which distance information is referenced. Then, the process returns to step S704. Specifically, if a pixel at coordinates (x+1, y) exists as a pixel for which the determination as to whether it is within the object area or not has not been completed, a determination is made for the pixel at coordinates (x+1, y). If a pixel at coordinates (x+1, y) does not exist and a pixel at coordinates (0, y+1) exists, the above determination is made for the pixel at coordinates (0, y+1). If a pixel at coordinates (x+1, y) does not exist and a pixel at coordinates (0, y+1) does not exist either, the above determination has been completed up to the bottom right pixel of the determination image area, and the object area determination process is terminated.
[0078] By the above processing, the subject area can be determined from the input image. Note that the method of determining the subject area is not limited to this, and an area in which the pixel shift amount described above is equal to or less than a threshold value may be determined as the subject area. Also, an edge extraction filter may be applied to the input image to extract the contour of the subject area, and an area in which equidistant pixels are continuous may be determined as the subject area. Furthermore, the input image may be input to a neural network such as a fully convolutional network, and an image area recognized as containing a specific subject may be determined as the subject area. Furthermore, a plurality of subject area determination methods may be combined.
[0079] In this embodiment, the process using the object region determination flag described above continues, but an object region map in which the determination flags are associated with the pixels of the captured image may be created and temporarily stored in the frame memory 111 or RAM 104.
[0080] Next, in step S603, image processing unit 105 outputs to display unit 114 the subject region determined by image processing unit 105 in step S602 so as to be highlighted. For example, as shown in FIG. 8, a highlighting method is to superimpose a highlighting image 803 such as a zebra pattern that highlights the subject region 802 on a display image 801 corresponding to the captured image or the determined image region, and display the superimposed image on display unit 800 (114). Note that display image 801 may be a partial image region cut out from the captured image or the determined image region. The highlighting image may be a frame that surrounds the subject region or an image that colors the subject region. The image that colors the subject region may have transparency.
[0081] Next, in step S604, image processing unit 105 determines whether the area of the subject region is equal to or greater than a specified value (predetermined value). If the area is equal to or greater than the specified value, processing in step S606 is performed, and if the area is less than the specified value, processing in step S605 is performed. The area of the subject region can be calculated, for example, by measuring the number of pixels contained in the subject region. The specified value may be set in advance by the user through a menu operation, or may be set as a preset value in CPU 102. By ensuring that the area of the subject region is equal to or greater than the specified value, it is possible to prevent a decrease in accuracy of facing determination due to the area being too small.
[0082] In step S605, image processing unit 105 displays a warning on display unit 114. By displaying the warning, the user is notified that the subject area has become small, and is prompted to reset the subject area. As a method of displaying the warning, a message to the effect that the subject area is small may be superimposed on display image 801 and displayed on display unit 114. In addition, the zebra pattern displayed in step S603 may be made to blink, or may be colored in a color different from the highlighting color.
[0083] In step S606, the image processing unit 105 determines the arrangement coordinates of the facing information. A method for determining the arrangement coordinates of the facing information will be described with reference to Fig. 9 and Fig. 10. The flowchart in Fig. 9 shows the process of determining the arrangement coordinates of the facing information executed by the image processing unit 105.
[0084] In step S901, the image processing unit 105 initializes the coordinates of the arrangement coordinate candidates. The arrangement coordinates are coordinates used to determine the coordinates of a pixel from which distance information for the facing information is obtained by a process described later, and the display position of a facing display marker as the facing information. In this embodiment, the four arrangement coordinate candidates are used to obtain the coordinates of the upper left, upper right, lower left, and lower right of the subject area. The arrangement coordinate candidates are stored in the RAM 104, and are initialized to the coordinates of pixels at four diagonal positions of the display image as initial values. Note that the number of arrangement coordinate candidates and the initial values are not limited to these.
[0085] Next, in step S902, the image processing unit 105 initializes the coordinates (x, y) of a pixel for which a placement coordinate candidate is to be determined to the origin (0, 0).
[0086] Next, in step S903, the image processing unit 105 determines whether or not the determination flag in the subject region of the pixel at coordinates (x, y) is 1. If the determination flag is 1, the process of step S904 is performed, and if the determination flag is 0, the process of step S907 is performed.
[0087] Next, in step S904, image processing unit 105 calculates the lengths from the coordinates (x, y) to the four corner pixels of the upper left, upper right, lower left, and lower right of the subject area in the determined image area determined in step S602. The length here refers to the distance between any two points on a plane when the image is treated as a two-dimensional plane, and is calculated using Pythagoras' theorem. Note that the length is not limited to this, and the sum of the horizontal length and the vertical length may be used.
[0088] Next, in step S905, the image processing unit 105 judges whether or not the length calculated in step S904 is shorter than the length from the placement coordinate candidate to the pixel at the four corners. The processing of this step will be described later. If the length calculated in step S904 is shorter than the length from the placement coordinate candidate, the image processing unit 105 performs the processing of step S906, and if not, the image processing unit 105 performs the processing of step S708.
[0089] In step S906, the image processing unit 105 stores the coordinates (x, y) as a new arrangement coordinate candidate.
[0090] In step S907, the image processing unit 105 checks whether the determination of the arrangement coordinate candidates has been performed for all pixels in the subject region. If the determination has been completed for all pixels, the process ends. If there are pixels that have not yet been determined, the process proceeds to step S908.
[0091] In step S908, the image processing unit 105 changes the coordinates of the pixel to be referenced in order to determine the placement coordinate candidates. After changing the coordinates, the process returns to step S903 to repeat the determination of the placement coordinate candidates. Finally, when the processing of the entire subject area is completed, the multiple coordinates stored as the placement coordinate candidates are adopted as the placement coordinates of the final facing information and are used in the subsequent processing.
[0092] A method for calculating and comparing lengths from the placement coordinate candidates to the four corners of the determination image area to determine the placement coordinates of the facing information will be described with reference to Fig. 10. In this embodiment, four placement coordinate candidates (multiple locations) are determined in order to place the facing information described later at four positions in the subject area: the upper left, upper right, lower left, and lower right. Fig. 10 shows, as an example, a method for obtaining and comparing lengths from the placement coordinate candidates to the upper left pixel 1004 of the determination image area 1001 to determine the placement coordinates of the upper left facing information. The pixels of the determination image area are input in the raster direction, and the upper left pixel of the determination image area is set as the origin.
[0093] 10, first, to determine the upper left position coordinate candidate of subject area 1002 in determination image area 1001, coordinates (7, 4) of pixel 1003 at the bottom right diagonally located at the upper left corner are obtained and stored in RAM 104 as the initial value of the position coordinate candidate. Next, using the subject area flag for pixel 1004 at the origin (0, 0), it is determined whether pixel 1004 is within the subject area. In FIG. 10, pixel 1004 is outside subject area 1002 and the subject area flag is 0, so no length calculation is performed.
[0094] Next, the pixel 1004 is judged to be within the subject area using the subject area flag for the pixel (1,0) adjacent to the right of the pixel 1004. Such judgments are repeated, and when the pixel 1005 at coordinate (2,1) is judged to be within the subject area 1002 because its subject area flag is 1, calculation and comparison of lengths are performed. Specifically, a first length from the coordinate (2,1) of the pixel 1005 to the coordinate (0,0) of the pixel 1004 in the upper left corner, and a second length from the placement coordinate candidate (7,4) to the coordinate (0,0) of the pixel 1004 in the upper left corner are calculated. Using Pythagoras' theorem, the first length is calculated to be 2.23, and the second length is calculated to be 8.06. Next, when the first length 2.23 is compared with the second length 8.06, the first length is found to be shorter than the second length, and therefore the coordinate (2,1) of the pixel 1005 is stored as a new placement coordinate candidate. In this embodiment, the arrangement coordinate candidate (7,4) in the RAM 104 is overwritten with the coordinate (2,1).
[0095] Similarly, the lengths of pixel 1006 are calculated and compared since it is within the subject region 1002. The first length from the coordinates (3,1) of pixel 1006 to the coordinates (0,0) of the upper left corner is calculated to be 3.16, and the length from the placement coordinate candidate (2,1) to the coordinates (0,0) of the upper left corner is calculated to be 2.23. As a result, since the second length is shorter than the first length, the placement coordinate candidate is not overwritten and saved.
[0096] When the above process is performed for all pixels in the judgment image area 1001, the coordinates of the placement coordinate candidate stored in the RAM 104 finally become (2, 1), which is the pixel closest to the pixel in the upper left corner among the pixels in the subject area 1002. In this embodiment, the coordinates (2, 1) are adopted as the placement coordinates of the facing information at the upper left and are used in subsequent processes. Then, by performing the same process for the upper right, lower left, and lower right, the placement coordinates of the facing information at those positions can be determined.
[0097] Next, in step S607, the image processing unit 105 acquires (generates) facing information at the arrangement coordinates determined in step S606. The facing information is information that serves as a guide for a user to adjust the inclination of the imaging device 100 so that the parallelism (facing degree) of the imaging element to the subject is increased. The facing information is acquired, for example, by acquiring distance information at the arrangement coordinates and calculating the difference between the distance from the imaging unit 107 to the subject and the distance to the focus position indicated by the distance information.
[0098] The facing information may be obtained not only from the distance information but also from the information on the defocus described above. Also, the facing information may be obtained by defining an area based on the arrangement coordinates and using the difference between the average distance in the area and the distance to the in-focus position as distance information.
[0099] Next, in step S608, the image processing unit 105 arranges the facing information acquired in step S607 at the arrangement coordinates determined in step S606. Specifically, a superimposed image (second image) is generated in which the highlighted image generated in step S603 and a facing display marker indicating the facing information are superimposed on the display image, and this is output to the display unit 114.
[0100] 11(a) and (b) show examples of the facing display markers on the display unit 800 (114). In Fig. 11(a), facing display markers 1101 indicating a facing state in which the subject is faced directly are displayed at the upper left, upper right, lower left, and lower right on an enhanced image 803 indicating a subject area in a display image 801. In Fig. 11(b), facing display markers 1102 and 1103 indicating a non-facing state in which the subject is not faced directly are displayed at the upper left, upper right, lower left, and lower right on the enhanced image 803.
[0101] In this embodiment, a focus marker is used as the facing display marker. The focus marker is a marker that displays the focus state (defocus direction and in-focus state) of the subject. When in-focus, a frame surrounding the focus target area and two triangles facing each other vertically so that their vertices coincide are displayed, as in each facing display marker 1101 in FIG. 11(a). When the facing display markers 1101 at the top left, top right, bottom left, and bottom right of the subject area indicate a focused state in this way, the user who sees this can know that the image capture unit 107 is highly parallel to the subject (almost parallel), that is, the image capture unit 107 is facing the subject.
[0102] In a front focus state where the camera is focused closer to the subject, a frame surrounding the focus target area, two lower triangles with vertices facing diagonally outwardly upward, and one triangle with vertex facing downward are displayed, as in the facing display marker 1102 in FIG. 11(b). In a back focus state where the camera is focused farther away from the subject, a frame surrounding the focus target area, two upper triangles with vertices facing diagonally inwardly downward, and one triangle with vertex facing upward are displayed, as in the facing display marker 1103 in FIG. 11(b). In this way, when the facing display markers 1102 at the top left and top right indicate a front focus state and the facing display markers 1103 at the bottom left and bottom right indicate a back focus state, the user who sees this can know that the imaging unit 107 is not facing the subject. In this case, the user can know that the imaging unit 107 is tilted diagonally upward with respect to the subject.
[0103] It is to be noted that a facing display marker different from the facing display markers shown in Fig. 11(a) and (b) may be displayed. For example, a facing display marker having a different color in the in-focus state, the front focus state, and the back focus state may be displayed, or a facing display marker having a different shape from the facing display markers in Fig. 11(a) and Fig. 11(b) may be displayed. Furthermore, the facing display marker may not indicate the focus state, but may indicate whether the imaging device 100 is in a facing state or not facing the subject, or may indicate the tilt adjustment direction of the imaging device 100 to face the subject.
[0104] Fig. 11(c) shows an example in which arrow symbols are used as the facing display markers 1104 and 1105, which are different from the facing display markers 1101 to 1103 in Fig. 11(a) and (b). The facing display markers 1104 on the upper left and upper right show arrow symbols pointing to the front side of the figure, and the facing display markers 1105 on the lower left and lower right show arrow symbols pointing to the back side of the figure. A user who sees these facing display markers 1104 and 1105 can know that a facing state can be obtained by adjusting the inclination of the imaging device 100 so that the upper part of the imaging unit 107 moves to the front side and the lower part moves to the back side.
[0105] Also, the degree of parallelism of the imaging unit 107 to the subject may be displayed as a quantitative value. For example, in Fig. 11(d), the deviation distance from a directly facing state (a position parallel to the subject) obtained from distance information is used as the quantitative value, and the deviation distance at each of the upper left, upper right, lower left, and lower right locations of the subject area is displayed within the frame of the directly facing display marker 1106. For example, the deviation distance is a positive value when the subject is present at a distance farther than the distance in the directly facing state, and a negative value when the subject is present at a distance closer than the distance in the directly facing state.
[0106] Also, the display position of the front-facing display marker may be shifted from the arrangement coordinates of the front-facing information. For example, in Fig. 11(b), a part of the front-facing display marker 1102 overlaps with the outer edge of the subject area 802 (highlighted image 803), which reduces the visibility of the front-facing display marker 1102. Therefore, as shown in Fig. 11(e), a predetermined amount of offset is provided to the display position of the front-facing display marker 1102 with respect to the arrangement coordinates of the front-facing information so that the entire front-facing display marker 1102 is displayed within the subject area 802. This makes it possible to avoid a reduction in visibility of the front-facing display marker 1102.
[0107] Next, in step S609, the image processing unit 105 judges whether or not the facing information (facing display marker) has been displayed at the arrangement coordinates of all the facing information. If the facing information has been displayed at all the arrangement coordinates, the process proceeds to step S610. If there are any undisplayed arrangement coordinates on which the facing information has not yet been displayed, the process returns to step S607 and the facing information is displayed at the undisplayed arrangement coordinates.
[0108] In step S610, the image processing unit 105 judges whether or not the facing information displayed at all the arrangement coordinates indicates a facing state. If it indicates a facing state, the process of step S611 is performed, and if it indicates a non-facing state, the process returns to step S601 and the above processes are repeated.
[0109] In step S611, the image processing unit 105 displays on the display unit 800 (114) in a highlighted manner that the imaging unit 107 faces the subject directly. After that, the process returns to step S601, and the above processes are repeated.
[0110] Fig. 11(f) shows an example in which, as an emphasis display of the front facing state, four front facing display markers 1101 are colored, and an emphasis image 803 having a pattern different from the zebra pattern of Fig. 8 is superimposed on the display image 801 and displayed on the display unit 800. The user can easily confirm that the imaging unit 107 is in a front facing state with respect to the subject by looking at the change in color of the front facing display markers 1101 and the pattern of the emphasis image 803. Note that, as an emphasis display of the front facing state, a front facing display marker having a different shape from the front facing display marker in the non-front facing state may be displayed, or the color of the emphasis image 803 may be made different from that in the non-front facing state.
[0111] In this embodiment, the case where the facing information is displayed at four positions, the upper left, upper right, lower left, and lower right, in the subject area has been described, but other display methods may be adopted. For example, one piece of facing information that collectively indicates the facing information at the upper left, upper right, lower left, and lower right in the subject area (for example, the arrow symbol in FIG. 11(c)) may be displayed in the subject area. In addition, in this embodiment, the case where a display image in which the facing information is superimposed is displayed on the display unit 114 of the imaging device 100 has been described. In contrast to this, the image including the information on the subject area and the facing information may be output to the outside using the video terminal 109 or the network module 108 shown in FIG. 1, or may be transferred to the outside via the recording medium 112.
[0112] According to this embodiment, during servo AF and continuous AF, the subject area is automatically determined, and distance information and the like are acquired within the subject area to display facing information. This allows the user to improve the accuracy with which the image sensor faces the subject. EXAMPLES
[0113] Next, a description will be given of a facing information display process executed by the imaging device 100 (CPU 102 and image processing unit 105) during single AF as Example 2. Single AF is a function that keeps the focus fixed even if the subject moves after the user presses the release button halfway to focus on the subject through AF.
[0114] The flowchart in Fig. 12 shows a process of superimposing and displaying facing information on a display image acquired by imaging performed by the imaging device 100 and displayed on the display unit 114. This process is executed when the user sets the single AF mode as the AF mode through the setting menu and the imaging unit 107 starts imaging. Note that, among the steps included in the flowchart in Fig. 12, steps that perform the same processes as those in the flowchart in Fig. 6 are assigned the same step numbers as in Fig. 6 and their description will be omitted.
[0115] First, in step S1201, CPU 102 determines whether or not the release button of operation unit 113 has been half-pressed. If the release button has been half-pressed, the process of step S601 is performed, and if the release button has not been half-pressed, the process waits until the release button is half-pressed.
[0116] The processes from step S601 to step S603 are similar to those from step S601 to step S603 in FIG.
[0117] In step S1202 after step S603, image processing unit 105 determines whether or not the subject area needs to be changed. If the subject area needs to be changed, the process of step S1203 is performed, and if the change is not necessary, the process of step S604 is performed. The determination of whether or not the subject area needs to be changed is performed, for example, by providing operation unit 113 with a change end switch that is operated by the user when the change of the subject area is ended, and CPU 102 determines whether or not the change end switch has been operated. If the change end switch has not been operated, the subject area is being changed by the user, and it is determined that the change is necessary. If the change end switch has been operated, it is determined that the change of the subject area by the user has been completed or that the change of the subject area is not necessary, and it is determined that the change is not necessary. Note that the method of determining whether or not the subject area needs to be changed is not limited to this, and it may be determined based on the presence or absence of a change operation of the subject area on display unit 114 equipped with a touch sensor as described later, or it may be determined that the change is not necessary when a certain time has passed since the change operation of the subject area was not performed.
[0118] In step S1203, image processing unit 105 changes the subject area based on a user's operation. The subject area is changed in response to a user touching display unit 114 equipped with a touch sensor to instruct a change in the position of the outer edge of the subject area. After that, the process returns to step S603, and image processing unit 105 highlights the changed subject area.
[0119] The processes from step S604 to step S609 are similar to those from step S604 to step S609 in FIG.
[0120] In step S1204 after step S609, the CPU 102 determines whether or not the half-press operation of the release button has been released. If the half-press operation has been released, the CPU 102 performs the process of step S1205, and if the half-press operation is still continuing, the CPU 102 performs the process of step S610.
[0121] In step S1205, the image processing unit 105 hides the orientation information that has been displayed on the display unit 114. After hiding the orientation information, the process returns to step S1201 and waits for the release button to be pressed halfway.
[0122] 6, and if the facing information displayed at the arrangement coordinates of all the facing information indicates a non-facing state with respect to the subject, the process returns to step S602 to continue the display determination of the facing information. If the facing information displayed at the arrangement coordinates of all the facing information indicates a facing state with respect to the subject, the process proceeds to step S611.
[0123] The process of step S611 is similar to the process of step S611 in FIG. 6, and after this process, the CPU 102 performs the process of step S1206.
[0124] In step S1206, the CPU 102 determines whether or not the release button has been fully pressed. If the release button has been fully pressed, the process ends. If the release button has not been fully pressed, the process returns to step S602 and the display determination of the facing information continues.
[0125] CPU 102 may perform other processing when the orientation information displayed at the arrangement coordinates of all the orientation information indicates a orientation state (i.e., it is determined that the image sensor is parallel to the subject) and the release button is fully pressed, which is a trigger. For example, CPU 102 may cause imaging unit 107 to capture an image and record the captured image thus obtained in recording medium 112. This allows the user to capture an image while maintaining the orientation state by fully pressing the release button after confirming the orientation state with respect to the subject.
[0126] According to this embodiment, during single AF, the subject area is automatically determined, and distance information and the like are acquired within the subject area to display facing information. This allows the user to improve the facing accuracy of the image sensor to the subject. EXAMPLES
[0127] Next, as a third embodiment, a sub-window display of facing information executed by the imaging apparatus 100 will be described with reference to Fig. 13 to Fig. 15. In this embodiment, a sub-window display of facing information during single AF will be described as an example.
[0128] The flowchart in Fig. 13 shows a process of displaying facing information in a sub-window. This process is executed when the user sets the sub-window display through the setting menu and the imaging unit 107 starts imaging. Note that, among the steps included in the flowchart in Fig. 13, steps that perform the same processes as those in the flowcharts in Fig. 6 and Fig. 12 are given the same step numbers as in Fig. 6 and Fig. 12 and their explanations are omitted.
[0129] The processes from step S1201 to step S1203 are similar to the processes from step S1201 to step S1203 in Fig. 12. Moreover, the processes from step S601 to step S611 are similar to the processes from step S601 to step S611 in Fig. 6.
[0130] In step S1301 after step S611, the image processing unit 105 displays a subwindow on the display unit 114. The flowchart in Fig. 14 shows the process in which the image processing unit 105 determines the display area of the subwindow and displays the subwindow. Figs. 15(a) to (e) show display examples on the display unit 800 (114) in this process.
[0131] First, in step S1401, the image processing unit 105 judges whether or not the display of the facing information is present in the central region of the display image. If the display of the facing information is present in the central region, the process of step S1403 is performed, and if not, the process of step S1402 is performed. The judgment of whether or not the display of the facing information is present in the central region of the display image is performed, for example, by dividing the display image 801 vertically and horizontally into nine regions as shown in FIG. 15(a), and judging whether or not the arrangement coordinates of the facing information are present in the central region. Note that the four lines in the display image 801 in FIG. 15(a) to (e) are drawn for the purpose of explaining the nine divided regions, and may not be actually displayed on the display unit 800. The method of judging the presence of the display of the facing information is not limited to this, and the size of the central region may be changed, and it may be judged whether or not the facing display markers 1101 to 1103 in FIG. 15(a) are present in the central region, instead of the arrangement coordinates of the facing information.
[0132] In step S1402, the image processing unit 105 displays a sub-window 1501 in the central region of the display image as shown in Fig. 15(b). In the sub-window 1501, a superimposed image in which the emphasized image 803 and the facing display markers 1102 and 1103 are superimposed on the display image 801, and an image reduced in size from the display image 801 is displayed. When the subject region occupies a large proportion of the display image 801 as shown in Fig. 15(b), the facing display markers 1102 and 1103 are displayed near the edge of the display image 801, making it difficult for the user to simultaneously check the facing display markers 1102 and 1103. For this reason, by displaying a reduced image in the sub-window 1501, the user can adjust the inclination of the imaging device 100 so as to face the subject directly, simply by checking the image in the sub-window 1501.
[0133] The image displayed in the sub-window is not limited to the image obtained by reducing the superimposed image as described above. For example, the sub-window may be given a degree of transparency and superimposed on the original superimposed image. Also, only the facing display marker may be displayed in the sub-window. Furthermore, in order to distinguish the facing display marker included in the original superimposed image from the facing display marker displayed in the sub-window, the color or shape of these facing display markers may be changed. Also, the facing information superimposed on the original superimposed image may be hidden and the facing information may be displayed only in the sub-window.
[0134] In step S1403, image processing unit 105 determines whether or not to display a sub-window in a region other than the central region of the display image. If a sub-window is to be displayed in a region other than the central region, the process of step S1404 is performed. If not, the process ends without displaying the sub-window. The determination of whether or not to display a sub-window in a region other than the central region is performed, for example, by determining whether or not the user has set the sub-window to be displayed in a region other than the central region through an operation of a setting menu.
[0135] In step S1404, image processing unit 105 determines whether or not there is an area in the display image where a sub-window can be displayed. If there is an area where a sub-window can be displayed, the process of step S1405 is performed, and if there is no area where a sub-window can be displayed, the process of step S1406 is performed. The area where a sub-window can be displayed is determined by determining whether or not there is facing information displayed in each of, for example, nine divided areas, and determining that the sub-window can be displayed if there is no such information.
[0136] 15(c), when the proportion of the subject area 802 in the display image 801 is small, if the sub-window 1501 is displayed, the sub-window 1501 and the front-facing display marker 1502 may overlap, reducing the visibility of the sub-window 1501. For this reason, the display is controlled according to the determination results in steps S1403 and S1404 so that the sub-window 1501 does not overlap with the front-facing display marker 1502 outside the sub-window display.
[0137] In step S1405, image processing unit 105 displays a subwindow in the area determined in step S1404 as being capable of displaying the subwindow. The process of displaying the subwindow is the same as in step S1402. If there are multiple areas capable of displaying the subwindow, image processing unit 105 determines one subwindow display area from the multiple areas. For example, starting from the upper left area, it determines whether or not the area is capable of displaying the subwindow in order moving rightward, and when it reaches the rightmost area, it continues to determine from the leftmost area one row below in the rightward direction. Then, it determines the area as the subwindow display area determined to be capable of displaying the subwindow first.
[0138] As shown in Fig. 15(d), when the subject area 802 is biased toward the lower right side of the display image 801, the upper left area is determined as the sub-window display area. In this case, as shown in Fig. 15(e), a sub-window 1501 is displayed in the upper left area.
[0139] The method of determining the sub-window display area is not limited to this, and for example, the displayable areas may be determined in order starting from an area other than the upper left area, or the display position of the sub-window may be changed by the user. For example, the displayed sub-window may be moved in response to a touch operation on display unit 114 equipped with a touch sensor.
[0140] The processes from step S1204 to step S1206 after step S1301 are similar to the processes from step S1204 to step S1206 in FIG.
[0141] In this embodiment, the sub-window display of the facing information during single AF has been described, but the sub-window display of the facing information can also be performed during servo AF or continuous AF. For example, this can be achieved by performing the process of step S1301 described in Fig. 13 after step S609 in Fig. 6 described in the first embodiment, and then returning to step S601 and repeating the process.
[0142] According to this embodiment, the visibility of the facing information by the user is improved by displaying the facing information in a sub-window, which can further improve the facing accuracy of the image sensor relative to the subject by the user.
[0143] In the above embodiment, the image processing unit 105 as an information processing device is built into the imaging device 100. However, a personal computer having the same functions as the image processing unit 105 (operating according to a program that realizes the same functions) may be provided as an information processing device separate from the imaging device. In this case, the information processing device may determine the subject area and superimpose an emphasis image and facing information on an image acquired from the imaging device, and display the superimposed image on a display unit of the imaging device or an external monitor.
[0144] The above embodiment includes the following configurations.
[0145] (Configuration 1) a determining means for determining a subject region including the subject in a first image obtained by capturing an image of the subject using an optical system and an image sensor; an acquisition means for acquiring first information that changes depending on a degree of parallelism of the imaging element with respect to the object at a plurality of points within the object region; An information processing device characterized by having a control means for generating second information that serves as a guide for a user to adjust the degree of parallelism in accordance with the first information, and outputting a second image including an image of the subject area and the second information. (Configuration 2) 2. The information processing device according to configuration 1, wherein the control means arranges the second information at positions corresponding to the plurality of locations in the second image. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the determination means determines the subject area using information about a distance from the imaging element obtained at a plurality of positions in the first image. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the first information is information relating to a focus state of the optical system with respect to the subject or information relating to a distance from the imaging element to the subject. (Configuration 5) each of the plurality of pixels of the image sensor includes a plurality of photoelectric conversion elements that receive light from different regions of a pupil of the optical system; 5. The information processing device according to configuration 4, wherein the acquiring means acquires the first information from a phase difference between a pair of image signals generated using signals output from the plurality of photoelectric conversion elements of the plurality of pixels. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the second information indicates a defocus direction and a focus state of the optical system with respect to the subject. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the second information indicates a distance of the imaging element from a position parallel to the subject at each of the plurality of locations. (Configuration 8) 8. The information processing device according to any one of configurations 1 to 7, wherein the control means outputs the second image to a display means capable of displaying an image. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the second image includes an image that emphasizes the subject region. (Configuration 10) The information processing device according to any one of configurations 1 to 9, characterized in that the control means determines a position at which to place the second information by using a length between a pixel in the subject area and one of the four corner pixels of the first image. (Configuration 11) 11. The information processing device according to any one of configurations 1 to 10, wherein the control means changes the subject area in response to an operation by a user. (Configuration 12) 12. The information processing device according to any one of configurations 1 to 11, wherein the control means outputs a warning when the area of the subject region is smaller than a predetermined value. (Configuration 13) 9. The information processing device according to configuration 8, wherein the control means outputs the second information to the display means so that the second information is displayed in a subwindow. (Configuration 14) 14. The information processing apparatus according to configuration 13, wherein the control means reduces and outputs the second image so that it is displayed on the display means in a subwindow. (Configuration 15) 15. The information processing apparatus according to claim 13, wherein the control means sets the subwindow in which the subwindow display is performed to a position where the subwindow display does not overlap with the second information outside the subwindow display. (Configuration 16) An information processing device according to any one of configurations 1 to 15; An imaging device comprising the imaging element. (Configuration 17) 17. The imaging device according to configuration 16, wherein the determining means uses autofocus to determine the subject area. (Configuration 18) 18. The imaging device according to configuration 16 or 17, wherein the imaging device captures an image of the subject in response to the second information determining that the imaging element is parallel to the subject.
[0146] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0147] The embodiments described above are merely representative examples, and various modifications and alterations are possible for each embodiment when implementing the present invention. [Explanation of symbols]
[0148] 100 Imaging device 102 CPU 105 Image processing section 106 Lens unit 107 Imaging unit 114 Display section 1101~1106 Front display marker
Claims
1. a determining unit that determines a subject region including the subject in a first image obtained by capturing an image of the subject using an optical system and an image sensor; an acquisition means for acquiring first information that changes depending on a degree of parallelism of the imaging element with respect to the object at a plurality of points within the object region; An information processing device characterized by having a control means for generating second information that serves as a guide for a user to adjust the degree of parallelism in accordance with the first information, and outputting a second image including an image of the subject area and the second information.
2. The information processing apparatus according to claim 1 , wherein the control means arranges the second information at positions corresponding to the plurality of locations in the second image.
3. The information processing apparatus according to claim 1 , wherein the determining means determines the subject area using information about a distance from the imaging element, the information being obtained at a plurality of positions in the first image.
4. 2 . The information processing apparatus according to claim 1 , wherein the first information is information about a focus state of the optical system with respect to the subject, or information about a distance from the imaging element to the subject.
5. each of the plurality of pixels of the image sensor includes a plurality of photoelectric conversion elements that receive light from different regions of a pupil of the optical system; 5 . The information processing apparatus according to claim 4 , wherein the acquiring means acquires the first information from a phase difference between a pair of image signals generated using signals output from the plurality of photoelectric conversion elements of the plurality of pixels.
6. The information processing apparatus according to claim 1 , wherein the second information indicates a defocus direction and a focus state of the optical system with respect to the subject.
7. The information processing apparatus according to claim 1 , wherein the second information indicates distances of the imaging element from positions parallel to the subject at the plurality of points.
8. 2. The information processing apparatus according to claim 1, wherein the control means outputs the second image to a display means capable of displaying an image.
9. The information processing apparatus according to claim 1 , wherein the second image includes an image that emphasizes the subject region.
10. 2 . The information processing apparatus according to claim 1 , wherein the control means determines a position at which the second information is to be placed by using a length between a pixel in the subject area and any one of the four corner pixels of the first image.
11. The information processing apparatus according to claim 1 , wherein the control means changes the subject area in response to an operation by a user.
12. 2. The information processing apparatus according to claim 1, wherein the control means outputs a warning when the area of the subject region is smaller than a predetermined value.
13. 9. The information processing apparatus according to claim 8, wherein the control means outputs the second information to the display means so that the second information is displayed in a subwindow.
14. 14. The information processing apparatus according to claim 13, wherein the control means outputs the second image by reducing it so that it is displayed on the display means in a sub-window.
15. 14. The information processing apparatus according to claim 13, wherein the control means sets the subwindow in which the subwindow display is performed at a position that does not overlap with the second information outside the subwindow display.
16. An information processing device according to any one of claims 1 to 15; An imaging device comprising the imaging element.
17. 17. The imaging apparatus according to claim 16, wherein the determining means uses autofocus to determine the subject area.
18. 17. The imaging apparatus according to claim 16, wherein an image of the subject is captured in response to a determination that the imaging element is parallel to the subject from the second information.
19. determining a subject region including the subject in a first image obtained by capturing an image of the subject using an optical system and an image sensor; acquiring first information at a plurality of points within the subject region, the first information varying according to a degree of parallelism of the imaging element with respect to the subject; An information processing method comprising the steps of generating second information that serves as a guide for a user to adjust the degree of parallelism in accordance with the first information, and outputting a second image including an image of the subject area and the second information.
20. 20. A program for causing a computer to execute a process according to the information processing method according to claim 19.
Citation Information
Patent Citations
Electronic camera
JP1996186757A
Information processing device, information processing method, and program
JP6679779B1
