Imaging system, method for operating imaging system, and program

Abstract

An imaging system according to the present disclosure comprises a processor. The processor acquires relative distance information in a region set in a captured image, and determines an imaging setting on the basis of the relative distance information.

Description

Imaging System, Method of Operating an Imaging System, and Program

[0001] The technology of the present disclosure relates to an imaging system, a method of operating an imaging system, and a program.

[0002] International Publication No. 2019 / 073814 discloses a focus detection device that outputs defocus amount relationship information related to a defocus amount by performing a learning-based operation based on the light reception amount distribution of an A pixel group having a first characteristic for phase difference detection and the light reception amount distribution of a B pixel group having a second characteristic different from the first characteristic.

[0003] Japanese Patent Application Laid-Open No. 2022-019374 discloses a process of acquiring input data including an imaging image and information related to the state of an optical system used for imaging the imaging image, and a process of inputting the input data into a machine learning model to estimate distance information of the imaging image, wherein the information related to the state of the optical system includes at least one of a focal length, an aperture value, or a focus distance.

[0004] Japanese Patent Application Laid-Open No. 2023-011489 discloses an imaging device including an imaging unit that captures a subject image to generate image data, a distance measuring unit that acquires distance information indicating the distance to the subject in the image shown by the image data, a detection unit that acquires subject detection information related to the region where the subject is located in the image, and a recognition unit that recognizes a subject region having a shape along the subject in the image based on the distance information and the subject detection information.

[0005] The technology of the present disclosure provides an imaging system, a method of operating an imaging system, and a program that enable accurate determination of imaging settings.

[0006] To achieve the above object, the imaging system of the present disclosure is an imaging system including a processor, and the processor acquires relative distance information within a region set in the imaging image and determines an imaging setting based on the relative distance information.

[0007] Preferably, the processor acquires information about the relative distance between objects within a region as relative distance information, and uses this relative distance information to determine the imaging settings to be performed using the image information within the region.

[0008] Preferably, the processor inputs image information into a machine learning model that has learned the relationship between the relative distance between objects in a region and the image information, and obtains the output data from the machine learning model as relative distance information.

[0009] Preferably, the processor classifies the region into multiple groups based on relative distance information and determines the imaging settings for a selected group that includes one or more groups selected from the multiple groups.

[0010] The imaging settings are preferably settings related to focusing.

[0011] The processor preferably divides the area into multiple sub-regions and determines the focusing settings based on multiple defocus amounts corresponding to the multiple sub-regions included in the selection group.

[0012] It is preferable that the processor be selected from multiple groups, with some of the remaining groups being excluded.

[0013] The processor preferably determines the focusing settings based on an index relating to the frequency of multiple defocus amounts.

[0014] The indicator should preferably be the median.

[0015] Preferably, the image sensor that acquires the captured image includes multiple phase-difference detection pixels, and the processor determines the amount of defocus based on the output values ​​of the multiple phase-difference detection pixels corresponding to a small area.

[0016] The processor preferably selects the group that is closest to the imaging system from among several groups.

[0017] The processor preferably determines the confidence level of the relative distance information for each of the multiple sub-regions and classifies the multiple sub-regions into multiple groups based on the confidence level.

[0018] The processor preferably selects a group from among several groups that has a reliability level above a certain level.

[0019] The processor preferably determines the confidence level based on at least one of the distribution of relative distances within a small region and the step difference in relative distances within that small region.

[0020] The reliability is preferably higher when the spread of the distribution is small or the step is large.

[0021] The processor may also divide the region into multiple sub-regions, classify the multiple sub-regions into multiple groups based on relative distance information, and determine the imaging settings based on the weights determined for each of the multiple groups.

[0022] The imaging settings are preferably either exposure or white balance settings.

[0023] The processor preferably modifies the photometric values ​​corresponding to small regions based on weights.

[0024] The method for operating the imaging system of the present disclosure is a method for operating an imaging system comprising a processor, wherein the processor performs a process that includes acquiring relative distance information within a region set in an captured image and determining imaging settings based on the relative distance information.

[0025] The program disclosed herein is a program for operating an imaging system equipped with a processor, and causes the processor to perform a process that includes acquiring relative distance information within a region set in an captured image and determining imaging settings based on the relative distance information.

[0026] This is a diagram showing an example of the configuration of an imaging device. This is a diagram showing an example of the light-receiving surface of an image sensor. This is a block diagram showing an example of the functional configuration of a processor. This is a diagram conceptually illustrating an example of AF calculation processing. This is a diagram conceptually illustrating an example of defocus amount calculation processing for each small region. This is a diagram conceptually illustrating an example of distance map acquisition processing. This is a diagram conceptually illustrating an example of grouping processing. This is a diagram conceptually illustrating an example of imaging setting determination processing. This is a flowchart showing the flow of AF control. This is a diagram showing an example of a distance map. This is a flowchart showing the flow of AF control related to the first modified example. This is a flowchart showing the details of the reliability determination processing. This is a block diagram showing an example of the functional configuration of a processor related to the second modified example. This is a diagram conceptually illustrating an example of AE calculation processing. This is a diagram conceptually illustrating an example of grouping related to the second modified example.

[0027] An example of an embodiment relating to the technology of this disclosure will be described with reference to the attached drawings.

[0028] First, let's explain the terminology used in the following explanation.

[0029] In the following explanation, "AF" is an abbreviation for "Auto Focus". "AE" is an abbreviation for "Auto Exposure". "MF" is an abbreviation for "Manual Focus". "IC" is an abbreviation for "Integrated Circuit". "CPU" is an abbreviation for "Central Processing Unit". "ROM" is an abbreviation for "Read Only Memory". "RAM" is an abbreviation for "Random Access Memory". "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor". "OVF" is an abbreviation for "Optical View Finder". "EVF" is an abbreviation for "Electronic View Finder". "CNN" is an abbreviation for "Convolutional Neural Network".

[0030] As one embodiment of the imaging system, the technology of this disclosure will be explained using a lens-interchangeable digital camera as an example. However, the technology of this disclosure is not limited to lens-interchangeable cameras, but can also be applied to lens-integrated digital cameras.

[0031] Figure 1 shows an example of the configuration of the imaging device 10. The imaging device 10 is a lens-interchangeable digital camera. The imaging device 10 consists of a main body 11 and an imaging lens 12 which is interchangeably mounted on the main body 11 and includes a focus lens 31. The imaging lens 12 is attached to the front side of the main body 11 via a camera-side mount 11A and a lens-side mount 12A.

[0032] The main unit 11 is provided with an operating section 13, which includes a dial, a shutter release button, and the like. The operating modes of the shooting device 10 include, for example, still image shooting mode, continuous shooting mode, video shooting mode, and image display mode. The operating section 13 is operated by the user when setting the operating mode. The operating section 13 is also operated by the user when starting still image shooting, continuous shooting, or video shooting.

[0033] Furthermore, the control unit 13 is operated by the user when selecting a focus mode. There are two focus modes: AF mode and MF mode. AF mode is a mode in which focus control is performed automatically on the focus target area (hereinafter referred to as the AF area) within the angle of view. The user can set the AF area within the angle of view using the control unit 13. MF mode is a mode in which the user manually controls the focus by operating the focus ring (not shown). The shooting device 10 may also be configured to allow the user to set the AF area via a display 15 with touch panel functionality or a viewfinder 14 with eye-tracking functionality.

[0034] Furthermore, the main body 11 is provided with a viewfinder 14. Here, the viewfinder 14 is a hybrid viewfinder (registered trademark). A hybrid viewfinder refers to a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF") are selectively used. The user can observe the optical image or live view image of the subject projected by the viewfinder 14 through the viewfinder eyepiece (not shown).

[0035] Furthermore, a display 15 is provided on the back of the main unit 11. The display 15 shows images based on image data obtained through shooting, as well as various menu screens, etc. The user can also observe the live view image displayed on the display 15 instead of the viewfinder 14.

[0036] The main unit 11 and the photographic lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera-side mount 11A and an electrical contact 12B provided on the lens-side mount 12A.

[0037] The photographic lens 12 includes an objective lens 30, a focusing lens 31, a rear-end lens 32, and an aperture 33. Each component is arranged along the optical axis A of the photographic lens 12, from the objective side, in the order of objective lens 30, aperture 33, focusing lens 31, and rear-end lens 32. The objective lens 30, focusing lens 31, and rear-end lens 32 constitute an optical system. The type, number, and arrangement order of the lenses constituting the optical system are not limited to the example shown in Figure 1.

[0038] Furthermore, the photographic lens 12 has a lens drive unit 34. For example, the lens drive unit 34 is configured to include a stepping motor that moves the focus lens 31 in the direction of the optical axis A, and a driver that supplies pulses to the stepping motor. The stepping motor moves the focus lens 31 in accordance with the pulses supplied from the driver.

[0039] The lens drive unit 34 is electrically connected to the processor 40 in the main body 11. The lens drive unit 34 drives the focus lens 31 based on control signals transmitted from the processor 40. The lens drive unit 34 drives the focus lens 31 based on focus control control signals transmitted from the processor 40 to adjust the position of the focus lens 31.

[0040] Furthermore, the main unit 11 is equipped with an image sensor 20, a processor 40, and a memory 42. The image sensor 20, memory 42, operation unit 13, viewfinder 14, and display 15 are all controlled by the processor 40.

[0041] The processor 40 is composed of, for example, a CPU, RAM, ROM, etc. In this case, the processor 40 executes various processes based on a program 43 stored in memory 42. The processor 40 may also be composed of an assembly of multiple IC chips.

[0042] The image sensor 20 is, for example, a CMOS type image sensor. The image sensor 20 is arranged such that the optical axis A is perpendicular to the light-receiving surface 20A and the optical axis A is located at the center of the light-receiving surface 20A. Light that has passed through the photographic lens 12 is incident on the light-receiving surface 20A. Multiple pixels are formed on the light-receiving surface 20A, which generate an imaging signal by performing photoelectric conversion. The image sensor 20 generates and outputs image data PD (hereinafter simply referred to as image data PD) by performing photoelectric conversion on the light incident on each pixel.

[0043] Furthermore, a Bayer-arranged color filter array is positioned on the light-receiving surface 20A of the image sensor 20, with one of the R (red), G (green), or B (blue) color filters positioned opposite each pixel. Some of the multiple pixels arranged on the light-receiving surface of the image sensor 20 are phase difference detection pixels that output a phase difference detection signal.

[0044] FIG. 2 shows an example of the light-receiving surface 20A of the imaging device 20. A plurality of imaging pixels 21 and a plurality of phase difference detection pixels 22 are arranged on the light-receiving surface 20A. The imaging pixel 21 is a pixel on which the above-described color filter is arranged. The imaging pixel 21 receives a light beam passing through the entire area of the exit pupil of the imaging optical system. The phase difference detection pixel 22 receives a light beam passing through a region that is half of the exit pupil of the imaging optical system.

[0045] In the example shown in FIG. 2, in the Bayer array, a part of the G pixels arranged diagonally is replaced by the phase difference detection pixels 22. The phase difference detection pixels 22 are arranged at regular intervals in the vertical and horizontal directions on the light-receiving surface 20A. The phase difference detection pixel 22 is divided into a first phase difference detection pixel 22a that receives a light beam passing through a region that is half (for example, the right half) of the exit pupil, and a second phase difference detection pixel 22b that receives a light beam passing through the other half (for example, the left half) of the exit pupil region.

[0046] The light-receiving surface 20A is a plane orthogonal to the X direction and the Y direction orthogonal to the X direction. The plurality of first phase difference detection pixels 22a are arranged at a predetermined pitch in the X direction. Similarly, the plurality of second phase difference detection pixels 22b are arranged at a predetermined pitch in the X direction. The rows in which the plurality of first phase difference detection pixels 22a are arranged and the rows in which the plurality of second phase difference detection pixels 22b are arranged are alternately arranged in the Y direction.

[0047] The plurality of imaging pixels 21 output an imaging signal for generating an image of a subject. The plurality of first phase difference detection pixels 22a output a first phase difference detection signal. The plurality of second phase difference detection pixels 22b output a second phase difference detection signal. The captured image PD output from the imaging device 20 includes an imaging signal, a first phase difference detection signal, and a second phase difference detection signal. The first phase difference detection signal and the second phase difference detection signal are an example of the "output values of a plurality of phase difference detection pixels" according to the technology of the present disclosure.

[0048] Figure 3 shows an example of the functional configuration of the processor 40. The processor 40 realizes various functional units by executing processing according to the program 43 stored in the memory 42. As shown in FIG. 3, for example, the processor 40 realizes a main control unit 50, an imaging control unit 51, an image processing unit 52, a display control unit 53, an image recording unit 54, an AF calculation unit 55, and a distance map acquisition unit 56. The AF calculation unit 55 and the distance map acquisition unit 56 operate when the AF mode is set.

[0049] The main control unit 50 comprehensively controls the operation of the imaging device 10 based on an instruction signal input from the operation unit 13. The imaging control unit 51 executes an imaging process for causing the imaging element 20 to generate an imaging image PD by controlling the imaging element 20. The imaging control unit 51 drives the imaging element 20 in the still image shooting mode, the continuous shooting mode, or the moving image shooting mode. The imaging element 20 outputs the imaging image PD generated by imaging through the imaging lens 12. The imaging image PD output from the imaging element 20 is supplied to the image processing unit 52, the AF calculation unit 55, and the distance map acquisition unit 56.

[0050] The image processing unit 52 acquires the imaging image PD output from the imaging element 20 and performs image processing including white balance adjustment, gamma correction processing, etc. on the imaging image PD.

[0051] The display control unit 53 causes the display 15 to display the imaging image PD subjected to image processing by the image processing unit 52 as a live view image. When the release button is fully pressed, the image recording unit 54 records the imaging image PD subjected to image processing by the image processing unit 52 in the memory 42 as a recorded image PR.

[0052] The AF calculation unit 55 divides the AF area into a plurality of small areas, calculates the defocus amount for each small area, and outputs it. Specifically, the AF calculation unit 55 acquires the first phase difference detection signal and the second phase difference detection signal from within the AF area of the imaging image PD output from the imaging element 20, and calculates a plurality of defocus amounts by performing a correlation operation for each small area. The defocus amount corresponds to the deviation amount from the in-focus position of the focus lens 31.

[0053] The distance map acquisition unit 56 acquires a distance map representing the relative distance between objects within the AF area based on the image information within the AF area of ​​the captured image PD. Specifically, the distance map acquisition unit 56 inputs the image information within the AF area to a machine learning model (hereinafter referred to as the trained model) LM and acquires the output data from the trained model LM as a distance map. The distance map is a two-dimensional image in which relative distances are represented as brightness values. The distance map is an example of "relative distance information" related to the technology disclosed herein. Relative distance information is information that classifies each pixel of a two-dimensional image based on relative distance, and is similar to semantic segmentation. In contrast, the amount of defocus obtained by AF calculation is information based on dispersedly arranged phase difference detection pixels, so the relative distance information has a higher resolution than the distance information obtained by AF calculation.

[0054] The pre-trained model LM is a model that has learned the relationship between the relative distance between objects in the AF area and image information, and is stored in memory 42 beforehand. For example, the pre-trained model LM is composed of a CNN.

[0055] While AF (autofocus) calculations can accurately determine the distance to high-contrast subjects, their accuracy in determining the relative distance between objects within the AF area is low. According to the trained model LM, the relative distance between objects within the AF area can be accurately determined based on the characteristics of the objects.

[0056] Furthermore, if a subject detection function is provided, the area containing a specific subject detected by subject detection may be designated as the AF area. The type of subject detected by subject detection is, for example, a human face. The AF area is an example of a "region set within the captured image" related to the technology of this disclosure.

[0057] The main control unit 50 classifies the AF area into multiple groups based on relative distance information and determines the imaging settings for one selected group from the multiple groups. As will be described in detail later, grouping is performed based on a distance map, and the imaging settings are determined based on multiple defocus amounts included in the selected group. In this embodiment, the imaging setting is the drive amount of the focus lens 31. The drive amount of the focus lens 31 is the amount of movement that moves the focus lens 31 from its current position to the in-focus position, and is an example of a "setting related to focusing" according to the technology of this disclosure. The main control unit 50 may select multiple groups as selected groups.

[0058] The main control unit 50 moves the focus lens 31 in the direction of the optical axis A based on the determined drive amount of the focus lens 31.

[0059] Figure 4 conceptually shows an example of AF calculation processing by the AF calculation unit 55. In Figure 4, reference numeral 60 denotes the AF area. For example, the AF area 60 is rectangular. The AF calculation unit 55 divides the AF area 60 into a plurality of small regions 60a. For example, the AF calculation unit 55 divides the AF area 60 into four equal parts in both the X and Y directions, thereby setting 16 small regions 60a. The AF calculation unit 55 then calculates the defocus amount DF from each of the plurality of small regions 60a.

[0060] Figure 5 conceptually shows an example of the calculation process of the defocus amount DF for each small region 60a by the AF calculation unit 55. S1 and S2 shown in Figure 5 represent the first phase difference detection signal and the second phase difference detection signal included in one small region 60a. The AF calculation unit 55 calculates the sum of squared differences while shifting one of the first phase difference detection signal S1 and the second phase difference detection signal S2 by one pixel in the X direction. The sum of squared differences is calculated by squaring the difference between the first phase difference detection signal S1 and the second phase difference detection signal S2 each time the signal is shifted by one pixel, and then adding up the results. The sum of squared differences corresponds to the reciprocal of the correlation value; the smaller the value, the higher the correlation value.

[0061] As shown in Figure 5, the AF calculation unit 55 identifies the shift amount δ that minimizes the sum of squared differences within the shift range (the range from the minimum to the maximum value of the shift amount ΔX). The defocus amount DF corresponds to the shift amount δ.

[0062] Figure 6 conceptually illustrates an example of the distance map acquisition process performed by the distance map acquisition unit 56. The distance map acquisition unit 56 extracts an image 61 corresponding to the AF area 60 from the captured image PD and inputs the extracted image 61 into the trained model LM. The distance map acquisition unit 56 then acquires the distance map 62 output from the trained model LM. The greater the brightness value (i.e., the whiter the color), the closer the distance to the imaging device 10. In the example shown in Figure 6, the distance map 62 indicates that the human face is located in front of the background trees within the AF area 60.

[0063] Figure 7 conceptually illustrates an example of grouping processing by the main control unit 50. The main control unit 50 divides the distance map 62 into a plurality of sub-regions 60a and classifies the plurality of sub-regions 60a into a plurality of groups based on the brightness value (i.e., relative distance information) within each sub-region 60a. In this embodiment, the main control unit 50 classifies sub-regions 60a with high brightness values ​​into the foreground group G1 and sub-regions 60a with low brightness values ​​into the background group G2. The foreground group G1 is the group that is closest to the imaging device 10 among the plurality of groups. In the example shown in Figure 7, the sub-region 60a corresponding to the area of ​​a human face is classified into the foreground group G1, and the sub-region 60a corresponding to trees, etc. is classified into the background group G2. Note that if there are multiple people in the AF area 60, the foreground group G1 may include two independent regions.

[0064] Figure 8 conceptually illustrates an example of the imaging setting determination process by the main control unit 50. The main control unit 50 selects one group from a plurality of groups classified by the grouping process and determines the drive amount of the focus lens 31 (i.e., the setting related to focusing) based on a plurality of defocus amounts DF corresponding to a plurality of sub-regions 60a included in the selected group.

[0065] In this embodiment, the main control unit 50 selects the foreground group G1 as the selection group from the foreground group G1 and the background group G2 (i.e., excluding the background group G2), and determines the drive amount of the focus lens 31 based on the median value of the multiple defocus amounts DF included in the foreground group G1. The median value is the value that is in the middle in rank in the frequency distribution. If the number of data points in the frequency distribution is even, the median value is the average of the two values ​​closest to the center. The median value is an example of an "indicator relating to the frequency of multiple defocus amounts" related to the technology of this disclosure.

[0066] In this way, by setting the imaging parameters based on the remaining foreground group G1 after excluding the background group G2, it is possible to accurately focus on the human face, which is the AF target within the AF area 60.

[0067] Figure 9 shows the flow of AF control performed by the processor 40. First, the processor 40 acquires the captured image PD output from the image sensor 20 (step S10). After this, the AF calculation unit 55 and the distance map acquisition unit 56 perform processing in parallel.

[0068] The AF calculation unit 55 divides the AF area 60 into a plurality of small regions 60a as described above (step S11), and calculates the defocus amount DF for each small region 60a (step S12).

[0069] As described above, the distance map acquisition unit 56 inputs the image 61 within the AF area 60 to the trained model LM (step S13) and acquires the distance map 62 output from the trained model LM (step S14). The main control unit 50 divides the distance map 62 into a plurality of small regions 60a (step S15) and classifies the plurality of small regions 60a into a plurality of groups based on the relative distance information within each small region 60a (step S16).

[0070] Next, the main control unit 50 selects the front group G1 as the selection group from the multiple groups that have been divided into groups, and calculates the median value of the multiple defocus amounts DF included in the selection group (step S17). Then, the main control unit 50 drives the focus lens 31 using the calculated median value as the focus position (step S18).

[0071] AF calculations become less accurate for low-contrast subjects (i.e., the accuracy of calculating the defocus amount DF is low). Therefore, if there is a high-contrast subject (trees in the example shown in Figure 4) in the background of the main subject to be AF (a human face in the example shown in Figure 4) within the AF area 60, the focus position may shift towards the far side (so-called Far side) due to the influence of the background (so-called back focus). This corresponds to the case where the median is calculated without separating the foreground group G1 and the background group G2, as shown in Figure 8, resulting in a shift in the median towards the far side compared to when the median is calculated using only the foreground group G1.

[0072] In this embodiment, the main control unit 50 uses the distance map 62 to determine the median value of multiple defocus amounts DF included in the foreground group G1, thereby eliminating the influence of a high-contrast background and enabling accurate focusing on the main subject of the AF target located in the foreground. In this embodiment, the median value is used as an indicator of frequency, but other indicators such as the average value may also be used. Thus, according to this embodiment, imaging settings can be determined with high accuracy.

[0073] Furthermore, in this embodiment, since the AF calculation process and the distance map acquisition process are executed in parallel, the overall AF control can be performed at high speed. In other words, according to this embodiment, imaging settings can be determined accurately and quickly.

[0074] Various modifications of the above embodiment will be described below.

[0075] [First Modified Example] In the above embodiment, the main control unit 50 groups the small areas 60a based on the luminance values ​​(i.e., relative distance information) within the small areas 60a. However, the reliability of the relative distance determination may be low depending on the luminance distribution within the small areas 60a. For example, the luminance distribution within the small areas 60a may widen due to camera shake, subject blur, noise, etc.

[0076] For example, if the spread of the luminance distribution within the small region 60a is small (i.e., the luminance values ​​are uniform), it can be determined that the distances of the objects contained within the small region 60a are almost uniform, which provides a high degree of reliability.

[0077] On the other hand, if the luminance distribution within the small region 60a is widespread (i.e., the luminance values ​​are not nearly uniform), the reliability may be low. As shown in Figure 10, even if the luminance distribution is widespread, in the case of a small region 60a with large luminance steps including the contours of a face, two distances are present, and the distance can be determined by comparing the two, resulting in high reliability. In contrast, in the case of a small region 60a with a wide luminance distribution and small luminance steps, the distance changes continuously, making it impossible to determine the distance, resulting in low reliability.

[0078] Therefore, in this modified example, the main control unit 50 excludes the small region 60a with low reliability from the selection group.

[0079] Figure 11 shows the flow of AF control executed by the processor 40 according to the first modified example. In this modified example, the processor 40 executes steps S20 and S21 between step S15 and step S16, which is different from the flow of AF control shown in Figure 9.

[0080] In step S15, the main control unit 50 divides the distance map 62 into a plurality of sub-regions 60a, then determines the reliability of each sub-region 60a (step S20), and excludes the sub-regions 60a with low reliability (step S21). In this modified example, in step S16, the main control unit 50 classifies the multiple sub-regions 60a with high reliability other than the excluded sub-regions 60a into multiple groups.

[0081] Figure 12 shows the details of the confidence level determination process (step S20). First, the main control unit 50 selects one of the divided sub-regions 60a (step S201) and determines whether the spread of the luminance distribution (e.g., the variance value) within the selected sub-region 60a is less than a threshold (step S202). If the spread of the luminance distribution is less than a threshold (step S202: YES), the main control unit 50 determines that the confidence level is high (step S204).

[0082] On the other hand, if the spread of the luminance distribution is not less than a threshold (step S202: NO), the main control unit 50 determines whether the difference in luminance within the selected small region 60a is greater than or equal to a certain value (step S203). If the difference in luminance is greater than or equal to a certain value (step S203: YES), the main control unit 50 determines that the reliability is high (step S204). On the other hand, if the difference in luminance is less than a certain value (step S203: NO), the main control unit 50 determines that the reliability is low (step S205).

[0083] After step S204 or step S205, the main control unit 50 determines whether the selected sub-region 60a is the last sub-region 60a (step S206). If it is not the last sub-region 60a (step S206: NO), the main control unit 50 selects the next sub-region 60a (step S207) and returns to step S202. If it is the last sub-region 60a (step S206: YES), the main control unit 50 terminates the process.

[0084] Thus, the main control unit 50 determines that small regions 60a with a small spread of luminance distribution and small regions 60a with a large luminance step are highly reliable, and small regions 60a with a small luminance step are less reliable. In other words, the reliability is higher the smaller the spread of luminance distribution or the larger the luminance step.

[0085] In this modified example, the low-reliability small region 60a is excluded, and a group for determining the imaging settings is selected from the small region 60a with a reliability of a certain level or higher, thereby enabling more accurate determination of the imaging settings.

[0086] In the above modified example, the reliability is determined using the spread of the luminance distribution and the luminance step, but the reliability may also be determined using only one of the spread of the luminance distribution or the luminance step. In other words, the reliability can be determined using at least one of the spread of the luminance distribution or the luminance step.

[0087] [Second Modified Example] In the above embodiment, the imaging setting determined by relative distance information within a region set in the captured image is used as the setting for focusing, but this imaging setting may also be used as the exposure setting.

[0088] Figure 13 shows an example of the functional configuration of the processor 40 according to the second modified example. In this modified example, the processor 40 includes a main control unit 50, an imaging control unit 51, an image processing unit 52, a display control unit 53, an image recording unit 54, an AF calculation unit 55, and a distance map acquisition unit 56, in addition to an AE calculation unit 57.

[0089] The AE calculation unit 57 performs multi-segment photometry using the captured image PD. Specifically, as shown in Figure 14, the entire area 70 of the captured image PD is divided into multiple small areas 70a, and a photometric value EV is calculated and output for each small area 70a. Specifically, the AE calculation unit 57 acquires the imaging signal of the captured image PD output from the image sensor 20 and calculates the brightness of each small area 70a as a photometric value EV. The entire area 70 is an example of a "region set within the captured image" according to the technology of this disclosure.

[0090] In this modified example, the distance map acquisition unit 56 inputs the entire captured image PD to the trained model LM and acquires the distance map 71 output from the trained model LM.

[0091] As shown in Figure 15, the main control unit 50 divides the distance map 71 into a plurality of sub-regions 70a and classifies the plurality of sub-regions 70a into a plurality of groups based on the brightness value (i.e., relative distance information) within each sub-region 70a. In the example shown in Figure 15, the main control unit 50 classifies the plurality of sub-regions 70a into three groups G1 to G3 according to their relative distance.

[0092] In this modified example, the main control unit 50 determines a weight for each of the multiple groups and assigns weights to the photometric value EV of each sub-region 70a. That is, the main control unit 50 changes the photometric value EV corresponding to the sub-region 70a based on the weight. For example, the main control unit 50 assigns a larger weight to groups that are closer to the imaging device 10.

[0093] In other words, in this modified example, the main control unit 50 calculates an evaluation value by weighting and adding a plurality of photometric values ​​EV calculated by the AE calculation unit 57 using weights determined based on the distance map 71, and sets the exposure based on the calculated evaluation value. The exposure setting includes setting at least one of the aperture value and the shutter speed.

[0094] According to this modified example, the photometric value EV can be weighted accurately according to the relative distance between objects, allowing for more appropriate exposure settings.

[0095] In this modified example, the imaging setting determined by relative distance information within a defined area of ​​the captured image is used as the exposure setting, but it can also be used as the white balance setting. Evaluation values ​​for estimating the light source in the white balance setting can be obtained using the same method as described above. In environments where outdoor and indoor light are mixed, the white balance may become inappropriate due to the influence of outdoor light. However, by weighting the settings according to the distance map, it becomes possible to set the white balance based on an appropriate light source for the subject.

[0096] [Other Modifications] In the above embodiment, the AF calculation process and the distance map acquisition process are executed in parallel, but the AF calculation process and the distance map acquisition process may be executed alternately. In this case, it is possible to adjust the parameters of the next AF calculation process to be executed based on the distance map (i.e., relative distance information). For example, the parameters of the AF calculation process are the pass frequency bands of the frequency filtering process performed on the first phase difference detection signal and the second phase difference detection signal before the correlation calculation. If the distance map suggests that the distribution of relative distances is wide (i.e., there is a lot of blur), the accuracy of calculating the amount of defocus can be improved by setting the pass frequency band of the low-pass filter to a lower value.

[0097] Furthermore, in the above embodiment, a distance map is obtained based on image information within the AF area, but a distance map may also be obtained based on image information from an area larger than the AF area. By using an image larger than the AF area, the amount of information about objects increases, thus improving the accuracy of calculating the relative distance between objects. In particular, when the set AF area is small, there are fewer objects included within the AF area, making it difficult to determine the relative distance between objects, so it is preferable to obtain a distance map based on image information from an area larger than the AF area.

[0098] Furthermore, in the above embodiment, the trained model LM is stored in the memory 42 within the main unit 11, but it may also be stored in an external device such as a computer connected via a network such as the Internet. In this case, the distance map acquisition unit 56 may input image information to the trained model LM of the external device via the network and acquire the distance map output from the trained model LM via the network.

[0099] Furthermore, the trained model LM may be incorporated into the processor 40. Even in this case, the processing unit that generates the distance map and the processing unit that performs imaging settings are different, so the processing unit that performs imaging settings will obtain the distance map from the processing unit that generates the distance map. Note that the method for generating the distance map is not limited to the method using the trained model LM. For example, it is possible to obtain the distance map using an infrared distance camera or the like.

[0100] Furthermore, the technology disclosed herein is not limited to digital cameras, but can also be applied to electronic devices such as smartphones and tablet devices that have a camera function.

[0101] In the above embodiments, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In this case, the processor is configured to work in cooperation with the program to execute the various processes in the above embodiments, and can function as a unit or means in the above embodiments. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0102] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these multiple hardware components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0103] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0104] Furthermore, although the above embodiment describes a configuration in which the program 43 is pre-stored (installed) in the memory 42, the invention is not limited to this configuration. The program 43 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program 43 may be provided in the form of a download from an external device via a network.

[0105] The technology disclosed herein extends to all program products. A program product includes all forms of products for providing programs. For example, a program product includes programs provided via a network such as the Internet, and non-temporary computer-readable recording media such as CD-ROMs, DVDs, and USB memory sticks on which programs are stored.

[0106] The following technologies can be understood from the above description. [Note 1] An imaging system comprising a processor, wherein the processor acquires relative distance information within a region set in an image, and determines imaging settings based on the relative distance information. [Note 2] The imaging system according to Note 1, wherein the processor acquires information regarding the relative distance between objects within the region as the relative distance information, and determines the imaging settings to be executed using the image information within the region based on the relative distance information. [Note 3] The imaging system according to Note 2, wherein the processor inputs the image information to a machine learning model that has learned the relationship between the relative distance between objects within the region and the image information, and acquires the output data from the machine learning model as the relative distance information. [Note 4] The imaging system according to any one of Notes 1 to 3, wherein the processor classifies the region into a plurality of groups based on the relative distance information, and determines the imaging settings for a selected group including one or more groups selected from the plurality of groups. [Note 5] The imaging system according to Note 4, wherein the imaging settings are settings related to focusing. [Addendum 6] The imaging system according to Addendum 5, wherein the processor divides the region into a plurality of sub-regions and determines the setting for focusing based on a plurality of defocus amounts corresponding to the plurality of sub-regions included in the selection group. [Addendum 7] The imaging system according to Addendum 6, wherein the processor excludes a portion of the plurality of groups and the remainder constitutes the selection group. [Addendum 8] The imaging system according to Addendum 6 or Addendum 7, wherein the processor determines the setting for focusing based on an index relating to the frequency of the plurality of defocus amounts. [Addendum 9] The imaging system according to Addendum 8, wherein the index is the median value. [Addendum 10] The imaging system according to any one of Addendum 6 to Addendum 9, wherein the image sensor that acquires the captured image includes a plurality of phase difference detection pixels, and the processor determines the defocus amount based on the output values ​​of the plurality of phase difference detection pixels corresponding to the sub-regions.[Note 11] The imaging system according to any one of Note 4 to Note 10, wherein the processor selects the group closest to the imaging system from among the plurality of groups as the selection group. [Note 12] The imaging system according to any one of Note 6 to Note 10, wherein the processor determines the reliability of the relative distance information for each of the plurality of small regions and classifies the plurality of small regions into the plurality of groups based on the reliability. [Note 13] The imaging system according to Note 12, wherein the processor selects the group with a certain level of reliability from among the plurality of groups as the selection group. [Note 14] The imaging system according to Note 12 or Note 13, wherein the processor determines the reliability based on at least one of the distribution of relative distances within the small region and the step difference in relative distances within the small region. [Note 15] The imaging system according to Note 14, wherein the reliability is higher the smaller the spread of the distribution or the larger the step difference. [Addendum 16] The imaging system according to any one of Addendum 1 to 3, wherein the processor divides the area into a plurality of sub-regions, classifies the plurality of sub-regions into a plurality of groups based on the relative distance information, and determines the imaging settings based on the weights determined for each of the plurality of groups. [Addendum 17] The imaging system according to Addendum 16, wherein the imaging settings are exposure or white balance settings. [Addendum 17] The imaging system according to Addendum 17, wherein the processor changes the photometric values ​​corresponding to the sub-regions based on the weights.

Claims

1. An imaging system comprising a processor, wherein the processor acquires relative distance information within a region set in an image, and determines imaging settings based on the relative distance information.

2. The imaging system according to claim 1, wherein the processor acquires information regarding the relative distance between objects within the region as relative distance information, and determines the imaging settings to be executed using the image information within the region based on the relative distance information.

3. The imaging system according to claim 2, wherein the processor inputs the image information to a machine learning model that has learned the relationship between the relative distance between objects in the region and the image information, and obtains the output data from the machine learning model as the relative distance information.

4. The imaging system according to claim 1, wherein the processor classifies the region into a plurality of groups based on the relative distance information, and determines the imaging settings for a selected group that includes one or more groups selected from the plurality of groups.

5. The imaging system according to claim 4, wherein the imaging setting is a setting related to focusing.

6. The imaging system according to claim 5, wherein the processor divides the region into a plurality of sub-regions and determines the setting for focusing based on a plurality of defocus amounts corresponding to the plurality of sub-regions included in the selection group.

7. The imaging system according to claim 6, wherein the processor comprises the selected group, with some of the groups excluded.

8. The imaging system according to claim 6, wherein the processor determines the setting for focusing based on an index relating to the frequency of the plurality of defocus amounts.

9. The imaging system according to claim 8, wherein the index is the median value.

10. The imaging system according to claim 6, wherein the image sensor that acquires the captured image includes a plurality of phase difference detection pixels, and the processor determines the amount of defocus based on the output values ​​of the plurality of phase difference detection pixels corresponding to the small region.

11. The imaging system according to any one of claims 4 to 10, wherein the processor selects the group that is closest in distance to the imaging system from among the plurality of groups.

12. The imaging system according to any one of claims 6 to 10, wherein the processor determines the reliability of the relative distance information for each of the plurality of subregions and classifies the plurality of subregions into the plurality of groups based on the reliability.

13. The imaging system according to claim 12, wherein the processor selects a group from among the plurality of groups whose reliability is above a certain level.

14. The imaging system according to claim 13, wherein the processor determines the reliability based on at least one of the distribution of relative distances within the small region and the step difference in relative distances within the small region.

15. The imaging system according to claim 14, wherein the reliability is higher the smaller the spread of the distribution or the larger the step difference.

16. The imaging system according to claim 1, wherein the processor divides the region into a plurality of subregions, classifies the plurality of subregions into a plurality of groups based on the relative distance information, and determines the imaging settings based on the weight determined for each of the plurality of groups.

17. The imaging system according to claim 16, wherein the imaging setting is an exposure or white balance setting.

18. The imaging system according to claim 17, wherein the processor modifies the photometric value corresponding to the small region based on the weight.

19. An operating method for an imaging system comprising a processor, wherein the processor performs a process including acquiring relative distance information within a region set in an image and determining imaging settings based on the relative distance information.

20. A program for operating an imaging system equipped with a processor, which causes the processor to perform a process including acquiring relative distance information within a region set in an image, and determining imaging settings based on the relative distance information.

Images