Imaging device, imaging method, and storage medium

Abstract

The present application provides a kind of camera device, camera method and camera program capable of inhibiting the misoperation caused by the subject being masked as moving object.The digital camera (100) detects the moving object (81) existing in the multiple pixel data obtained by the imaging element (5) at different time points.And, the digital camera (100) carries out the detection processing of the masking of the object (82) at future time t3 to the moving object (81) according to the comparison result of the first distance and the second distance, the first distance is the predicted distance of the moving object (81) at future time t3, and the second distance is the distance of the object existing in the predicted position coordinates of the moving object (81) at future time t3.And, the digital camera (100) carries out the camera control according to the result of the detection processing.

Description

Technical Field

[0001] This invention relates to a camera device, camera method, and camera program. Background Technology

[0002] In recent years, with the increasing resolution of imaging elements such as CMOS (Complementary Metal Oxide Semiconductor) image sensors, the demand for information devices with video recording capabilities, such as digital cameras, digital camcorders, and smartphones, has increased dramatically. Furthermore, information devices with such video recording capabilities are referred to as video recording devices. In these video recording devices, either contrast-based autofocus (AF) or phase-difference autofocus (AF) is used to control focus on the main subject.

[0003] Patent document 1 describes a camera device that detects obstructions around the tracking position by comparing the relative distance information of the tracking position to the subject with the relative distance information of the surrounding area of ​​the tracking position.

[0004] Patent document 2 describes a focus adjustment device that, when a camera transitions from a state of detecting a subject to a state of not detecting a subject, if the focus of a specified area is higher than a preset threshold, then the driving of the focus adjustment component is suppressed for a preset time after the subject is no longer detected.

[0005] Patent document 3 describes a camera device that detects the position of a tracked object in a camera image by evaluating the correlation between a camera image from a tracking sensor and a reference image. The camera device tracks the subject that is in focus during focus adjustment as the main subject and tracks the subject located in front of the main subject when viewed from the camera device as the masking subject. Focus adjustment is performed at a ranging point that exists on the main subject but not on the masking subject.

[0006] Previous technical documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2013-131996

[0009] Patent Document 2: Japanese Patent Application Publication No. 2010-156851

[0010] Patent Document 3: Japanese Patent Application Publication No. 2014-202875 Summary of the Invention

[0011] One embodiment of the present invention provides a camera device, camera method, and camera program capable of suppressing malfunctions caused by the obscuring of the subject, which is a moving object.

[0012] means for solving technical problems

[0013] One embodiment of the camera device according to the present invention includes an imaging element that captures a subject through a camera optical system and a processor. The processor performs the following processing: detecting a moving object present in multiple pixel data obtained by the imaging element at different times; performing a detection process to detect the masking of the moving object by the object at a future time based on a comparison result of a first distance and a second distance, wherein the first distance is a predicted distance of the moving object at a future time, the second distance is the distance of an object present in the predicted position coordinates of the moving object at a future time, and performing camera control based on the result of the detection process.

[0014] One embodiment of the imaging method involved in the present invention includes an imaging element that captures a subject through an imaging optical system and a processor. The processor performs the following processing: detecting a moving object present in multiple pixel data obtained by the imaging element at different times; performing a detection process to detect the object's masking of the moving object at a future time based on a comparison result of a first distance and a second distance; wherein the first distance is a predicted distance of the moving object at a future time, the second distance is the distance of an object present in the predicted position coordinates of the moving object at a future time, and performing camera control based on the result of the detection process.

[0015] The camera program of one embodiment of the present invention is a camera program of a camera device including an imaging element that captures a subject through a camera optical system and a processor. The camera program is used to cause the processor to perform the following processing: detecting a moving object present in multiple pixel data obtained by the imaging element at different times; performing a detection process to detect the masking of the moving object by the object at a future time based on a comparison result of a first distance and a second distance, wherein the first distance is a predicted distance of the moving object at a future time, the second distance is the distance of the object present in the predicted position coordinates of the moving object at a future time, and performing camera control based on the result of the detection process.

[0016] Invention Effects

[0017] According to the present invention, a camera device, camera method, and camera program are provided that can suppress malfunctions caused by the obscuring of the subject, which is a moving object. Attached Figure Description

[0018] Figure 1 This is a diagram showing the schematic structure of a digital camera 100, which is one embodiment of the imaging device of the present invention.

[0019] Figure 2 This is a planar schematic diagram showing an example of the structure of the imaging element 5 mounted on the digital camera 100.

[0020] Figure 3 yes Figure 2 A magnified view of one AF region 53 shown.

[0021] Figure 4 It indicates composition Figure 3 The image shown is a pixel map for phase difference detection of any pair of rows.

[0022] Figure 5 This is a diagram showing the cross-sectional structure of pixel 52A used for phase difference detection.

[0023] Figure 6 This diagram shows a structure in which all pixels 51 contained in the imaging element 5 are used as pixels for shooting, and each pixel 51 is divided into two.

[0024] Figure 7 This is a flowchart illustrating an example of the masking detection process of a digital camera 100.

[0025] Figure 8 Figure 1 shows an example of using a digital camera 100 to detect the obstruction of a subject.

[0026] Figure 9 Figure 2 shows an example of using a digital camera 100 to detect the obstruction of a subject.

[0027] Figure 10 Figure 3 shows an example of using a digital camera 100 to detect the obstruction of a subject.

[0028] Figure 11 Figure 4 shows an example of using a digital camera 100 to detect the obstruction of a subject.

[0029] Figure 12 This is a flowchart illustrating a specific example 1 of camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0030] Figure 13 This is a flowchart illustrating a specific example 2 of camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0031] Figure 14This is a flowchart of a specific example 3 of the camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0032] Figure 15 This is a flowchart of a specific example 4 of the camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0033] Figure 16 This is a flowchart of a specific example 5 of the camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0034] Figure 17 This is a flowchart of a specific example 6 of the camera control of a digital camera 100 based on the detection results of the masking of a moving object 81.

[0035] Figure 18 This is a diagram showing the appearance of a smartphone 200, which is another embodiment of the camera device of the present invention.

[0036] Figure 19 It means Figure 18 The diagram shows the structure of the smartphone 200. Detailed Implementation

[0037] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0038] Figure 1 This is a diagram showing the schematic structure of a digital camera 100, which is one embodiment of the imaging device of the present invention.

[0039] Figure 1 The digital camera 100 shown includes a lens assembly 40, which has an imaging lens 1, an aperture 2, a lens control unit 4, a lens drive unit 8, and an aperture drive unit 9.

[0040] In this embodiment, the lens device 40 can be attached to or detached from the main body of the digital camera 100, or it can be fixed to the main body of the digital camera 100.

[0041] Imaging lens 1 and aperture 2 constitute the camera optical system. Imaging lens 1 is in Figure 1 While described as a single lens, an imaging lens can be composed of multiple lenses, including at least one focusing lens, or it can consist of only one focusing lens. This focusing lens is used to adjust the focusing position of a camera optical system and can be composed of a single lens or multiple lenses. The focusing position is adjusted by moving the focusing lens along the optical axis of the camera optical system. Alternatively, a liquid lens can be used as the focusing lens, where the focusing position can be changed by variably controlling the curvature of the lens.

[0042] The lens control unit 4 of the lens device 40 is configured to communicate with the system control unit 11 of the main body of the digital camera 100 via wired or wireless means.

[0043] The lens control unit 4 drives the focusing lens contained in the imaging lens 1 via the lens drive unit 8, or drives the aperture 2 via the aperture drive unit 9, according to the instructions from the system control unit 11.

[0044] The main body of the digital camera 100 includes: an imaging element 5 such as a CMOS image sensor or a CCD image sensor that captures the subject through a camera optical system; an analog signal processing unit 6 connected to the output of the imaging element 5, which performs analog signal processing such as correlation double sampling; an analog-to-digital conversion circuit 7 that converts the analog signal output from the analog signal processing unit 6 into a digital signal; an imaging element drive unit 10; a system control unit 11 that centrally controls the entire system; and an operation unit 14. Alternatively, the analog signal processing unit 6 and the analog-to-digital conversion circuit 7 can be integrated with the imaging element 5.

[0045] The analog signal processing unit 6, the analog-to-digital conversion circuit 7, and the imaging element driving unit 10 are controlled by the system control unit 11. The imaging element driving unit 10 may also be included in the system control unit 11.

[0046] The system control unit 11 drives the imaging element 5 via the imaging element drive unit 10, and outputs the image of the subject captured by the camera optical system as a captured image signal. Command signals from the user are input to the system control unit 11 via the operation unit 14.

[0047] The system control unit 11 consists of a processor, RAM (Random Access Memory), and ROM (Read Only Memory) storage such as flash memory. When using flash memory, the stored program can be rewritten as needed.

[0048] The system control unit 11 performs the functions described later by executing a program containing a camera program stored in the built-in ROM.

[0049] Furthermore, the electrical control system of the digital camera 100 includes: a main memory 16; a memory control unit 15 connected to the main memory 16; a digital signal processing unit 17 that processes the image signal output from the analog-to-digital converter 7 to generate image data; a contrast AF processing unit 18 that calculates the focus position using contrast AF; a phase difference AF processing unit 19 that calculates the focus position using phase difference AF; an external memory control unit 20 connected to a removable recording medium 21; and a display control unit 22 connected to a display unit 23 mounted on the back of the camera, etc.

[0050] The focus position is the position of the focusing lens that focuses on a particular subject. The focus position changes by driving the focusing lens included in the imaging lens 1. The contrast AF processing unit 18 and the phase difference AF processing unit 19 calculate the appropriate focus position for the target subject. The appropriate focus position for a particular subject changes depending on the distance between the digital camera 100 and the subject.

[0051] The appropriate focus position for the subject is calculated based on defocus data, for example, in phase-difference AF. Defocus data indicates the direction and degree of focus deviation relative to the subject. For example, in contrast AF, the focus position is calculated based on the subject's contrast.

[0052] The memory control unit 15, digital signal processing unit 17, contrast AF processing unit 18, phase difference AF processing unit 19, external memory control unit 20, and display control unit 22 are interconnected via control bus 24 and data bus 25, and are controlled by instructions from system control unit 11.

[0053] Figure 2 This is a planar schematic diagram showing an example of the structure of the imaging element 5 mounted on the digital camera 100.

[0054] The imaging element 5 has a light-receiving surface 50 on which a large number of pixels are arranged in a two-dimensional manner in the row direction X and the column direction Y orthogonal to the row direction X.

[0055] exist Figure 2 In the example, 63 focus detection areas (hereinafter referred to as AF areas) 53 are provided on the light-receiving surface 50. The focus detection area 53 is the area of ​​the object that becomes the focus (the area where the image of the subject that becomes the focus is formed).

[0056] In digital camera 100, from Figure 2 Select one or multiple consecutive AF areas 53 from the 63 AF areas 53 shown to control the focus on the subject being photographed through the selected AF area 53.

[0057] AF region 53 is a region that includes both imaging pixels and phase difference detection pixels. The portion of the light-receiving surface 50 other than AF region 53 is configured only with imaging pixels.

[0058] Figure 3 yes Figure 2 A magnified view of one AF region 53 shown.

[0059] Within the AF region 53, pixels 51 (square blocks in the figure) are arranged in a two-dimensional pattern. Each pixel 51 includes a photoelectric conversion unit such as a photodiode and a color filter formed above the photoelectric conversion unit. Alternatively, each pixel 51 may also be configured to perform light splitting through a photodiode structure without using a color filter.

[0060] exist Figure 3 In the image, pixel 51 (R pixel 51), which includes a color filter (R filter) that transmits red light, is marked with the character "R".

[0061] exist Figure 3 In the image, pixel 51 (G pixel 51), which includes a color filter (G filter) that transmits green light, is marked with the character "G".

[0062] exist Figure 3 In the image, pixel 51 (B pixel 51), which includes a color filter (B filter) that transmits blue light, is marked with the letter "B". The arrangement of the color filters in the light-receiving surface 50 is called a Bayer arrangement.

[0063] In AF region 53, a portion of G pixel 51 ( Figure 3 The pixels marked with shadows in the image become pixels 52A and 52B for phase difference detection. Figure 3 In the example, each G pixel 51 in any pixel row including R pixel 51 and G pixel 51 becomes a phase difference detection pixel 52A, and the G pixel 51 of the same color that is closest to each G pixel 51 in the column direction Y becomes a phase difference detection pixel 52B.

[0064] A phase difference detection pixel 52A and a phase difference detection pixel 52B of the same color that is closest to the phase difference detection pixel 52A in the column direction Y form a pair. However, the configuration of these phase difference detection pixels is only one example and can also be other configurations. For example, a portion of G pixels 51 can be phase difference detection pixels, or they can be configured in R pixels 51 or B pixels 51.

[0065] By located Figure 3 The phase difference detection pixel 52A in the third row of pixels from the top and located in Figure 3 The phase difference detection of the fifth row of pixels from the top is constructed by pixels 52B, which is composed of multiple pairs arranged in the row direction X, forming a pair row PL1.

[0066] By located Figure 3 The phase difference detection pixel 52A in the 7th row of pixels from the top and located in Figure 3 The phase difference detection of the 9th row of pixels from the top is composed of pixels 52B, which are multiple pairs arranged in the row direction X, forming a pair row PL2.

[0067] By located Figure 3 The phase difference detection pixel 52A in the 11th row of pixels from the top and located in Figure 3 The phase difference detection of the 13th row of pixels from the top is composed of pixels 52B, which are multiple pairs arranged in the row direction X, forming a pair row PL3.

[0068] Thus, in region AF 53, there are multiple pairs of rows arranged in the column direction Y.

[0069] Figure 4 It indicates composition Figure 3 The image shown is a pixel map for phase difference detection of any pair of rows.

[0070] The phase difference detection pixel 52A is a first signal detection unit that receives a light beam passing through a segmented region of the pupil region of the imaging lens 1, which is divided into two regions in the row direction X, and detects a signal corresponding to the amount of light received.

[0071] The phase difference detection pixel 52B is a second signal detection unit that receives the light beam passing through another segmented region of the aforementioned pupil region and detects the signal corresponding to the amount of light received.

[0072] In addition, in the AF region 53, several pixels 51, other than the phase difference detection pixels 52A and 52B, are imaging pixels. These imaging pixels receive light beams from both of the aforementioned two segmented regions of the pupil region of the imaging lens 1, and detect signals corresponding to the amount of light received.

[0073] A light-shielding film is provided above the photoelectric conversion section of each pixel 51, and an opening is formed on the light-shielding film to limit the light-receiving area of ​​the photoelectric conversion section.

[0074] The center of the aperture of the imaging pixel coincides with the center of the photoelectric conversion section of the imaging pixel. In contrast, the aperture of the phase difference detection pixel 52A ( Figure 4 The center of the hollow part is offset to one side (right side) relative to the center of the photoelectric conversion unit of the phase difference detection pixel 52A.

[0075] Furthermore, the aperture of pixel 52B for phase difference detection ( Figure 4 The center of the hollow part is offset to the other side (left side) relative to the center of the photoelectric conversion unit of the phase difference detection pixel 52B.

[0076] Figure 5 This is a diagram showing the cross-sectional structure of pixel 52A used for phase difference detection. For example... Figure 5 As shown, the opening c of the phase difference detection pixel 52A is offset to one side (right) relative to the photoelectric conversion unit PD.

[0077] like Figure 5As shown, by covering one side of the photoelectric conversion unit PD with a light-shielding film, light incident from the opposite direction to the direction covered by the light-shielding film can be selectively blocked.

[0078] With this structure, the phase difference in the row direction X in the images captured by the two pixel groups can be detected by a pixel group including a pixel group comprising a phase difference detection pixel 52A constituting an arbitrary pair of rows and a pixel group including a phase difference detection pixel 52B constituting the pair of rows.

[0079] Furthermore, the pixel structure of imaging element 5 is not limited to Figures 2-5 The structure shown.

[0080] For example, all the pixels contained in the imaging element 5 can be used as imaging pixels, and each imaging pixel can be divided into two parts in the row direction X, with one part used as phase difference detection pixel 52A and the other part used as phase difference detection pixel 52B.

[0081] Figure 6 This diagram shows a structure in which all pixels 51 contained in the imaging element 5 are used as pixels for shooting, and each pixel 51 is divided into two.

[0082] exist Figure 6 In the structure, the pixel 51 marked with R in the imaging element 5 is divided into two, and the two divided pixels are used as phase difference detection pixels r1 and r2, respectively.

[0083] Furthermore, in the imaging element 5, the pixel 51 marked with G is divided into two, and the two divided pixels are used as pixel g1 and pixel g2 for phase difference detection, respectively.

[0084] Furthermore, in the imaging element 5, the pixel 51 marked with B is divided into two, and the two divided pixels are used as phase difference detection pixel b1 and phase difference detection pixel b2, respectively.

[0085] In this structure, pixels r1, g1, and b1 for phase difference detection become the first signal detection units, and pixels r2, g2, and b2 for phase difference detection become the second signal detection units.

[0086] exist Figure 6 In the structural example, if the signals from the first signal detection unit and the second signal detection unit contained in one pixel 51 are added together, a normal imaging signal without phase difference is obtained. That is, in Figure 6 In this structure, all pixels can be used as both phase difference detection pixels and imaging pixels. Furthermore, in... Figure 6 In the structural example, the degree of freedom in setting the size and shape of the AF area can be increased.

[0087] The system control unit 11 selectively performs either phase difference-based (AF) focusing control or contrast-based focusing control.

[0088] According to the command of the system control unit 11, the phase difference AF processing unit 19 uses the detection signal group read from the phase difference detection pixel 52A and phase difference detection pixel 52B selected from one or more AF regions 53 selected from 63 AF regions 53 by user operation, etc., to calculate the relative misalignment amount, i.e., the phase difference, of the two images formed by the pair of beams.

[0089] Furthermore, the phase difference AF processing unit 19 determines the focusing adjustment state of the imaging lens 1 based on this phase difference, which is here expressed as defocus amount and defocus direction data. The defocus amount is the difference between the current focusing position and the current position of the focusing lens, representing how much the focusing lens needs to be moved to reach the focusing position. The defocus direction indicates the direction in which the focusing lens is moved. The driving amount and driving direction of the focusing lens can be calculated based on the defocus data and the current lens data.

[0090] The system control unit 11 drives the focusing lens according to the defocus data and performs focusing control based on the phase difference AF method using the relevant calculation results.

[0091] The contrast AF processing unit 18 analyzes the image captured by the imaging element 5 and determines the focus position of the imaging lens 1 by contrast AF.

[0092] That is, the contrast AF processing unit 18 moves the focusing lens position of the imaging lens 1 under the control of the system control unit 11, and calculates the contrast (brightness difference) of the captured image at each (multiple) position after the movement. Furthermore, the focusing lens position where the contrast is maximized is determined as the maximum contrast focusing position.

[0093] The system control unit 11 drives the focusing lens according to the maximum contrast focusing position determined by the contrast AF processing unit 18, thereby performing focus control based on the contrast AF method that utilizes the contrast of the image.

[0094] Furthermore, the digital camera 100 can also be equipped with a continuous AF mode, which performs focus control on the subject multiple times in succession. For example, the digital camera 100 performs focus control on the subject multiple times in succession while displaying a live view image (instant preview image) that represents the image captured by continuous shooting through the imaging element 5.

[0095] Figure 7This is a flowchart illustrating an example of the mask detection process of a digital camera 100. The digital camera 100 repeatedly performs, for example, mask detection processing to detect the masking of the main subject. Figure 7 The process is shown. This process is performed, for example, by at least one of the system control unit 11 and the phase difference AF processing unit 19.

[0096] The term "masking" here refers to a situation where, when a subject is located in the shooting direction (front) of the digital camera 100, there are other objects between the subject and the digital camera 100, and due to these other objects, the subject is not visible from the digital camera 100, making it impossible to photograph the subject with the digital camera 100.

[0097] First, the digital camera 100 performs subject detection processing (step S701). The subject can be a subject specified by the user or a subject automatically detected by the digital camera 100.

[0098] Next, the digital camera 100 determines whether a subject has been detected in step S701 (step S702). If no subject is detected (step S702: "No"), the digital camera 100 invalidates the masking flag (step S703) and ends the series of processes. The masking flag is information indicating whether a moving object is masked, and is stored in the memory of the digital camera 100 (e.g., main memory 16).

[0099] In step S702, if a subject is detected (step S702: "Yes"), the digital camera 100 determines whether the subject detected in the previous step S701 is the same subject as the subject detected in the previous step S701 (step S704). For example, the determination of whether the detected subjects are the same is based on the similarity of the detected subjects. If the subjects are not the same (step S704: "No"), the subject is not identified as a moving object at the current moment, so the digital camera 100 proceeds to step S703 to invalidate the masking flag.

[0100] In step S704, if the subject is the same (step S704: "Yes"), the digital camera 100 calculates the defocus data of the subject detected in the previous step S701 (step S705). For example, the digital camera 100 calculates the defocus data in the latest pixel data obtained from the imaging element 5, based on the phase difference information of the subject area detected in the previous step S701.

[0101] As mentioned above, defocus data includes the amount and direction of defocus. The precision of the subject's defocus data is not particularly limited; for example, the subject's defocus data can also be information on both "far" and "near." However, in phase-difference AF, if the defocus data is known, the distance between the subject and the digital camera 100 can be calculated using the lens data and the defocus data at that time as the defocus data. In contrast-difference AF, defocus data can also be calculated based on the focus position.

[0102] Next, the digital camera 100 calculates defocus data D1 as a predicted value of the defocus data of the subject at a future time t3, based on the defocus data calculated in the previous step S705 and the defocus data calculated for the same subject in the previous step S705 (step S706). The future time t3 is, for example, the time of the next lens drive (the drive of the focusing lens of the imaging lens 1).

[0103] Next, the digital camera 100 calculates the predicted XY coordinates, which are the predicted values ​​of the XY coordinates of the subject detected in step S701 at a future time t3 (step S707). The XY coordinates are the position coordinates representing the position within the two-dimensional image.

[0104] Next, the digital camera 100 calculates defocus data D2, which is based on the defocus data of the predicted XY coordinates calculated in step S707 (step S708). For example, the digital camera 100 calculates defocus data based on the phase difference information of the portion of the predicted XY coordinates in the latest pixel data obtained from the imaging element 5 as defocus data D2.

[0105] Next, the digital camera 100 determines whether the focus position based on the defocus data D1 calculated in step S706 is farther than the focus position based on the defocus data D2 calculated in step S708 (in... Figure 7 In this context, it is expressed as "D1 > D2?". (Step S709).

[0106] In step S709, if the focus position based on defocus data D1 is not farther than the focus position based on defocus data D2 (step S709: "No"), it can be predicted that the subject detected by the just-conducted step S701 will not be obscured by the obstruction at time t3. In this case, the digital camera 100 proceeds to step S703, deactivating the obstruction flag.

[0107] In step S709, if the focus position based on defocus data D1 is farther than the focus position based on defocus data D2 (step S709: "Yes"), it can be predicted that the subject detected by the just-conducted step S701 will be obscured by the mask at time t3. In this case, the digital camera 100 activates the masking flag (step S710) and ends the series of processes.

[0108] Figures 8-11 This is an example of using a digital camera 100 to detect the concealment of a subject. Figure 8 The pixel data PD1 shown is the pixel data obtained by imaging element 5 at time t1. Figure 9 The pixel data PD2 shown is the pixel data obtained by imaging element 5 at time t2 after time t1. Figure 10 The pixel data PD3 shown is the pixel data obtained by imaging element 5 at time t3 after time t2. Figure 11 The pixel data PD4 shown is pixel data acquired by imaging element 5 at time t4, after time t3. Additionally, in Figures 8-11 The image shows the de-mosaic processing performed on pixel data PD1 to PD4.

[0109] Moving object 81 is a moving object (in this example, a person) existing within the imaging area of ​​digital camera 100 at times t1 to t4. For example, suppose digital camera 100 detects moving object 81 as a subject through subject detection processing based on pixel data PD1 and PD2. Figures 8-11 In the example, at times t1 to t4, when observed from the digital camera 100, the moving object 81 moves from the left front to the right rear.

[0110] Object 82 is a stationary object (in this example, a telephone pole) existing within the imaging area of ​​digital camera 100 from time t1 to t4. For example, at time t3, moving object 81 is located behind object 82, resulting in at least a portion of moving object 81 being obscured by object 82. Furthermore, at time t4, when observed from digital camera 100, moving object 81 is located to the right of object 82, thus eliminating the obscuration of moving object 81 by object 82.

[0111] As an example, the processing of the digital camera 100 at time t2 will be explained. Here, it is assumed that at time t1, the digital camera 100 has detected the moving object 81 as the subject and has also calculated the defocus data of the moving object 81 at time t1. At time t2, the digital camera 100 detects the moving object 81 again as the subject and calculates the defocus data of the moving object 81 at time t2.

[0112] Next, the digital camera 100 calculates the defocus data D1 of the moving object 81 at time t3, which is a future time. For example, the digital camera 100 calculates the defocus data D1 of the moving object 81 at time t3 based on the calculation results of the defocus data of the moving object 81 at times t1 and t2.

[0113] Furthermore, the digital camera 100 calculates the predicted XY coordinates of the moving object 81 at time t3. For example, the digital camera 100 calculates the predicted XY coordinates of the moving object 81 at time t3 based on the detection results of the moving object 81 at times t1 and t2.

[0114] Next, the digital camera 100 uses the calculated predicted XY coordinates to calculate the defocus data D2. For example, the digital camera 100 calculates the defocus data D2 based on the phase difference information in the portion of the pixel data PD2 obtained by the imaging element 5 at time t2, which is based on the calculated predicted XY coordinates.

[0115] Furthermore, the digital camera 100 compares and calculates the defocus data D1 and D2. Figures 8-11 In the example, at time t3, the moving object 81 is behind the object 82, so the focus position based on defocus data D1 is farther than the focus position based on defocus data D2. Therefore, the digital camera 100 predicts that the moving object 81 will be masked by the object 82 at time t3, making the masking flag effective.

[0116] Thus, the digital camera 100 detects a moving object 81 present in the pixel data PD1 and PD2 (multiple pixel data) obtained by the imaging element 5 at times t1 and t2 (different times). Furthermore, the digital camera 100 calculates defocus data D1 (first distance), which is the predicted distance of the moving object 81 at a future time t3. Also, the digital camera 100 calculates defocus data D2 (second distance), which is the distance of the object 82 present in the predicted XY coordinates (predicted position coordinates) of the moving object 81 at a future time t3.

[0117] Furthermore, the digital camera 100 performs detection processing to determine whether the object 82 will mask the moving object 81 at a future time t3, based on the comparison results of defocus data D1 and D2. Specifically, if the focus position based on defocus data D1 is farther than the focus position based on defocus data D2, the digital camera 100 determines that the object 82 will mask the moving object 81 at time t3. The digital camera 100 then performs camera control based on the result of this detection processing. This suppresses erroneous camera control caused by the moving object 81 being masked by the object 82.

[0118] Digital camera 100 calculates the predicted XY coordinates of the moving object 81 at a future time t3, for example, based on the XY coordinates (position coordinates) of the moving object 81 in pixel data PD1, PD2 (multiple pixel data). However, the predicted XY coordinates of the moving object 81 at a future time t3 are not limited to pixel data PD1, PD2, and can be calculated based on the XY coordinates of the moving object 81 in any multiple pixel data obtained by imaging element 5 before the future time t3.

[0119] Furthermore, the digital camera 100 calculates defocus data D1 (first distance) as the predicted distance of the moving object 81 at a future time t3, for example, based on the detection results of the distance of the moving object 81 at times t1, t2 (multiple times). However, the defocus data D1 is not limited to times t1 and t2, and can be calculated based on the detection results of the distance of the moving object 81 at any multiple times before the future time t3.

[0120] Furthermore, the digital camera 100 calculates the defocus data of object 82 at time t2, which is the distance of object 82 in the predicted XY coordinates of moving object 81 existing in the future time t3, i.e., defocus data D2 (second distance). However, the defocus data D2 is not limited to the defocus data of object 82 at time t2, but can also be the defocus data of object 82 at any time before time t3.

[0121] Figure 12 This is a flowchart illustrating a specific example 1 of camera control for a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 repeatedly performs, for example... Figure 7 The processing and Figure 12 The processing shown. Figure 12 The processing is performed, for example, by at least one of the system control unit 11 and the phase difference AF processing unit 19. Furthermore, it is performed, for example, during the timing of each lens drive (the drive of the focusing lens of the imaging lens 1). Figure 12 The processing.

[0122] First, the digital camera 100 determines whether the masking flag is valid (step S121). If the masking flag is valid (step S121: "No"), it can be inferred that during the timing of this lens drive, the moving object 81, which is the subject, is not masked by the object 82. In this case, the digital camera 100 performs lens drive based on the latest detection result of the defocus data of the subject (step S122) and ends a series of processes. Specifically, the digital camera 100 drives the focusing lens of the imaging lens 1 based on the detection result of the defocus data of the phase difference information based on the current XY coordinates of the subject in the latest pixel data obtained from the imaging element 5.

[0123] In step S121, if the masking flag is valid (step S121: "Yes"), the digital camera 100 can infer that during the timing of this lens drive, the moving object 81, which is the subject, is masked by the object 82. In this case, the digital camera 100 maintains the driving state of the focusing lens of the imaging lens 1 (step S123) and ends a series of processes. Through step S123, the lens drive based on the latest detection result of the defocus data of the subject can be stopped.

[0124] For example, due to the masking markers in Figures 8-11 In the example shown, time t3 is valid, so the digital camera 100, for example, maintains the driving state of the focusing lens of the imaging lens 1 as a driving state based on the pixel data obtained at time t2.

[0125] Thus, upon detecting that the moving object 81 is obstructing the view at time t3, the digital camera 100 stops driving the focusing lens of the imaging lens 1 at time t3 based on the detection result of the defocus data of the moving object 81, wherein the defocus data of the moving object 81 is based on the latest pixel data. This suppresses erroneous focusing on the object 82 obstructing the moving object 81.

[0126] Furthermore, if the focusing lens of the imaging lens 1 is stopped when the moving object 81 is detected as masked at time t3, and the driving of the focusing lens of the imaging lens 1 based on the detection result of the defocus data of the moving object 81 at time t3 is stopped, the digital camera 100 can calculate time t4 after time t3 as the predicted time for eliminating the masking of the moving object 81. For example, the digital camera 100 calculates time t4 after time t3, based on the detection results of the XY coordinates of the moving object 81 at times t1 and t2, at which the XY coordinates of the moving object 81 deviate from the XY coordinates of the object 82. Then, the digital camera 100 restarts the driving of the imaging lens 1 of the imaging optical system based on the detection result of the defocus data of the moving object 81 at time t4. Thus, even if the driving of the focusing lens of the imaging lens 1 based on the detection result of the defocus data of the moving object 81 is stopped, the driving of the focusing lens of the imaging lens 1 based on the detection result of the defocus data of the moving object 81 can be restarted at time t4 when the masking of the moving object 81 is eliminated.

[0127] Furthermore, the digital camera 100 can, for example, calculate the predicted XY coordinates of the moving object 81 at time t4 (when the masking of the moving object 81 is removed) based on the detection results of the XY coordinates of the moving object 81 at times t1 and t2, and perform subject detection processing based on the data in the pixel data obtained by the imaging element 5 at time t4 corresponding to the calculated predicted XY coordinates of the moving object 81 at time t4. This allows for the reduction of the object area for subject detection processing based on the pixel data obtained at time t4 (when the masking of the moving object 81 is removed). Therefore, the processing time required for subject detection processing can be shortened. Furthermore, the accuracy of subject detection processing can be improved.

[0128] Figure 13 This is a flowchart illustrating a specific example 2 of camera control of a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 may also repeatedly execute, for example... Figure 7 The processing and Figure 13 The processing is shown. Figure 13 The processing is performed, for example, by at least one of the system control unit 11 and the phase difference AF processing unit 19. Furthermore, it is performed, for example, during the timing of each lens drive. Figure 12 The processing.

[0129] First, the digital camera 100 determines whether the masking mark is valid (step S131). If the masking mark is invalid (step S131: "No"), the digital camera 100 proceeds to step S132. Step S132 and... Figure 12 The steps shown are the same as S122.

[0130] In step S131, if the masking flag is valid (step S131: "Yes"), the digital camera 100 does not perform lens driving based on the latest detection result of the subject's defocus data, but instead performs lens driving based on the detection results of the subject's defocus data at multiple past moments (step S133), and ends a series of processes.

[0131] For example, due to the masking markers in Figures 8-11 In the example shown, time t3 is valid. Therefore, the digital camera 100 calculates the predicted value of the defocus data of the moving object 81 at time t3 based on the phase difference information of each pixel data obtained by the imaging element 5 at times t1 and t2, and drives the focusing lens of the imaging lens 1 based on the calculated predicted value of the defocus data.

[0132] Thus, if the moving object 81 is detected to be obscured at time t3, the digital camera 100 can drive the focusing mechanism of the imaging lens 1 at time t3 based on the detection results of the defocus data of the moving object 81 at multiple times prior to time t3. Therefore, focus control can continue even when the moving object 81 is obscured. Furthermore, the multiple times prior to time t3 are not limited to times t1 and t2.

[0133] Figure 14 This is a flowchart illustrating a specific example 3 of camera control for a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 may also repeatedly execute, for example... Figure 7 The processing and Figure 14 The processing is shown. Figure 14 The processing is performed, for example, by the system control unit 11. And, for example, it is performed at each subject detection timing. Figure 14 The subject detection timing can be any timing that acquires pixel data from the imaging element 5, or it can be a timing within any timing that acquires pixel data from the imaging element 5 and contains processing resources capable of performing subject detection processing.

[0134] First, the digital camera 100 determines whether the masking flag is valid (step S141). If the masking flag is invalid (step S141: "No"), the digital camera 100 performs subject detection processing, including subject detection processing based on the latest pixel data (step S142), and ends a series of processes. The subject detection processing in step S142 is, for example,... Figure 7 The process is as shown. In step S141, if the masking flag is valid (step S141: "Yes"), the digital camera 100 does not perform subject detection processing and ends a series of processes.

[0135] Thus, the digital camera 100 can be configured to perform subject detection processing based on pixel data obtained from the imaging element 5 during the period when the masking flag is valid, and to stop the subject detection processing based on pixel data obtained from the imaging element 5 during the period when the masking flag is invalid (e.g., time t3). This can suppress false detections caused by the moving object 81 being masked by the object 82, leading to the misjudgment that the moving object 81 does not exist. Furthermore, by stopping the subject detection processing, power consumption can be reduced. In addition to the process of re-detecting the subject, the subject detection processing can also include tracking processing to track the movement of the detected subject.

[0136] Figure 15 This is a flowchart illustrating a specific example 4 of camera control for a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 may also repeatedly execute, for example... Figure 7 The processing and Figure 15 The processing is shown. Figure 14 The processing is performed, for example, by the system control unit 11. And, for example, it is performed during each of the aforementioned subject detection timings. Figure 14 The processing.

[0137] First, the digital camera 100 determines whether the masking mark is valid (step S151). If the masking mark is invalid (step S151: "No"), the digital camera 100 proceeds to step S152. Step S152 and... Figure 14 The steps shown are the same as S142.

[0138] In step S151, if the masking flag is valid (step S151: "Yes"), the digital camera 100 performs subject detection processing (step S152) and ends a series of processes. However, even if the subject detection processing in step S152 fails, the digital camera 100 maintains the detection state of the subject detected by the subject detection processing based on the pixel data obtained by the imaging element 5 previously.

[0139] For example, in Figures 8-11 In the example shown, even if the moving object 81, which was detected in the pixel data obtained at time t1 and t2, cannot be detected from the pixel data obtained at time t3 when the moving object 81 is obscured, the digital camera 100 does not determine that the moving object 81 does not exist, and continues to track the moving object 81.

[0140] Thus, the digital camera 100 can also be configured to maintain the detection state of the subject detected by the subject detection process based on the pixel data obtained by the imaging element 5 during the period when the masking flag is invalid (e.g., time t3), even if the subject detection processing fails. Therefore, in the event that the moving object 81 is temporarily masked, erroneous actions such as stopping the tracking processing of the moving object 81 or switching the subject detection target to an object different from the moving object 81 can be suppressed.

[0141] Figure 16 This is a flowchart illustrating a specific example 5 of camera control for a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 may also repeatedly execute, for example... Figure 7 The processing and Figure 16 The processing is shown. Figure 16 The processing is performed, for example, by the system control unit 11. Furthermore, for example, during continuous imaging by the imaging element 5, the process is performed whenever pixel data is acquired by the imaging element 5. Figure 16 The processing.

[0142] When the imaging element 5 is used for continuous shooting, it refers to the display of a real-time viewfinder image (instant preview image) in still image shooting mode, during dynamic image shooting, or during continuous shooting (burst shooting) of still images, etc.

[0143] First, the digital camera 100 determines whether the masking flag is valid (step S161). If the masking flag is invalid (step S161: "No"), the digital camera 100 records the image of the acquired pixel data (step S162) and ends a series of processes. Image recording based on pixel data refers to the recording of pixel data to the recording medium 21, or the recording of an image generated based on the pixel data to the recording medium 21.

[0144] In step S161, if the masking flag is valid (step S161: "Yes"), the digital camera 100 does not record the image of the acquired pixel data and ends a series of processes.

[0145] In example Figures 8-11 In the example shown, the digital camera 100 records the image of pixel data PD1 and PD2 obtained at times t1 and t2, but does not record the image of pixel data PD3 obtained at time t3.

[0146] Thus, the digital camera 100 can also stop image recording based on pixel data obtained by the imaging element 5 at time t3 if it detects that the moving object 81 is obscured at time t3. This prevents image recording based on pixel data obtained at time t3 when the moving object 81 is obscured, i.e., pixel data that suggests a high probability that the moving object 81 was not captured in a satisfactory state. Therefore, the effort and time of deleting unsatisfactory images can be saved for the user. Furthermore, the recording capacity of the recording medium 21 can be reduced.

[0147] Figure 17 This is a flowchart illustrating a specific example 6 of camera control for a digital camera 100 based on the detection results of the masking of a moving object 81. The digital camera 100 may also repeatedly execute, for example... Figure 7 The processing and Figure 17 The processing is shown. Figure 17 The processing is performed, for example, by the system control unit 11. Furthermore, for example, during continuous imaging by the imaging element 5, the process is performed whenever pixel data is acquired by the imaging element 5. Figure 17 The processing.

[0148] First, the digital camera 100 determines whether the masking mark is valid (step S171). If the masking mark is invalid (step S171: "No"), the digital camera 100 updates the exposure and white balance based on the pixel data obtained this time (step S172) and ends a series of processes.

[0149] In step S172, for example, the digital camera 100 calculates appropriate exposure and white balance to be applied to the pixel data based on the data of the subject portion in the pixel data obtained this time. Furthermore, the digital camera 100 performs processing such as recording the calculated exposure or white balance to the recording medium 21 or displaying a live view image based on the pixel data.

[0150] In step S171, if the masking flag is valid (step S171: "Yes"), the digital camera 100 maintains the exposure and white balance of the pixel data (step S173) and ends a series of processes.

[0151] For example, in Figures 8-11 In the example shown, the digital camera 100 maintains the exposure and white balance applicable to recording pixel data acquired at time t3 to the recording medium 21, displaying a live view image based on the pixel data acquired at time t3, etc., as calculated based on the data of the subject portion in the pixel data acquired at time t2. Furthermore, regarding exposure and white balance... Figure 17 The processing method has been explained, but it can also be set to process only one of exposure and white balance. Figure 17 The processing.

[0152] Thus, the digital camera 100 can also stop controlling at least one of the exposure and white balance based on the pixel data obtained by the imaging element 5 at time t3 if it detects that the moving object 81 is obstructing the view at time t3. This prevents changes to the exposure or white balance based on the pixel data obtained at time t3 when the moving object 81 is obstructed, i.e., pixel data that suggests a high probability that the subject, which serves as the basis for automatic exposure and white balance settings, has not been sufficiently captured.

[0153] Figure 18 This is a diagram showing the appearance of a smartphone 200, which is another embodiment of the camera device of the present invention.

[0154] Figure 18 The smartphone 200 shown has a flat frame 201, and a display input section 204 is provided on one side of the frame 201. The display input section 204 is formed by integrating a display panel 202 as a display section and an operation panel 203 as an input section.

[0155] Furthermore, this housing 201 includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. However, the structure of the housing 201 is not limited to this; for example, it can have a structure where the display unit and input unit are independent, or a structure with a folding structure or a sliding mechanism.

[0156] Figure 19 It means Figure 18 The diagram shows the structure of the smartphone 200.

[0157] like Figure 19 As shown, the main components of a smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiver unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0158] Furthermore, as a primary function of the smartphone 200, it possesses wireless communication capabilities, enabling mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0159] The wireless communication unit 210 performs wireless communication with the base station device BS housed in the mobile communication network NW according to the commands of the main control unit 220. Using this wireless communication, it sends and receives various types of data, such as voice data, image data, and email data, as well as receives network data or streaming data.

[0160] The display input unit 204 is a touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information to convey information to the user's vision and detects user operations on the displayed information. It also includes a display panel 202 and an operation panel 203.

[0161] Regarding the display panel 202, LCD (Liquid Crystal Display) and OELD (Organic Electro-Luminescence Display) are used as display devices.

[0162] The operation panel 203 is a device that allows visual recognition of images displayed on the display surface of the display panel 202, and detects one or more coordinates being operated by a user's finger or stylus. If the device is operated by the user's finger or stylus, a detection signal generated by the operation is output to the main control unit 220. Then, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0163] like Figure 19As shown, in an embodiment of the photographic apparatus of the present invention, the display panel 202 and the operation panel 203 of the smartphone 200 are integrated to form a display input unit 204, but the operation panel 203 is configured to completely cover the display panel 202.

[0164] With this configuration, the operation panel 203 can also detect user operations in areas other than the display panel 202. In other words, the operation panel 203 can have a detection area (hereinafter referred to as the display area) for the overlapping portion that overlaps with the display panel 202 and a detection area (hereinafter referred to as the non-display area) for the outer edge portion that does not overlap with the display panel 202.

[0165] Alternatively, the size of the display area can be exactly the same as the size of the display panel 202, but it is not necessary for them to be identical. Furthermore, the operation panel 203 can have two sensing areas: an outer edge and an inner portion. The width of the outer edge is appropriately designed based on the size of the frame 201, etc.

[0166] Furthermore, as for the position detection method used in the operation panel 203, matrix switch method, resistive film method, surface elastic wave method, infrared method, electromagnetic induction method, electrostatic capacitance method, etc. can be used, and any method can be adopted.

[0167] The call unit 211 includes a speaker 205 or a microphone 206, which converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes the voice data received through the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0168] And, as Figure 18 As shown, for example, the speaker 205 can be mounted on the same surface as the surface where the display input section 204 is provided, and the microphone 206 can be mounted on the side of the frame 201.

[0169] The operation unit 207 uses hardware keys such as push-button switches and receives commands from the user. For example, such as Figure 18 As shown, the operation unit 207 is mounted on the side of the frame 201 of the smartphone 200, and is a button-type switch that is turned on when pressed by a finger or the like, and turned off by the restoring force of a spring or the like when the finger is removed.

[0170] Storage unit 212 stores the control program and control data of main control unit 220, application software, address data that establishes corresponding associations for names or phone numbers of communication objects, data of sent and received emails, web data downloaded through a web browser, downloaded content data, and temporarily stores streaming data, etc. Furthermore, storage unit 212 consists of an internal storage unit 217 built into the smartphone and an external storage unit 218 with a removable external memory slot.

[0171] In addition, the internal storage units 217 and external storage units 218 constituting the storage unit 212 are implemented using storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., MicroSD memory), RAM (Random Access Memory), ROM (Read Only Memory).

[0172] The external input / output unit 213 serves as an interface for all external devices connected to the smartphone 200, and is used to connect directly or indirectly to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, etc.) or networks (e.g., Ethernet, Wireless LAN, etc.).

[0173] External devices that connect to the smartphone 200 include, for example, wired / wireless headphones, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card slots, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video devices connected via audio / video I / O (Input / Output) terminals, wireless external audio / video devices, wired / wireless smartphones, wired / wireless computers, wired / wireless personal computers, headphones, etc.

[0174] The external input / output unit 213 can be configured to transmit data received from such an external device to the internal components of the smartphone 200, or to transmit internal data of the smartphone 200 to an external device.

[0175] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn according to the command of the main control unit 220, performs positioning calculations based on the received multiple GNSS signals, and detects the location, including the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiver 214 can obtain location information from the wireless communication unit 210 or the external input / output unit 213 (e.g., a wireless LAN), it can also use that location information to detect its location.

[0176] The motion sensor unit 215, for example, includes a 3-axis accelerometer, and detects the physical movement of the smartphone 200 according to commands from the main control unit 220. By detecting the physical movement of the smartphone 200, it detects the direction of movement or acceleration of the smartphone 200. The detection result is then output to the main control unit 220.

[0177] The power supply unit 216 supplies the power stored in the battery (not shown) to each part of the smartphone 200 according to the command of the main control unit 220.

[0178] The main control unit 220 is equipped with a microprocessor and operates according to the control program and control data stored in the storage unit 212, thereby centrally controlling all parts of the smartphone 200. Furthermore, the main control unit 220 has mobile communication control functions and application processing functions for controlling the various parts of the communication system for voice or data communication via the wireless communication unit 210.

[0179] The application processing functions are implemented by the main control unit 220 according to the application software stored in the storage unit 212. Examples of application processing functions include infrared communication functions that control the external input / output unit 213 to communicate with the target device, email functions that send and receive emails, and web browser functions that browse web pages.

[0180] Furthermore, the main control unit 220 has image processing functions such as displaying images on the display input unit 204 based on received data or downloaded streaming data (data of still images or moving images).

[0181] The image processing function refers to the function of the main control unit 220 to decode the above-mentioned image data, perform image processing on the decoding result, and then display the image on the display input unit 204.

[0182] Furthermore, the main control unit 220 performs display control on the display panel 202 and operation detection control for user operations via the operation unit 207 and the operation panel 203.

[0183] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for launching application software, or displays a window for creating emails.

[0184] Additionally, the scroll bar refers to a software key used to receive commands for moving the display portion of a large image that cannot be converged to the display area of ​​the display panel 202.

[0185] Furthermore, by performing operation detection control, the main control unit 220 detects user operations through the operation unit 207, or receives operations on the aforementioned icons or input of strings into the input fields of the aforementioned windows through the operation panel 203, or receives scrolling requests for the displayed images via the scroll bar.

[0186] Furthermore, by performing operation detection control, the main control unit 220 has a touch panel control function that determines whether the operation position of the operation panel 203 is an overlapping part (display area) that overlaps with the display panel 202, or whether it is an outer edge part (non-display area) that does not overlap with the display panel 202, and controls the display position of the sensing area or software key of the operation panel 203.

[0187] Furthermore, the main control unit 220 can also detect gesture operations on the operation panel 203 and execute preset functions based on the detected gesture operations.

[0188] Gesture operation is not the simple touch operation of the past, but refers to the operation of drawing a trajectory with a finger or other means, or specifying multiple positions at the same time, or combining these to draw a trajectory from multiple positions to at least one position.

[0189] Camera Department 208 includes, in addition to Figure 1 The structure of the digital camera 100 shown, excluding the external memory control unit 20, recording medium 21, display control unit 22, display unit 23, and operation unit 14, is capable of recording the image data generated by the camera unit 208 in the storage unit 212, or outputting it through the external input / output unit 213 or the wireless communication unit 210.

[0190] exist Figure 18 In the smartphone 200 shown, the camera unit 208 is mounted on the same side as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it can also be mounted on the back of the display input unit 204.

[0191] Furthermore, the camera unit 208 can utilize various functions of the smartphone 200. For example, images captured by the camera unit 208 can be displayed on the display panel 202, or images from the camera unit 208 can be used as one of the operation inputs on the operation panel 203.

[0192] Furthermore, when the GNSS receiver 214 detects its position, it can also refer to the image from the camera unit 208 to detect the position. Moreover, it can refer to the image from the camera unit 208 to determine the optical axis direction of the smartphone 200's camera unit 208, or to determine the current operating environment, without using the 3-axis accelerometer, or by using it in conjunction with the 3-axis accelerometer. Of course, the image from the camera unit 208 can also be used within the application software.

[0193] In addition, the location information acquired by the GNSS receiver 214, the voice information acquired by the microphone 206 (which can be converted into text information by the main control unit, etc.), and the posture information acquired by the motion sensor 215 can be added to still image data or moving image data and stored in the storage unit 212, or output through the external input / output unit 213 or the wireless communication unit 210.

[0194] In the smartphone 200 with the structure described above, via Figure 1 The system control unit 11 shown can perform the above-described processing to suppress malfunctions caused by the subject being obscured as a moving object.

[0195] (Variation Example 1)

[0196] In the above embodiments, the case of detecting phase difference in the row direction X was used as an example, but the present invention can also be applied to the case of detecting phase difference in the column direction Y.

[0197] (Variation Example 2)

[0198] In the above embodiments, the case of AF detection performed by the phase difference AF processing unit 19 using the phase difference AF method has been described as AF detection. However, as AF detection, a structure in which the contrast AF processing unit 18 performs AF detection using the contrast method can also be used. Furthermore, as AF detection, a structure that combines the phase difference AF method and the contrast method can also be used.

[0199] As explained above, the following matters are disclosed in this specification. (1)

[0201] A camera device comprising an imaging element that captures images of a subject via a camera optical system, and a processor.

[0202] The processor described above performs the following processing:

[0203] Detecting moving objects present in multiple pixel data acquired by the aforementioned imaging elements at different times.

[0204] Based on the comparison between the first distance and the second distance, the detection process for the masking of the moving object by the object at the future time is performed. The first distance is the predicted distance of the moving object at the future time, and the second distance is the distance of the object existing in the predicted position coordinates of the moving object at the future time.

[0205] Camera control will be implemented based on the results of the above detection and processing. (2)

[0207] According to the camera device described in (1), wherein,

[0208] The processor calculates the predicted position coordinates of the moving object at the future time based on the position coordinates of the moving object in the plurality of pixel data. (3)

[0210] According to the camera device described in (1) or (2), wherein,

[0211] The processor calculates the first distance based on the detection results of the distances of the moving object at multiple times prior to the future time. (4)

[0213] The camera device according to any one of (1) to (3), wherein,

[0214] The processor calculates the distance of the object to the time preceding the future time as the second distance. (5)

[0216] The camera device according to any one of (1) to (4), wherein,

[0217] In the above detection process, the processor detects the masking when the first distance is longer than the second distance. (6)

[0219] The camera device according to any one of (1) to (5), wherein,

[0220] If the processor detects the masking through the detection process, it will stop driving the focusing mechanism of the camera optical system based on the detection result of the distance to the moving object at the future time. (7)

[0222] According to the camera device described in (6), wherein,

[0223] When the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it calculates a predicted time after the aforementioned future time to eliminate the aforementioned masking, and at the aforementioned predicted time, it restarts the driving of the focusing mechanism of the aforementioned camera optical system based on the detection result of the distance of the aforementioned moving object. (8)

[0225] The camera device according to any one of (1) to (7), wherein,

[0226] When the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it drives the focusing mechanism of the aforementioned camera optical system at the aforementioned future time based on the detection results of the distance of the aforementioned moving object at multiple times prior to the aforementioned future time. (9)

[0228] The camera device according to any one of (1) to (8), wherein,

[0229] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops the subject detection process based on the pixel data obtained by the aforementioned imaging element at the aforementioned future time. (10)

[0231] The camera device according to any one of (1) to (9), wherein,

[0232] Even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails, the processor maintains the detection state of the subject detected by the subject detection processing based on pixel data obtained by the imaging element before the future time, even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails when the above-mentioned detection processing detects the above-mentioned masking through the above-mentioned detection processing. (11)

[0234] The camera device according to any one of (1) to (10), wherein,

[0235] The processor described above performs the following processing:

[0236] If the aforementioned masking is detected, calculate the predicted time for the removal of the aforementioned masking after the aforementioned future time.

[0237] Calculate the predicted position coordinates of the moving object at the predicted time.

[0238] Subject detection processing is performed based on the data corresponding to the predicted position coordinates of the moving object at the predicted time, which are obtained from the pixel data of the imaging element at the predicted time. (12)

[0240] The camera device according to any one of (1) to (11), wherein,

[0241] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops image recording based on pixel data obtained by the aforementioned imaging element at the aforementioned future time. (13)

[0243] The camera device according to any one of (1) to (12), wherein,

[0244] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops controlling at least one of the exposure and white balance based on the pixel data obtained by the aforementioned imaging element at the aforementioned future time. (14)

[0246] A method for capturing images using a camera device, the camera device comprising an imaging element that captures images of a subject through a camera optical system, and a processor.

[0247] The processor described above performs the following processing:

[0248] Detecting moving objects present in multiple pixel data acquired by the aforementioned imaging elements at different times.

[0249] Based on the comparison between the first distance and the second distance, the detection process for the masking of the moving object by the object at the future time is performed. The first distance is the predicted distance of the moving object at the future time, and the second distance is the distance of the object existing in the predicted position coordinates of the moving object at the future time.

[0250] Camera control will be implemented based on the results of the above detection and processing. (15)

[0252] According to the imaging method described in (14), wherein,

[0253] The processor calculates the predicted position coordinates of the moving object at the future time based on the position coordinates of the moving object in the plurality of pixel data. (16)

[0255] According to the imaging method described in (14) or (15), wherein,

[0256] The processor calculates the first distance based on the detection results of the distances of the moving object at multiple times prior to the future time. (17)

[0258] The imaging method according to any one of (14) to (16), wherein,

[0259] The processor calculates the distance of the object to the time preceding the future time as the second distance. (18)

[0261] The imaging method according to any one of (14) to (17), wherein,

[0262] In the above detection process, the processor detects the masking when the first distance is longer than the second distance. (19)

[0264] The imaging method according to any one of (14) to (18), wherein,

[0265] If the processor detects the masking through the detection process, it will stop driving the focusing mechanism of the camera optical system based on the detection result of the distance to the moving object at the future time. (20)

[0267] According to the imaging method described in (19), wherein,

[0268] When the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it calculates a predicted time after the aforementioned future time to eliminate the aforementioned masking, and at the aforementioned predicted time, it restarts the driving of the focusing mechanism of the aforementioned camera optical system based on the detection result of the distance of the aforementioned moving object. (twenty one)

[0270] The imaging method according to any one of (14) to (20), wherein,

[0271] When the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it drives the focusing mechanism of the aforementioned camera optical system at the aforementioned future time based on the detection results of the distance of the aforementioned moving object at multiple times prior to the aforementioned future time. (twenty two)

[0273] The imaging method according to any one of (14) to (21), wherein,

[0274] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops the subject detection process based on the pixel data obtained by the aforementioned imaging element at the aforementioned future time. (twenty three)

[0276] The imaging method according to any one of (14) to (22), wherein,

[0277] Even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails, the processor maintains the detection state of the subject detected by the subject detection processing based on pixel data obtained by the imaging element before the future time, even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails when the above-mentioned detection processing detects the above-mentioned masking through the above-mentioned detection processing. (twenty four)

[0279] The imaging method according to any one of (14) to (23), wherein,

[0280] The processor described above performs the following processing:

[0281] If the aforementioned masking is detected, calculate the predicted time for the removal of the aforementioned masking after the aforementioned future time.

[0282] Calculate the predicted position coordinates of the moving object at the predicted time.

[0283] Subject detection processing is performed based on the data corresponding to the predicted position coordinates of the moving object at the predicted time, which are obtained from the pixel data of the imaging element at the predicted time. (25)

[0285] The imaging method according to any one of (14) to (24), wherein,

[0286] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops image recording based on pixel data obtained by the aforementioned imaging element at the aforementioned future time. (26)

[0288] The imaging method according to any one of (14) to (25), wherein,

[0289] If the aforementioned processor detects the aforementioned masking through the aforementioned detection process, it stops controlling at least one of the exposure and white balance based on the pixel data obtained by the aforementioned imaging element at the aforementioned future time. (27)

[0291] A camera program for a camera device, the camera device including an imaging element that captures images of a subject through a camera optical system and a processor, the camera program being used to cause the processor to perform the following processing:

[0292] Detecting moving objects present in multiple pixel data acquired by the aforementioned imaging elements at different times.

[0293] Based on the comparison between the first distance and the second distance, the detection process for the masking of the moving object by the object at the future time is performed. The first distance is the predicted distance of the moving object at the future time, and the second distance is the distance of the object existing in the predicted position coordinates of the moving object at the future time.

[0294] Camera control will be implemented based on the results of the above detection and processing.

[0295] The various embodiments have been described above with reference to the accompanying drawings, but the present invention is not limited to these examples. Those skilled in the art will obviously be able to conceive of various modifications or alterations within the scope of the patented technical solutions, and these should be understood to fall within the technical scope of the present invention as well. Furthermore, the constituent elements of the above embodiments can be arbitrarily combined without departing from the spirit of the invention.

[0296] Furthermore, this application is based on Japanese Patent Application No. 2021-005139, filed on January 15, 2021, the contents of which are incorporated herein by reference.

[0297] Industrial availability

[0298] This invention is particularly suitable for digital cameras and the like, offering high convenience and effectiveness.

[0299] Symbol Explanation

[0300] 1-Imaging lens, 4-Lens control unit, 5-Imaging element, 6-Analog signal processing unit, 7-Analog-to-digital conversion circuit, 8-Lens drive unit, 9-Aperture drive unit, 10-Imaging element drive unit, 11-System control unit, 14, 207-Operation unit, 15-Memory control unit, 16-Main memory, 17-Digital signal processing unit, 18-Contrast AF processing unit, 19-Phase AF processing unit, 20-External memory control unit, 21-Recording medium, 22-Display control unit, 23-Display unit, 25-Data bus, 40-Lens assembly, 50-Light-receiving surface, 51-Pixel, 53-AF area, 81-Moving object, 82-Object, 100 - Digital camera, 200- Smartphone, 201- Frame, 202- Display panel, 203- Operation panel, 204- Display input unit, 205- Speaker, 206- Microphone, 208- Camera unit, 210- Wireless communication unit, 211- Talk unit, 212- Storage unit, 213- External input / output unit, 214- GNSS receiver, 215- Motion sensor unit, 216- Power supply unit, 217- Internal storage unit, 218- External storage unit, 220- Main control unit, r1, r2, g1, g2, b1, b2, 52A, 52B- Pixels for phase difference detection, PD1~P4- Pixel data, ST1~STn- GNSS satellites.

Claims

1. A camera device comprising an imaging element that captures images of a subject via a camera optical system, and a processor. The processor performs the following processing: Detecting moving objects present in multiple pixel data acquired by the imaging element at different times. Based on the comparison between the predicted distance of the moving object at a future time (i.e., the first defocus data) and the distance of an object existing in the predicted position coordinates of the moving object at the future time (i.e., the second defocus data), a detection process is performed to detect the masking of the moving object by the object at the future time. Camera control is performed based on the results of the detection and processing. The first defocus data is calculated based on defocus data previously calculated for the moving object. The second defocus data is defocus data based on phase difference information of a portion of the predicted value of the position coordinates within the two-dimensional image of the moving object in the latest pixel data obtained by the imaging element.

2. The camera device according to claim 1, wherein, The processor calculates the predicted position coordinates of the moving object at a future time based on the position coordinates of the moving object in the plurality of pixel data.

3. The camera device according to claim 1 or 2, wherein, The processor calculates the first defocus data based on the detection results of the distance of the moving object at multiple times prior to the future time.

4. The camera device according to claim 1 or 2, wherein, The processor calculates the distance of the object at a time prior to the future time as the second defocus data.

5. The camera device according to claim 1 or 2, wherein, In the detection process, the processor detects the masking when the first defocus data is longer than the second defocus data.

6. The camera device according to claim 1 or 2, wherein, If the processor detects the masking through the detection process, it stops driving the focusing mechanism of the camera optical system based on the detection result of the distance of the moving object at the future time.

7. The camera device according to claim 6, wherein, If the processor detects the masking through the detection process, it calculates a predicted time after the future time to eliminate the masking, and at the predicted time, it restarts the driving of the focusing mechanism of the camera optical system based on the detection result of the distance of the moving object.

8. The camera device according to claim 1 or 2, wherein, When the processor detects the masking through the detection process, it drives the focusing mechanism of the camera optical system at a future time based on the detection results of the distance of the moving object at multiple times prior to the future time.

9. The camera device according to claim 1 or 2, wherein, If the processor detects the masking through the detection process, it stops the subject detection process based on the pixel data obtained by the imaging element at the future time.

10. The camera device according to claim 1 or 2, wherein, Even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails, the processor maintains the detection state of the subject detected by the subject detection processing based on pixel data obtained by the imaging element before the future time, even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails when the masking is detected by the detection processing.

11. The camera device according to claim 1 or 2, wherein, The processor performs the following processing: If the masking is detected, calculate the predicted time for removing the masking after the future time. Calculate the predicted position coordinates of the moving object at the predicted time. Subject detection processing is performed based on the data in the pixel data obtained by the imaging element at the predicted time that corresponds to the predicted position coordinates of the moving object at the predicted time.

12. The camera device according to claim 1 or 2, wherein, If the processor detects the masking through the detection process, it stops image recording based on pixel data obtained by the imaging element at the future time.

13. The camera device according to claim 1 or 2, wherein, If the processor detects the masking through the detection process, it stops controlling at least one of the exposure and white balance based on the pixel data obtained from the imaging element at the future time.

14. A method for capturing images using a camera device, the camera device comprising an imaging element that captures images of a subject through a camera optical system and a processor. The processor performs the following processing: Detecting moving objects present in multiple pixel data acquired by the imaging element at different times. Based on the comparison between the predicted distance of the moving object at a future time (i.e., the first defocus data) and the distance of an object existing in the predicted position coordinates of the moving object at the future time (i.e., the second defocus data), a detection process is performed to detect the masking of the moving object by the object at the future time. Camera control is performed based on the results of the detection and processing. The first defocus data is calculated based on defocus data previously calculated for the moving object. The second defocus data is defocus data based on phase difference information of a portion of the predicted value of the position coordinates within the two-dimensional image of the moving object in the latest pixel data obtained by the imaging element.

15. The imaging method according to claim 14, wherein, The processor calculates the predicted position coordinates of the moving object at a future time based on the position coordinates of the moving object in the plurality of pixel data.

16. The imaging method according to claim 14 or 15, wherein, The processor calculates the first defocus data based on the detection results of the distance of the moving object at multiple times prior to the future time.

17. The imaging method according to claim 14 or 15, wherein, The processor calculates the distance of the object at a time prior to the future time as the second defocus data.

18. The imaging method according to claim 14 or 15, wherein, In the detection process, the processor detects the masking when the first defocus data is longer than the second defocus data.

19. The imaging method according to claim 14 or 15, wherein, If the processor detects the masking through the detection process, it stops driving the focusing mechanism of the camera optical system based on the detection result of the distance of the moving object at the future time.

20. The imaging method according to claim 19, wherein, If the processor detects the masking through the detection process, it calculates a predicted time after the future time to eliminate the masking, and at the predicted time, it restarts the driving of the focusing mechanism of the camera optical system based on the detection result of the distance of the moving object.

21. The imaging method according to claim 14 or 15, wherein, When the processor detects the masking through the detection process, it drives the focusing mechanism of the camera optical system at a future time based on the detection results of the distance of the moving object at multiple times prior to the future time.

22. The imaging method according to claim 14 or 15, wherein, If the processor detects the masking through the detection process, it stops the subject detection process based on the pixel data obtained by the imaging element at the future time.

23. The imaging method according to claim 14 or 15, wherein, Even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails, the processor maintains the detection state of the subject detected by the subject detection processing based on pixel data obtained by the imaging element before the future time, even if the subject detection processing based on pixel data obtained by the imaging element at the future time fails when the masking is detected by the detection processing.

24. The imaging method according to claim 14 or 15, wherein, The processor performs the following processing: If the masking is detected, calculate the predicted time for removing the masking after the future time. Calculate the predicted position coordinates of the moving object at the predicted time. Subject detection processing is performed based on the data in the pixel data obtained by the imaging element at the predicted time that corresponds to the predicted position coordinates of the moving object at the predicted time.

25. The imaging method according to claim 14 or 15, wherein, If the processor detects the masking through the detection process, it stops image recording based on pixel data obtained by the imaging element at the future time.

26. The imaging method according to claim 14 or 15, wherein, If the processor detects the masking through the detection process, it stops controlling at least one of the exposure and white balance based on the pixel data obtained from the imaging element at the future time.

27. A storage medium storing a camera program for a camera device, the camera device comprising an imaging element that captures an image of a subject via a camera optical system, and a processor, the camera program being configured to cause the processor to perform the following processing: Detecting moving objects present in multiple pixel data acquired by the imaging element at different times. Based on the comparison between the predicted distance of the moving object at a future time (i.e., the first defocus data) and the distance of an object existing in the predicted position coordinates of the moving object at the future time (i.e., the second defocus data), a detection process is performed to detect the masking of the moving object by the object at the future time. Camera control is performed based on the results of the detection and processing. The first defocus data is calculated based on defocus data previously calculated for the moving object. The second defocus data is defocus data based on phase difference information of a portion of the predicted value of the position coordinates within the two-dimensional image of the moving object in the latest pixel data obtained by the imaging element.

Images