Focus control device and method, imaging device, program, and recording medium

Abstract

Provided are a focus control device and a method for detecting a subject from an infrared image and performing focus control in a visible light image of the detected subject, an imaging device, a program, and a recording medium. This focus control device is provided with: a focus control unit (37) that performs focus control when capturing an image; and processors (30, 120). The processor (30) acquires a visible light image from an image sensor (20) of a visible light camera (10) and the processor (120) acquires an infrared image from an image sensor (110) of an infrared camera (100). The processor (120) recognizes a subject to be photographed on the basis of the temperature information relating to the subject included in the infrared image, acquires first position information relating to a representative point of the subject in the infrared image, and converts the first position information to second position information relating to the representative point of the subject in the visible light image. The processor (30) determines, on the basis of the second position information, a focus region in which the focus control unit (37) performs focus control on the visible light image.

Description

Focus control device and method, imaging device, program and recording medium

[0001] The present invention relates to a focus control device and method, an imaging device, a program, and a recording medium, and more particularly to a technique for focusing on a desired subject.

[0002] When taking animal photographs with a camera, the essential steps are (1) to find the animal that will be the subject of the photograph, (2) to focus the camera on the subject, and (3) to take the photograph. However, the process leading up to (3) taking the photograph depends largely on the skill of the photographer, and there is a need for the development of auxiliary functions for cameras.

[0003] For example, in situations where it is difficult to find a wild animal with the naked eye, such as in the bush, or when an animal is outside the angle of view of the lens (outside the field of view), step (1) can be problematic.

[0004] In recent years, improvements in the processing speed of cameras have led to improvements in subject detection accuracy, leading to the expansion of animal detection functions to assist photographers. (2) Similarly, improvements in subject detection accuracy and tracking have been made to focus on subjects, allowing photographers to take the photos they intended.

[0005] However, if the photographer is still unable to recognize the animal subject, the expected effect cannot be achieved.

[0006] Animal detection, in conjunction with an infrared camera, utilizes the temperature distribution shown in infrared images, allowing for a simple method of locating animals that were previously unrecognizable by using the infrared camera as a third set of eyes in addition to the naked eye and visible light camera.

[0007] Patent Literature 1 describes a technology in which an infrared camera is used in addition to a visible light camera for autonomous driving or driving assistance of a vehicle, and a modal image (an image of a person area, which is a specific temperature area included in the infrared image) that is an infrared image captured by the infrared camera and converted from the infrared image is superimposed on the visible light image captured by the visible light camera, and a frame indicating the image of the person area is added, in order to improve the visibility of the modal image superimposed on the visible light image.

[0008] Japanese Patent Application Laid-Open No. 2003-144992 discloses a focus position interlocking means for adjusting the focus position of an infrared camera based on focus position information of a visible light camera.

[0009] International Publication No. 2019 / 111464 Japanese Patent Application Laid-Open No. 2022-113614

[0010] One embodiment of the technique of the present disclosure provides a focus control device and method, an imaging device, a program, and a recording medium that detect a subject from an infrared image and perform focus control in a visible light image of the detected subject.

[0011] The invention of a first aspect is a focus control device that includes a focus control unit that performs focus control when capturing an image, and a processor, wherein the processor acquires a visible light image obtained by capturing an image in a wavelength range that includes the visible light range, acquires an infrared image that has parallax with the visible light image and is obtained by capturing an image in a wavelength range that includes the infrared range, acquires first position information of a first subject based on temperature information of the subject included in the infrared image, converts the first position information into second position information of the first subject in the visible light image, and determines a focus area for performing focus control on the visible light image based on the second position information.

[0012] In the focus control device according to the second aspect of the present invention, in the first aspect, it is preferable that the processor acquires a visible light image from a first imaging unit and acquires an infrared image from a second imaging unit having parallax with respect to the first imaging unit.

[0013] In the focus control device of the third aspect of the present invention, in the second aspect, it is preferable that the processor calculates multiple pieces of second position information based on the first position information, the angle of view and image size of the first imaging unit, the angle of view and image size of the second imaging unit, and information regarding parallax, and determines multiple focus areas on the visible light image based on the multiple pieces of second position information.

[0014] A fourth aspect of the present invention provides a focus control device according to any one of the first to third aspects, wherein the focus control section performs focus control based on a focus region.

[0015] In the focus control device according to the fifth aspect of the present invention, in the third aspect, it is preferable that the distance to the first subject is virtually changed sequentially from a close distance to a long distance, or from a long distance to a close distance, and a plurality of pieces of second position information are calculated.

[0016] In the focus control device of the sixth aspect of the present invention, in the fifth aspect, it is preferable that the processor sets the focus area in the calculated order, and when the focus evaluation value by the focus control unit becomes equal to or greater than the first threshold, stops the next setting of the focus area and terminates focus control by the focus control unit.

[0017] In the focus control device of the seventh aspect of the present invention, in the third or fifth aspect, it is preferable that the processor executes focus control in each of multiple focus areas by the focus control unit, and applies focus control in the focus area among the multiple focus areas in which the focus evaluation value by the focus control unit is maximum.

[0018] In the focus control device of an eighth aspect of the present invention, in any of the second, third, and fifth to seventh aspects, the optical axis of the first imaging unit and the optical axis of the second imaging unit are parallel, and the processor sets the parallax information to zero when the distance of the first subject is equal to or greater than a second threshold, and calculates second position information based on the first position information, the angle of view and image size of the first imaging unit, and the angle of view and image size of the second imaging unit.

[0019] A focus control device according to a ninth aspect of the present invention is, in any of the first to eighth aspects, such that the first position information is coordinate information in the infrared image of a representative point of a first subject included in the infrared image, and the second position information is coordinate information in the visible light image of a representative point of the first subject included in the visible light image.

[0020] A focus control device according to a tenth aspect of the present invention is preferably a focus control device according to any one of the first to ninth aspects, further comprising a display for displaying a visible light image, and the processor preferably causes a first indicator indicating the determined focus area to be superimposed on the visible light image displayed on the display.

[0021] An eleventh aspect of the present invention provides the focus control device of the tenth aspect, wherein the first index is preferably a frame line surrounding the in-focus region.

[0022] In the focus control device of the 12th aspect of the present invention, in the 10th or 11th aspect, when the processor detects multiple subjects included in the infrared image based on temperature information obtained from the infrared image, it is preferable that the processor determines one of the multiple subjects as a first subject and determines the other subjects as second subjects, obtains third position information of the second subject in the infrared image, converts the third position information into fourth position information of the second subject in the visible light image, and superimposes a second indicator indicating the area of ​​the second subject on the visible light image displayed on the display based on the fourth position information.

[0023] In the thirteenth aspect of the present invention, the focus control device of the twelfth aspect is preferably provided with a user interface for setting priorities when multiple subjects are detected, and when multiple subjects are detected, the processor determines the first and second subjects in accordance with the set priorities.

[0024] In the focus control device of the 14th aspect of the present invention, in the 12th or 13th aspect, it is preferable that the first index is a frame line indicating the focus area, and the second index is a frame line indicating the area of ​​the second subject.

[0025] A fifteenth aspect of the present invention provides the focus control device of the fourteenth aspect, wherein the frame line of the first index and the frame line of the second index preferably have different line types or colors.

[0026] In the focus control device of the 16th aspect of the present invention, in any of the 1st to 15th aspects, it is preferable that the processor notify the user that the focus area does not exist if the determined focus area does not exist in the visible light image.

[0027] In the focus control device of the 17th aspect of the present invention, in any of the 10th to 15th aspects, if the determined focus area does not exist in the visible light image, it is preferable that the processor superimposes a third indicator indicating the direction in which the focus area exists on the visible light image displayed on the display, or notifies the user of the direction in which the focus area exists by voice.

[0028] An eighteenth aspect of the present invention provides the focus control device of the seventeenth aspect, wherein the third indicator includes a symbol or character indicating a direction in which the focus region exists.

[0029] A focus control device according to a 19th aspect of the present invention is preferably provided in any of the second, third, and fifth to eighth aspects with an exposure control unit that controls the brightness of the subject imaged by the first imaging unit, and the processor outputs area information indicating the focal area or the focal area and its surrounding area to the exposure control unit, and the exposure control unit controls the exposure based on the area information.

[0030] A twentieth aspect of the invention is an imaging apparatus including the focus control device according to any one of the first to nineteenth aspects.

[0031] The invention of a 21st aspect is a focus control method in a focus control device that includes a focus control unit that performs focus control when capturing an image and a processor, in which the processor executes the following steps: acquiring a visible light image obtained by capturing an image in a wavelength range that includes the visible light range; acquiring an infrared image that has parallax with the visible light image and is obtained by capturing an image in a wavelength range that includes the infrared range; acquiring first position information of a first subject based on temperature information of the subject included in the infrared image; converting the first position information into second position information of the first subject in the visible light image; and determining a focus region in which the focus control unit performs focus control based on the second position information.

[0032] A twenty-second aspect of the invention is a program for causing a computer to execute the focus control method of the twenty-first aspect.

[0033] A twenty-third aspect of the invention is a non-transitory computer-readable recording medium on which the program of the twenty-second aspect is recorded.

[0034] FIG. 1 is a side view showing the appearance of an imaging device according to the present invention. FIG. 2 is a schematic diagram showing the imaging ranges for each distance of an object imaged by a visible light camera and an infrared camera. FIG. 3 is a diagram showing the relationship between the imaging ranges shown in a visible light image and an infrared image at a certain object distance. FIG. 4 is a diagram showing the relationship between the principal point, optical axis, angle of view, and object position of the infrared camera. FIG. 5 is a diagram showing an example of the object position (coordinates) in an infrared image. FIG. 6 is a diagram showing the relationship between the principal point, optical axis, angle of view, and object position of the visible light camera. FIG. 7 is a diagram showing the object position (coordinates) in a visible light image. FIG. 8 is a diagram showing the object position (coordinates) at a distance z (=z ＩＲ 9 is a diagram showing the position (coordinates) of the subject on the plane of distance Z (=z Ｔ５ 1 is a diagram showing the position (coordinates) of a subject on a plane of the infrared camera and the visible light camera. FIG. 10 is a table showing a list of constants and parameters indicating the image size, angle of view, parallax, etc. of each of the infrared camera and the visible light camera. FIG. 11 is a diagram showing that a subject is present at a position on a line extending from the principal point of the infrared camera. FIG. 12 is a schematic diagram showing the position of a subject in a visible light image and an infrared image, which changes depending on the distance to the subject. FIG. 13 is a block diagram showing an outline of an imaging device according to the present invention. FIG. 14 is a block diagram showing an embodiment of an imaging device according to the present invention. FIG. 15 is a diagram showing a first embodiment of a viewfinder display screen. FIG. 16 is a diagram showing a second embodiment of a viewfinder display screen. FIG. 17 is a diagram showing a third embodiment of a viewfinder display screen. FIG. 18 is a diagram showing an example of a setting screen for subject detection AF settings. FIG. 19 is a diagram showing another example of a setting screen for subject detection AF settings. FIG. 20 is a flowchart showing an embodiment of a focus control method according to the present invention.

[0035] Hereinafter, preferred embodiments of a focus control device and method, an imaging device, a program, and a recording medium according to the present invention will be described with reference to the accompanying drawings.

[0036] [Outline of the Present Invention] The outline of the present invention will be described with reference to FIGS. 1 to 12. FIG.

[0037] FIG. 1 is a side view showing the appearance of an imaging device according to the present invention, in which an infrared camera is set in a visible light camera.

[0038] The imaging device shown in FIG. 1 is composed of a visible light camera 10 as a first imaging section and an infrared camera 100 as a second imaging section.

[0039] The visible light camera 10 captures an image of the first subject to be photographed, subject 1', in a wavelength range including the visible light range, and obtains a visible light image including subject 1', and the infrared camera 100 captures an image of the subject 1' in a wavelength range including the infrared range, and obtains an infrared image including subject 1'.

[0040] In this example, the subject 1' is an animal (warm-blooded animal) present in the shooting space. Therefore, position information (first position information) of the subject 1', which is a warm-blooded animal, within the infrared image can be obtained based on the temperature information of the subject included in the infrared image.

[0041] The infrared camera 100 is mounted on the top surface of the visible light camera 10 , and the infrared image acquired by the infrared camera 100 has parallax with the visible light image acquired by the visible light camera 10 .

[0042] Furthermore, the angle of view of the infrared camera 100 in this example is larger than the angle of view of the visible light camera 10 , and the infrared camera 100 can also capture images of subjects outside the angle of view (outside the field of view) of the visible light camera 10 .

[0043] FIG. 2 is a schematic diagram showing the imaging ranges of a subject imaged by a visible light camera and an infrared camera at different distances.

[0044] In FIG. 2, O′ indicates the principal point of the photographing lens of the visible light camera 10, and O indicates the principal point of the photographing lens of the infrared camera 100. Ｔ５ indicates the optical axis of the photographing lens of the visible light camera 10, and A ＩＲ indicates the optical axis of the imaging lens of the infrared camera 100.

[0045] As shown in Figure 2, the position of the principal point O' of the photographing lens of the visible light camera 10 is different from the position of the principal point O of the photographing lens of the infrared camera 100. In other words, there is a parallax between the visible light camera 10 and the infrared camera 100. In addition, the optical axis A of the visible light camera 10 Ｔ５ and the optical axis A of the infrared camera 100 ＩＲ is parallel to

[0046] Therefore, the relationship between the imaging range captured by the visible light camera 10 and the imaging range captured by the infrared camera 100 changes depending on the distance z of the subject.

[0047] FIG. 3 is a diagram showing the relationship between the imaging ranges shown in a visible light image and an infrared image at a certain distance from a subject.

[0048] As shown in FIG. 3, the visible light image I Ｔ５ and the infrared image I ＩＲ This does not match the second position information of the subject 1 in the image.

[0049] Therefore, the infrared image I ＩＲ The first position information (coordinate information) of the subject 1 in the visible light image I Ｔ５ , and the second position information (coordinate information) of the subject 1 in the visible light image I Ｔ５ Visible light image I applied when performing focus control Ｔ５ The focal area can be determined, and an indicator indicating the focal area (a first indicator such as a frame surrounding the subject 1) can be displayed on the display.

[0050] Next, infrared image I ＩＲ The first position information (coordinates (x, y)) of the subject 1 in the visible light image I Ｔ５ The function f for converting the second position information (coordinates (X, Y)) of the subject 1 in the above will be described.

[0051] FIG. 4 is a diagram showing the relationship between the principal point, optical axis, angle of view, and subject position of the infrared camera.

[0052] In FIG. 4, z indicates the distance of the object 1', and r' indicates the optical axis A of the object 1'. ＩＲ denotes the distance from ＩＲ is the optical axis A of the infrared camera 100ＩＲ and the angle formed by the line segment connecting the principal point O and the subject 1'.

[0053] FIG. 5 is a diagram showing an example of the position (coordinates) of a subject in an infrared image.

[0054] In FIG. 5, infrared image I ＩＲ The image size of the infrared image I is h pixels in height and w pixels in width. ＩＲ The length of half the diagonal (length converted into image size) is 1 mm.

[0055] Infrared Image I ＩＲ The coordinates of the object 1 in the image are expressed in terms of the number of pixels of the image sensor of the infrared camera 100 as coordinates (x ｐ ,y ｐ ), which can be converted to length (mm) as coordinates (x, y). If the object 1 is expressed in polar coordinates (r, θ), the distance from the center of the infrared image of the object 1 is r, and the angle is θ.

[0056] Similarly, FIG. 6 is a diagram showing the relationship between the principal point, optical axis, angle of view, and subject position of a visible light camera.

[0057] In FIG. 6, Z indicates the distance to the subject 1', and R' indicates the optical axis A of the subject 1'. Ｔ５ denotes the distance from Ｔ５ is the optical axis A of the visible light camera 10 Ｔ５ and the angle formed by the line segment connecting the principal point O' and the subject 1'.

[0058] FIG. 7 is a diagram showing the position (coordinates) of a subject in a visible light image.

[0059] In FIG. 7, the visible light image I Ｔ５ The image size of the visible light image I is H pixels in height and W pixels in width. Ｔ５ The length of half the diagonal of the image (length converted into image size) is L mm.

[0060] Visible light image I Ｔ５ The coordinates of the subject 1 in the image are expressed in terms of the number of pixels of the image sensor of the visible light camera 10 as coordinates (X Ｐ ,Y Ｐ), which can be converted to length (mm) as coordinates (X, Y). Furthermore, when object 1 is expressed in polar coordinates (R, Θ), the distance of object 1 from the center of the visible light image is R, and the angle is Θ.

[0061] FIG. 8 shows the distance z (=z ＩＲ 6 is a diagram showing the position (coordinates) of the subject on the plane of FIG. 5, and shows the position in real space corresponding to the infrared image shown in FIG. 5.

[0062] In FIG. 8, 1′ (mm) is the infrared image I ＩＲ The coordinates (x', y') represent the length of the real space corresponding to the half length l (mm) of the diagonal of the infrared image I ＩＲ The coordinates of the object 1 in the infrared image I are shown in real space corresponding to the coordinates (x, y) of the object 1 in the infrared image I. Also, when the object 1' in the real space is expressed by polar coordinates (r', θ), ＩＲ The distance from the center of the infrared image of the subject 1 is r' and the angle is θ.

[0063] FIG. 9 shows the distance Z (= z Ｔ５ 8 is a diagram showing the position (coordinates) of the subject on the plane of FIG. 7, and shows the position in real space corresponding to the visible light image shown in FIG. 7.

[0064] In FIG. 9, L′ (mm) is the visible light image I Ｔ５ The coordinates (X', Y') represent the length of the real space corresponding to half the diagonal length L (mm) of the visible light image I Ｔ５ The coordinates of the object 1 in the visible light image I are shown in real space corresponding to the coordinates (X, Y) of the object 1 in the visible light image I. Also, when the object 1' in the real space is expressed by polar coordinates (R', Θ), Ｔ５ The distance from the center of the visible light image of the subject 1 is R' and the angle is Θ.

[0065] FIG. 10 is a table showing a list of constants and parameters indicating the image size, angle of view, parallax, etc. of each of the infrared camera and the visible light camera.

[0066] As shown in FIG. 10, the parallax between the visible light camera 10 and the infrared camera 100 (the spatial positional deviation between the principal points (O, O') of the cameras shown in FIG. 2) is (Δx, Δy, Δz), and the focal length of the visible light camera 10 is f Ｔ５ , the focal length of the infrared camera 100 is f ＩＲ The angle of view of the visible light camera 10 and the infrared camera 100 is determined by the focal length f Ｔ５ , f ＩＲ and can be obtained based on the size (diagonal size) of each image sensor.

[0067] As described above, it is possible to obtain position information of the subject 1', which is a warm-blooded animal, in the infrared image based on the temperature information of the subject contained in the infrared image.

[0068] As shown in FIG. 5, the infrared image I ＩＲ The coordinates of the representative point of the object 1 in ｐ ,y ｐ The representative point of the object 1 can be the center of gravity of the temperature distribution area corresponding to the object 1, or the point with the highest temperature.

[0069] The coordinates (x ｐ ,y ｐ ) is converted into the visible light image I of the visible light camera 10 shown in FIG. 7 by the following conversion formula f. Ｔ５ The coordinates of the representative point of subject 1 in (X Ｐ ,Y Ｐ ) can be converted to

[0070]

[0071] The formula [1] can be derived from the formulas [6] and [7] using the formulas [2] to [5] below.

[0072] Here, a and b are the horizontal and vertical lengths (mm) of the image sensor of the infrared camera 100, and A and B are the horizontal and vertical lengths (mm) of the image sensor of the visible light camera 10.

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] Furthermore, when polar coordinates are used instead of XY coordinates, the conversion formula f is a conversion to obtain polar coordinates (R, Θ) from polar coordinates (r, θ).

[0079] Infrared Image I ＩＲ The polar coordinates (r, θ) of the representative point of the object 1 in the visible light image I are calculated based on the following equation: Ｔ５ can be converted into polar coordinates (R, Θ) of the representative point of the subject 1 in

[0080]

[0081] R and Θ in the formula [8] can be obtained using r, r', R', and θ obtained from the following formulas [9] to

[12] .

[0082]

[0083]

[0084]

[0085]

[0086] If the infrared camera 100 has a distance measurement function or an autofocus function, the distance z (unit: m) of the subject 1' can be obtained. However, if the infrared camera 100 does not have a distance measurement function, the distance z (unit: m) of the subject 1' can be obtained by the angle φ ＩＲ The object 1' is located somewhere on the line in the direction z, and the distance z to the object 1' is unknown.

[0087] When the distance z of the object 1' is unknown, multiple virtual distances are given as the distance of the object 1', so that the infrared image I ＩＲ The coordinates of the representative point of the object 1 in ｐ ,y ｐ ) and generate visible light images I Ｔ５The coordinates of the representative point of subject 1 in (X Ｐ ,Y Ｐ ) can be calculated (converted).

[0088] FIG. 12 is a schematic diagram showing the position of a subject in a visible light image and an infrared image, which changes depending on the distance to the subject.

[0089] In FIG. 12, an infrared image I for object recognition is ＩＲ The position of the subject 1 in the visible light image I for recognizing the focal region is Ｔ５ The position of the subject 1 at the distance z (z １ , z ２ , z ３ , …, z ｎ ) depending on the

[0090] Infrared Image I ＩＲ Subject 1 and visible light image I in Ｔ５ The closer the distance z between the subject 1 in the real space and the subject 1' in the visible light image I Ｔ５ The angle Φ to the visible light image I is also different. Ｔ５ Infrared image I for the center o' of ＩＲ This is because the center o of varies depending on the distance z of the subject 1'.

[0091] In this way, the visible light image I Ｔ５ Position of object 1 in (coordinates of the representative point of object 1 (X Ｐ ,Y Ｐ ) changes, the visible light image I Ｔ５ The focal area (i.e., the coordinates of the representative point (X Ｐ ,Y Ｐ ) based on the coordinates of the representative point (X Ｐ ,Y Ｐ ) along the locus, and focus control must be performed so that the image within the determined focal region is in focus.

[0092] 12, when the distance z of the subject 1 is equal to or greater than the threshold value (second threshold value) (in the example shown in FIG. 12, the distance z ｎ In the above cases), visible light image I Ｔ５The center o' of the infrared image I ＩＲ This is because the parallax (Δx, Δy, Δz) between the visible light camera 10 and the infrared camera 100 becomes relatively small for subject distances equal to or greater than the second threshold. In this case, the parallax can be ignored.

[0093] By ignoring the parallax (Δx, Δy, Δz), that is, by setting Δx=Δy=Δz=0, the above-mentioned formula (1) can be expressed as the following formula.

[0094]

[0095] Since there is no parameter for the distance z in [Equation 13], the infrared image I ＩＲ The coordinates of the representative point of the object 1 in ｐ ,y ｐ ) into the visible light image I Ｔ５ The coordinates of the representative point of subject 1 in (X Ｐ ,Y Ｐ ) and the visible light image I Ｔ５ The focal region for performing focus control on the object can be uniquely determined.

[0096] [Imaging Apparatus] FIG. 13 is a block diagram showing an outline of an imaging apparatus according to the present invention.

[0097] As shown in FIG. 13, an infrared camera 100 is attached to a visible light camera 10 via an attachment 12 to form an imaging device.

[0098] 1, the infrared camera 100 is mounted on the top surface of the visible light camera 10, and in this case, the hot shoe provided on the top surface of the visible light camera 10 can be used as an attachment 12 when mounting the infrared camera 100. Furthermore, the visible light camera 10 and the infrared camera 100 can communicate with each other via the attachment 12.

[0099] FIG. 14 is a block diagram showing an embodiment of an imaging device according to the present invention.

[0100] <Infrared Camera> In FIG. 14, an infrared camera 100 includes an image sensor 110, a processor 120, a memory 130, and the like.

[0101] The image sensor 110 is configured with a CMOS (Complementary Metal-Oxide Semiconductor) type infrared imaging element. The image sensor 110 is not limited to a CMOS type, and may be a CCD (Charge Coupled Device) type infrared imaging element. The image sensor 110 is composed of photoelectric conversion elements (photodiodes) arranged two-dimensionally in the x direction (horizontal direction) and y direction (vertical direction), and includes photoelectric conversion elements that are sensitive to a wavelength range that includes the infrared range. It is preferable that a visible light cut filter be provided on the light receiving surface of the image sensor 110.

[0102] The processor 120 is composed of a CPU (Central Processing Unit) and other components, and controls each part of the infrared camera 100, performs processing to capture infrared images, and also functions as a processing unit 122 that recognizes objects and a coordinate conversion unit 124.

[0103] The memory 130 stores programs for causing the processor 120 to execute various processes, sensor information of the image sensor 110 (image size including the number of vertical and horizontal pixels and pixel pitch), and information such as the angle of view.

[0104] A fixed-focus wide-angle lens is preferably used as the imaging lens (not shown) of the infrared camera 100. The aperture of the infrared camera 100 is preferably set so that deep focus imaging is possible.

[0105] An infrared image of the subject formed on the light receiving surface of the image sensor 110 through the photographing lens is converted into an electrical signal (infrared image signal) by the image sensor 110 and sent to the processor 120. It is preferable that the infrared camera 100 captures infrared images at a constant frame rate (e.g., 30 frames per second), and the image sensor 110 outputs an infrared image signal representing an infrared video.

[0106] The processing unit 122 of the processor 120 recognizes the object 1 (first object) based on the temperature information of the object contained in the infrared image signal input from the image sensor 110, and calculates the coordinates (x ｐ ,y ｐ ) (first position information). It is also preferable that the processing unit 122 acquires the size (number of vertical and horizontal pixels) of the subject 1 on the image sensor 110.

[0107] The processing unit 122 extracts a predetermined temperature region corresponding to the body temperature of the animal from the temperature information of the subject included in the infrared image, and recognizes the extracted predetermined temperature region as the presence region of the subject 1. The processing unit 122 also detects the number of vertical and horizontal pixels of a rectangular region surrounding the presence region of the subject 1 as the size of the subject 1 on the image sensor 110, and can use the center of gravity of the presence region of the subject 1 or the center of the rectangular region as the representative point of the subject 1.

[0108] The coordinate conversion unit 124 of the processor 120 converts the coordinates (x ｐ ,y ｐ ) on the image sensor 20 of the visible light camera 10. Ｐ ,Y Ｐ ) to

[0109] The coordinate conversion unit 124 receives the coordinates (x ｐ ,y ｐ ), the information also includes the angle of view and focal length of the infrared camera 100, sensor information of the image sensor 110 of the infrared camera 100 (image size including the number of vertical and horizontal pixels and pixel pitch), the angle of view and focal length of the visible light camera 10, sensor information of the image sensor 20 of the visible light camera 10 (image size including the number of vertical and horizontal pixels and pixel pitch), and information about the parallax between the infrared camera 100 and the visible light camera 10 (parallax (Δx, Δy, Δz) in this example). Based on this information, the coordinate conversion unit 124 converts the coordinates (x, Δy, Δz) on the image sensor 110 into the coordinates (x, Δy, Δz) on the visible light camera 10 using the conversion formula f shown in the above-mentioned [Formula 1]. ｐ ,y ｐ ) on the image sensor 20 (X Ｐ ,Y Ｐ ) to

[0110] Furthermore, when the distance z of the object 1' is unknown, the processing unit 122 virtually sets multiple distances z of the object 1' and calculates the coordinates (x ｐ ,y ｐ ) on the image sensor 20 according to the virtually set distance z. Ｐ ,Y Ｐ ) to

[0111] For example, the device may be equipped with a determination mode in which the distance z is sequentially changed (scanned) from a close distance to a long distance to determine whether focus has been achieved, a determination mode in which the distance z is scanned from a long distance to a close distance to determine whether focus has been achieved, or a determination mode in which the range of the distance z to be focused is narrowed down in advance, and the distance z can be virtually set according to these determination modes.

[0112] For example, in the case of a focus determination mode in which the distance z is scanned from a short distance to a long distance, the scanning distance (distance z shown in FIG. 12) is １ , z ２ , z ３ , …, z ｎ It is preferable that the interval between the infrared image and the visible light image is narrower as the distance becomes shorter. This is because the positional deviation between the object on the infrared image and the object on the visible light image becomes larger as the distance becomes shorter. ｎ The number n is not limited to a fixed number and may be changed depending on the position of the subject on the infrared image, etc. For example, if the position of the subject on the infrared image is close to the center of the infrared image, the amount of movement of the subject on the visible light image due to the distance of the subject will be small.

[0113] However, the distance z ｉ The coordinates (X Ｐ ,Y Ｐ ) and a focus area centered at a distance z ｉ＋１ The coordinates (X Ｐ ,Y Ｐ It is preferable to set each scanning distance so that the focus area centered on the target object overlaps with the focus area centered on the target object by half or more. This is to ensure that the subject is included in one of the focus areas.

[0114] The coordinates (X Ｐ ,Y Ｐ ) and the coordinates (X Ｐ ,Y Ｐ ) is output to the visible light camera 10.

[0115] 14 , the visible light camera 10 includes an image sensor 20, a processor 30, a focus control unit 37, an exposure control unit 38, a memory 39, a photographing lens 40, a viewfinder 50, and an imaging unit 60. The visible light camera 10 may include another display (such as a liquid crystal monitor provided on the back of the visible light camera 10) instead of or in addition to the viewfinder 50. The liquid crystal monitor can display live view images and the like, just like the viewfinder 50, and can also be used as part of a user interface that accepts inputs from the photographer by displaying various setting screens.

[0116] The image sensor 20 is composed of a CMOS-type color imaging element. The image sensor 20 is not limited to a CMOS-type, and may be a CCD-type color imaging element. The image sensor 20 is composed of photoelectric conversion elements arranged two-dimensionally in the x and y directions, and red (R), green (G), and blue (B) color filters are arranged in a periodic color array (e.g., Bayer array, X-Trans (registered trademark), etc.) on a plurality of pixels composed of photoelectric conversion elements sensitive to a wavelength range including the visible light range. In addition, it is preferable that an infrared cut filter be provided on the light receiving surface of the image sensor 20.

[0117] Furthermore, in the image sensor 20 of this example, apart from the R pixels, G pixels, and B pixels in which R, G, and B color filters are arranged, phase difference pixel groups each consisting of a pair of a first phase difference pixel and a second phase difference pixel having different pupil division directions are discretely arranged on the entire surface of the image sensor 20. The phase difference pixel groups are used when phase difference AF (Autofocus) is performed by the focus control unit 37.

[0118] The photographing lens 40 may be an integral part of the camera body of the visible light camera 10, or may be an interchangeable lens that can be attached to or detached from the camera body. The photographing lens 40 in this example is a zoom lens whose focal length can be changed, but it may also be a fixed focal length lens.

[0119] The processor 30 is a main component of the focus control device according to the present invention and is composed of a CPU, etc. The processor 30 controls each part of the visible light camera 10, performs processing to capture visible light images, and also functions as a focal region determination part 32, a composition part 34, and an exposure determination part 36. Details of these functions will be described later.

[0120] The memory 39 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), etc. The non-volatile memory such as flash memory stores firmware that operates the visible light camera 10, various programs including a program that causes the processor 30 to execute the focus control method according to the present invention, as well as sensor information of the image sensor 20 (image size including the number of vertical and horizontal pixels and pixel pitch), lens information (information indicating the focal length, angle of view, etc. of the photographing lens 40), and the captured visible light image, etc.

[0121] The RAM functions as a work area for processing by the processor 30. It also temporarily stores various programs stored in flash memory or the like, data used in arithmetic processing, etc. Note that the processor 30 may have a part of the memory 39 (RAM) built in.

[0122] The viewfinder 50 functions as a display device that displays a live view image, which is a visible light image, and also superimposes an index (first index) indicating the in-focus area on the live view image. While viewing the live view image displayed on the viewfinder 50, the photographer can determine the composition, adjust the angle of view (photography magnification), and so on.

[0123] The visible light image of the subject formed on the light receiving surface of the image sensor 20 via the photographing lens 40 is converted by the image sensor 20 into an electrical signal (visible light image signal) and sent to the processor 30. The visible light camera 10 can capture still images or videos depending on the selected shooting mode, but the following description will focus on the case where still images are captured.

[0124] In the preparation stage for shooting in still image shooting mode, the visible light camera 10 captures visible light images at a constant frame rate (e.g., 30 frames / second), and the image sensor 20 outputs a visible light image signal representing a visible light moving image to the processor 30.

[0125] The focal area determination unit 32 of the processor 30 determines the coordinates (X Ｐ ,Y Ｐ ), and the distance z (z １ , z ２ , z ３ , …, z ｎ ) corresponding to multiple coordinates (X Ｐ ,Y Ｐ ) is input from the coordinate conversion unit 124.

[0126] In the case of a determination mode in which focus determination is performed by sequentially changing the virtual distance z from a short distance to a long distance, first, the distance of the subject 1' in real space is set to the distance z １ The coordinates of the representative point of the object 1 on the image sensor 20 when Ｐ ,Y Ｐ ) and take this coordinate (X Ｐ ,Y Ｐ ) is determined as a focal area for performing focus control on the visible light image.

[0127] As described above, the processing unit 122 can detect the number of vertical and horizontal pixels in a rectangular area surrounding the area where the subject 1 exists as the size of the subject 1 on the image sensor 110, and therefore the size on the image sensor 20 of the visible light camera 10 that corresponds to the size of the subject 1 on the image sensor 110 of the infrared camera 100 can be set as the "predetermined area." Note that the "predetermined area" on the image sensor 20 changes depending on the ratio between the focal length of the infrared camera 100 and the focal length of the visible light camera 10. Also, if the distance of the subject 1' is actually the distance z １ In this case, the subject 1 on the image sensor 20 falls within the "predetermined area."

[0128] When the focal region on the image sensor 20 is determined by the focal region determination unit 32, the focus control unit 37 calculates an integrated value of the absolute differences between the first image data acquired from the first phase difference pixel group and the second image data acquired from the second phase difference pixel group among the phase difference pixel groups within the determined focal region, and calculates the phase difference from the shift amount between the first image data and the second image data when the integrated value is minimum. The focus control unit 37 calculates the defocus amount of the focus lens of the photographing lens 40 based on the calculated phase difference and the current focal length of the photographing lens 40, and focuses the optical image of the subject incident on the focal region by moving the focus lens to a position where the defocus amount is zero.

[0129] The focus control unit 37 also calculates a focus evaluation value that serves as an index for determining whether or not the subject 1' is present in the determined focus region. The focus evaluation value can be the magnitude of the contrast of the image within the focus region (the integrated value of the absolute values ​​of the high-frequency components of the image within the focus region) of the visible light image acquired after focus control (phase difference AF) has been performed in the determined focus region.

[0130] If the focus evaluation value (magnitude of contrast) calculated by the focus area determination unit 32 is equal to or greater than a threshold value (first threshold value), the processor 30 can determine that the subject 1' is present in the determined focus area.

[0131] As another determination method, the focus control unit 37 first calculates the distance to the subject 1′ as a distance z １When the focus lens of the photographing lens 40 is positioned at a distance z １ The focus lens is controlled so that the subject is in focus.

[0132] Thereafter, in the same manner as described above, the focus control unit 37 calculates a phase difference from the shift amount between the first image data acquired from the first phase difference pixel group and the second image data acquired from the second phase difference pixel group when the integrated value of the absolute difference between the first image data acquired from the first phase difference pixel group and the second image data acquired from the second phase difference pixel group becomes the smallest. When the calculated phase difference is small, the distance z １ In this case, the processor 30 determines that the smaller the phase difference calculated by the focus control unit 37 is, the higher the focus evaluation value is. If the focus evaluation value is equal to or greater than the first threshold value, the processor 30 determines that the object (object 1') is present at the distance z １ It can be determined that the subject 1' exists in the area.

[0133] As yet another method of determination, the processor 30 may ＩＲ The shape of a predetermined temperature region showing the object 1 extracted from the visible light image I Ｔ５ The method calculates the similarity between the shape of the subject in the image and the shape of the subject in the determined focal region, and determines whether or not the subject 1' is present in the determined focal region based on the similarity. Note that the method for determining whether or not the subject 1' is present in the determined focal region is not limited to the above method, and various methods are possible.

[0134] Now, let the distance to the subject 1' be distance z １ When this is done, the distance z １ If it is determined that the subject 1 does not exist within the focal region determined in accordance with ２ The coordinates of the representative point of the object 1 on the image sensor 20 when Ｐ ,Y Ｐ ) and take this coordinate (X Ｐ ,Y Ｐ ) to determine the focal region.

[0135] The focus control unit 37 performs phase difference AF based on the data of the phase difference pixel group within the determined focus area, and the processor 30 determines whether or not the subject 1 is present within the determined focus area based on the focus evaluation value calculated by the focus control unit 37.

[0136] In this way, the distance z of the subject 1' is virtually set to the short distance z １ distance z from ２ , z ３ , . . . , and performs phase difference AF based on the data of the phase difference pixel group in the determined in-focus area, and determines whether or not the subject 1 is present in the determined in-focus area.

[0137] When the processor 30 determines the focal area by sequentially changing the distance z from a close distance to a long distance and determines whether or not the subject 1 is present in the determined focal area, if the focus evaluation value by the focus control unit 37 becomes equal to or greater than the first threshold value for the first time, the processor 30 stops the next setting of the focal area and terminates focus control by the focus control unit 37. Conversely, when the processor 30 determines the focal area by sequentially changing the distance z from a long distance to a close distance and determines whether or not the subject 1 is present in the determined focal area, the processor 30 may stop the next setting of the focal area and terminate focus control by the focus control unit 37 if the focus evaluation value by the focus control unit 37 becomes equal to or greater than the first threshold value for the first time.

[0138] In addition, the processor 30 may cause the focus control unit 37 to perform focus control for each of the multiple focus areas determined according to the multiple distances z that are virtually changed, and apply focus control to the focus area among the multiple focus areas in which the focus evaluation value by the focus control unit 37 is maximized (the focus area in which it can be determined that the subject 1 is present).

[0139] In this example, the focus control unit 37 performs phase difference AF, but is not limited to this. For example, it may also perform so-called contrast AF, in which the focus lens is moved so as to maximize the contrast of the image within the determined focus area.

[0140] The processor 30 receives visible light image signals from the image sensor 20 at a constant frame rate, and generates a live view image from the received visible light image signals.

[0141] The synthesis unit 34 of the processor 30 superimposes the subject information on the live view image, and outputs the live view image with the superimposed subject information to the viewfinder 50. The photographer can operate the camera to determine the composition based on the live view image displayed on the viewfinder 50, and can also obtain information necessary for operating the camera by checking the subject information.

[0142] The processor 30 also outputs area information indicating the focal area determined by the focal area determination section 32 or the focal area and its surrounding area to the exposure determination section 36 .

[0143] The exposure determination unit 36 ​​is a part that determines the exposure value (EV) of the visible light camera 10, which is the first imaging unit, and determines the aperture value (F-number), shutter speed, ISO sensitivity, etc. based on the determined EV value and a program diagram, etc.

[0144] The exposure determination unit 36 ​​calculates the visible light image I based on the focal area determined by the focal area determination unit 32 or the area information indicating the focal area and its surrounding area. Ｔ５ A visible light image signal corresponding to the focal region etc. in the image is acquired, and the current brightness (EV value) of the focal region etc. is calculated from the visible light image signal. The F-number, shutter speed, and ISO sensitivity are determined to set this EV value to a predetermined appropriate exposure value.

[0145] The exposure control unit 38 controls the aperture of the photographing lens 40, the shutter speed, and the gain (sensitivity) of the visible light image signal based on the F-number, shutter speed, and ISO sensitivity determined by the exposure determination unit 36. This allows the photographing unit 60 to photograph the subject within the focal area with appropriate exposure.

[0146] [Viewfinder Display Example] <First Embodiment of Viewfinder Display Screen> FIG. 15 is a diagram showing a first embodiment of a viewfinder display screen.

[0147] As shown in FIGS. 15A and 15B, on the display screen 52 of the viewfinder 50, frame lines 54A and 54B surrounding the determined in-focus area are displayed as indicators (first indicators) superimposed on the live view image.

[0148] The frame line 54A shown in FIG. 15A is a frame line drawn in dotted lines, whereas the frame line 54A shown in FIG. 15B is a frame line drawn in solid lines, which is a difference.

[0149] The composition unit 34 superimposes a frame line 54A or 54B on the live view image, and outputs the live view image with the superimposed frame line 54A or 54B to the viewfinder 50. It is also preferable that the composition unit 34 generates the frame line 54A or 54B based on the in-focus area used when focus control by the focus control unit 37 ended, and composites it with the live view image. In this way, an image showing the subject will be placed within the frame line 54A or 54B displayed on the viewfinder 50, and will serve as an indicator of the position of the subject.

[0150] <Second Embodiment of Viewfinder Display Screen> FIG. 16 is a diagram showing a second embodiment of the viewfinder display screen.

[0151] As shown in FIG. 16A, a triangular symbol 55A indicating the direction in which the focal region exists is displayed as an index (third index) on the display screen 52 of the viewfinder 50, superimposed on the live view image.

[0152] Infrared Image I ＩＲ The coordinates of the representative point of the object 1 in ｐ ,y ｐ ) are converted into visible light images I Ｔ５ The coordinates of the representative point of subject 1 in (X Ｐ ,Y Ｐ ), the coordinates (X Ｐ ,Y Ｐ ) is the visible light image I Ｔ５ This is because there is parallax between the infrared camera 100 and the visible light camera 10, and the angles of view of the two cameras are different, and this is particularly noticeable when the focal length of the visible light camera 10 is long.

[0153] That is, |XＰ |>W / 2 and |Y Ｐ If at least one of |>H / 2 is satisfied, the coordinate (X Ｐ ,Y Ｐ ) is the visible light image I Ｔ５ As mentioned above, the visible light image I Ｔ５ The vertical and horizontal image size is H x W (pixels).

[0154] The processor 30 calculates the transformed coordinates (X Ｐ ,Y Ｐ ) is the visible light image I Ｔ５ If it is determined that the coordinates are outside the range of Ｐ ,Y Ｐ ) is captured in the visible light image I Ｔ５ This is to prompt the photographer to change the shooting direction or angle of view of the visible light camera 10 so that the subject is within the angle of view of the visible light camera 10.

[0155] Therefore, as shown in Figure 16 (A), the processor 30 superimposes a symbol 55A indicating the direction in which the focal area exists on the live view image using the synthesis unit 34, and displays the live view image with the symbol 55A superimposed on it on the viewfinder 50.

[0156] The photographer can recognize that the focal area (subject) is in the direction indicated by the symbol 55A superimposed on the live view image, and can operate the camera so that the subject is within the angle of view of the visible light camera 10.

[0157] 16B shows another example of a third indicator indicating the direction in which the focal region exists. As shown in FIG. 16B, the direction in which the focal region exists is indicated by the character "Left!" 55B.

[0158] As shown in FIG. 16, the direction in which the focal area (subject) is located is indicated by symbols 55A and letters 55B superimposed on the live view image, but instead of this, or simultaneously, the indication may be made by sound.

[0159] <Third Embodiment of Viewfinder Display Screen> FIG. 17 is a diagram showing a third embodiment of the viewfinder display screen.

[0160] FIG. 17 shows the display screen 52 of the viewfinder 50 when a plurality of (four) subjects are extracted by the infrared camera 100.

[0161] In the example shown in FIG. 17A, a visible light image I is displayed on the display screen 52 of the viewfinder 50. Ｔ５ and a frame line 56A corresponding to three subjects in the visible light image I Ｔ５ A symbol 56B indicating the direction in which the subject is located outside the range of the subject 10 is superimposed on the live view image. The frame lines 56A corresponding to the three subjects are each drawn in solid lines.

[0162] The processor 30 receives the infrared image I ＩＲ Based on the temperature information obtained from the infrared image I ＩＲ When multiple subjects are detected, one of the multiple subjects is determined as subject 1 (first subject), and the other subject is determined as a second subject. ＩＲ The coordinates (x ｐ ,y ｐ ) (third position information), and the coordinates (x ｐ ,y ｐ ) into visible light image I Ｔ５ The coordinates of the representative point of the second object in (X Ｐ ,Y Ｐ ) (fourth position information).

[0163] The processor 30 calculates the coordinates (X Ｐ ,Y Ｐ ) and displays an index (second index) indicating the area of ​​the second subject superimposed on the live view image displayed on the viewfinder 50. In this case, it is preferable that the first index indicating the in-focus area of ​​the subject 1 and the second index indicating the area of ​​the second subject are different so that they can be distinguished by the photographer.

[0164] Furthermore, the processor 30 can perform the AF operation when multiple subjects are extracted as follows: First, the processor 30 receives the coordinates (X Ｐ ,Y Ｐ ) {(X １ ,Y １ ), (X ２ ,Y ２ ), (X ３ ,Y ３ The processor 30 acquires the coordinates (X Ｐ ,Y Ｐ ) {(X １ ,Y １ ), (X ２ ,Y ２ ), (X ３ ,Y ３ ), …} and sort the coordinates (X Ｐ ,Y Ｐ AF operation is performed based on the infrared image I. ＩＲ Infrared image I ＩＲ In the order of the image size of the object in the infrared image I ＩＲ The coordinates of the representative point of the subject (x ｐ ,y ｐ ) can be sorted in order of proximity to the center of the infrared image.

[0165] In another example shown in FIG. 17B, a visible light image I is displayed on the display screen 52 of the viewfinder 50. Ｔ５ and the visible light image I Ｔ５ A symbol 57C indicating the direction in which the subject is outside the range is superimposed on the live view image.

[0166] 17B, ​​the frame lines 57A and 57B are of different line types, with the frame line 57A surrounding the in-focus area corresponding to the subject with the highest priority being drawn in a solid line, and the frame line 57B surrounding the in-focus area corresponding to the other subject (second subject) being drawn in a dashed line. Furthermore, it is preferable that the live view image be one in which focus has been controlled based on the in-focus area corresponding to the subject with the highest priority.

[0167] In yet another example shown in FIG. 17C, a visible light image I is displayed on the display screen 52 of the viewfinder 50 in the same manner as in FIG. 17B. Ｔ５ and the visible light image I Ｔ５ A symbol 58C indicating the direction in which the subject is outside the range is superimposed on the live view image.

[0168] 17(C) are different in color or brightness, and it is preferable that the frame line 58A surrounding the in-focus area corresponding to the subject with the highest priority is drawn in a conspicuous color or brightness, and the frame line 58B surrounding the in-focus area corresponding to the other subject (second subject) is drawn in an inconspicuous color or brightness. Ｔ５ The symbol 58C indicating the direction in which the subject is located outside the range 100 also has a different color or brightness from the symbol 57C shown in FIG. 17(B).

[0169] When multiple subjects are within the field of view as shown in Figures 17(B) and (C), the photographer may wish to make a subject with a lower priority (second subject) within frame lines 57B and 58B, other than frame lines 57A and 58A indicating the focus range corresponding to the subject with the highest priority, the main subject.

[0170] In this case, it is preferable that the processor 30 accepts the subject position of the subject desired by the photographer through a user operation via a user interface, and performs focus control based on a focus area including the accepted subject position.

[0171] FIG. 18 is a diagram showing an example of a setting screen for subject detection AF setting.

[0172] The setting screen shown in FIG. 18 can be displayed on the liquid crystal monitor of the visible light camera 10 by the photographer operating the menu / enter button (not shown) of the visible light camera 10.

[0173] 18, "Subject detection" is further selected, and "Human / animal (M: 35-38 degrees)" is selected as the subject. These selections can be made by the photographer operating the select button while looking at the setting screen.

[0174] The processor 120 of the infrared camera 100 receives temperature information of the subject set on the setting screen of the visible light camera 10 through processor communication with the processor 30 of the visible light camera 10, and based on the received temperature information of the subject, extracts an area of ​​the temperature distribution shown in the infrared image that corresponds to the temperature information of the set subject, thereby extracting the subject desired by the user.

[0175] In addition, on the setting screen shown in FIG. 18, in addition to "Humans and Animals (M: 35-38 degrees)," it is possible to select "Birds (H: 39 degrees or higher)" and "User Specified." If "User Specified" is selected, the photographer can further set the temperature range of the subject.

[0176] FIG. 19 is a diagram showing another example of the setting screen for the subject detection AF setting.

[0177] The setting screen shown in FIG. 19 can be displayed on the liquid crystal monitor of the visible light camera 10 by the photographer operating the menu / enter button of the visible light camera 10.

[0178] On the setting screen shown in FIG. 19, the user can set one of "differentiate AF notification methods," "do not differentiate AF notification methods," and "AF notification off."

[0179] When "Differentiate AF notification method" is set, when multiple subjects are extracted, processor 30 superimposes a first indicator indicating the area of ​​the first subject with the highest priority and a second indicator indicating the areas of other second subjects on the live view image so as to differentiate the notification method according to the priority of the multiple subjects. For example, in the example shown in Figures 17(B) and (C), the display form of frame lines 57A and 58A surrounding the area of ​​the subject with the highest priority is differentiated from the display form of frame lines 57B and 58B surrounding the areas of the other subjects.

[0180] If "No difference in AF notification method" is set, when multiple subjects are extracted, processor 30 superimposes an indicator indicating the area of ​​each subject on the live view image so that the notification method is not different regardless of the priority of the multiple subjects. For example, in the example shown in Figure 17(A), there is no difference in the display form of frame lines 56A that surround the areas of multiple subjects.

[0181] Furthermore, when "AF notification off" is set, the processor 30 does not superimpose and display an indicator indicating the area of ​​the subject as shown in FIGS. 15 and 17 on the live view image.

[0182] [Focus Control Method] FIG. 20 is a flowchart showing an embodiment of a focus control method according to the present invention.

[0183] The focus control method shown in FIG. 20 is a method performed by the processors of the imaging device shown in FIG. 14 (the processor 30 of the visible light camera 10 and the processor 120 of the infrared camera 100).

[0184] 20, the processor determines whether or not an infrared camera 100 is connected to the visible light camera 10 (step S10). The focus control method according to the present invention is a focus control method for when an infrared camera 100 is connected to the visible light camera 10, and if it is determined that an infrared camera 100 is connected to the visible light camera 10, the processor executes the processes from step S12 onwards. Whether or not the infrared camera 100 is connected to the visible light camera 10 can be determined by whether communication between the processor 30 and the processor 120 is possible.

[0185] The processor 120 of the infrared camera 100 acquires from the memory 130 sensor information (image size including the number of vertical and horizontal pixels and pixel pitch) and information such as the angle of view of the image sensor 110 of the infrared camera 100 .

[0186] Next, the processing unit 122 of the processor 120 processes the infrared image I captured by the infrared camera 100. ＩＲ It is determined whether or not a subject to be photographed has been detected based on the temperature information of the subject contained in the infrared image signal (step S14). The determination of subject detection can be made based on whether or not a predetermined temperature range corresponding to the body temperature of the animal that is the subject to be photographed can be extracted from the temperature information of the subject contained in the infrared image signal. If the subject cannot be detected (step S16), the determination process of step S14 is repeated until the subject is detected.

[0187] If the subject is detected (step S18), the processing unit 122 outputs the infrared image I ＩＲ The coordinates of the representative point of the subject (x ｐ ,y ｐ ) is calculated (step S20). ｐ ,y ｐ ) is the visible light image I captured by the visible light camera 10. Ｔ５ The coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) (step S22). This coordinate conversion can be performed by the coordinate conversion unit 124 using the formula (1). That is, by substituting the sensor information of the infrared camera 100 (image size including the number of vertical and horizontal pixels and pixel pitch) and information on the angle of view acquired in step S12, and the sensor information of the image sensor 20 of the visible light camera 10 (image size including the number of vertical and horizontal pixels and pixel pitch), the angle of view, and information on the parallax between the infrared camera 100 and the visible light camera 10 into the formula (1), the coordinates (x ｐ ,y ｐ ) to the coordinate (X Ｐ ,Y Ｐ ) coordinate transformation.

[0188] Next, the processor 30 calculates the coordinates of the representative point of the object (X Ｐ ,Y Ｐ ) is the visible light image IＴ５ The processor 30 determines whether the coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) is the visible light image I Ｔ５ 16, a third indicator (triangle symbol 55A, character 55B) indicating the presence of a focal region including the subject is superimposed on the live view image, thereby prompting the photographer to change the shooting direction of the visible light camera 10 or adjust the angle of view so that the subject is within the angle of view of the visible light camera 10 (step S26).

[0189] On the other hand, the coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) is the visible light image I Ｔ５ If it is determined that the subject is within the range (if "Yes"), the processor 30 notifies the photographer of the position of the subject (step S28). As described with reference to FIG. 15 and other figures, the processor 30 displays the coordinates (X Ｐ ,Y Ｐ The position of the subject is notified to the photographer by superimposing frame lines 54A and 54B surrounding the in-focus area including the subject on the live view image.

[0190] Next, the processor 30 determines whether or not the photographer has performed a zoom operation (change in zoom magnification) (step S30). If the zoom magnification has been changed, the processor 30 acquires lens information (information indicating the focal length, angle of view, etc. of the photographing lens 40) of the visible light camera 10 (step S32), and proceeds to step S22. The lens information acquired in step S32 is the coordinates (x ｐ ,y ｐ ) to the coordinate (X Ｐ ,Y Ｐ ) to the new coordinate (X Ｐ ,Y Ｐ ) is calculated.

[0191] If the processor 30 determines in step S30 that the zoom magnification has not been changed, it calculates the coordinates (X Ｐ ,Y Ｐ) as a reference (step S34). That is, the processor 30 virtually changes the distance z of the subject while detecting the visible light image I Ｔ５ The coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) as a reference, and it is determined whether or not a subject is present in the determined focal region based on the focus evaluation value. Then, phase difference AF is performed based on the data of the phase difference pixel group in the focal region where it is determined that a subject is present (steps S36 and S38).

[0192] Next, the processor 30 determines whether or not the subject is in focus by the phase difference AF (step S40). In the phase difference AF, when the contrast of the data of the phase difference pixel group is low, the phase difference measurement accuracy is low. Therefore, when the phase difference measurement accuracy is low, it is determined that the subject is not in focus, and the process proceeds to step S42.

[0193] In step S42, the coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) is judged to be appropriate or not, and if it is judged that the exposure is not appropriate, the coordinates of the representative point of the subject (X Ｐ ,Y Ｐ ) is adjusted to the appropriate exposure (step S44), and the process proceeds to step S36. This allows exposure control to enable focusing even when it is difficult to focus using the visible light camera 10 (for example, when the animal to be photographed is in a dark place).

[0194] On the other hand, if the exposure is determined to be appropriate, the processor 30 prompts the photographer to change the subject, or automatically changes the subject (step S46), and then transitions to step S36. The subject can be changed when multiple subjects are detected. Alternatively, the processor 30 may prompt the photographer to change the zoom magnification or the shooting position instead of changing the subject.

[0195] On the other hand, if the processor 30 determines in step S40 that the subject is in focus, it notifies the photographer of the end of focus control (step S48). After being notified of the end of focus control, when the photographer operates the shutter button, a still image is taken (step S50).

[0196] [Others] In this embodiment, the processor 120 of the infrared camera 100 recognizes the object and calculates the coordinates (x, y, z) of the representative point of the object based on the temperature information of the object to be photographed contained in the infrared image signal. ｐ ,y ｐ ), and the processing unit 122 acquires the coordinates (x ｐ ,y ｐ ) on the visible light image (X Ｐ ,Y Ｐ ), but this is not limiting, and the processor 30 of the visible light camera 10 may have the functions of the processing unit 122 and the coordinate conversion unit 124. In this case, the visible light camera 10 only needs to acquire the infrared image signal from the infrared camera 100.

[0197] Furthermore, in this embodiment, the case where a still image is taken by the visible light camera 10 has been described, but the focus control device according to the present invention can also be used for focus control when taking a moving image.

[0198] Furthermore, in this embodiment, the hardware structure of a processing unit that executes various processes, such as a CPU (Central Processing Unit), is the following various processors: The various processors include a CPU, which is a general-purpose processor that executes software (programs) to function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0199] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0200] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0201] The present invention also includes a program that, when installed on a computer, causes the computer to execute the focus control method of the present invention, and a non-transitory computer-readable recording medium on which the program is recorded.

[0202] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

[0203] 1, 1'...Subject 10...Visible light camera 12...Attachment 20, 110...Image sensor 30, 120...Processor 32...Focal area determination unit 34...Synthesis unit 36...Exposure determination unit 37...Focus control unit 38...Exposure control unit 39, 130...Memory 40...Photographing lens 50...Viewfinder 52...Display screen 54A, 54B, 56A, 57A, 57B, 58A, 58B...Frame lines 55A, 56B, 57C, 58C...Symbols 55B...Characters 60...Photographing unit 100...Infrared camera 122...Processing unit 124...Coordinate conversion unit A ＩＲ , A Ｔ５ ...Optical axis I ＩＲ ...Infrared image I Ｔ５ ...Visible light image O, O'...Principal point S10 to S50...Step

Claims

1. The image capturing device includes a focus control unit that controls focus when capturing an image, and a processor. The processor: acquiring a visible light image from the first imaging unit, the visible light image being obtained by imaging in a wavelength range including the visible light range; an infrared image having a parallax with respect to the visible light image and obtained by capturing an infrared image in a wavelength region including an infrared region is acquired from a second imaging unit having a parallax with respect to the first imaging unit; acquiring first position information of a first object based on temperature information of the object included in the infrared image; converting the first position information into second position information of the first object in the visible light image; determining a focal region for performing focus control on the visible light image based on the second position information; When the distance from the second imaging unit to the first object is unknown, the processor the first position information, the angle of view and the image size of the first imaging unit; An angle of view and an image size of the second imaging unit; calculating a plurality of pieces of second position information for the same first object based on the information about the parallax; determining a plurality of focal regions on the visible light image based on the plurality of pieces of second position information; Focus control device.

2. The focus control unit performs focus control based on the focus area. The focus control device of claim 1 .

3. calculating a plurality of pieces of second position information by virtually sequentially changing the distance of the first object from a close distance to a long distance or from a long distance to a close distance; The focus control device of claim 1 .

4. The processor: setting the focal region in accordance with the calculated order; When the focus evaluation value by the focus control unit becomes equal to or greater than a first threshold value, the next setting of the focus area is stopped, and focus control by the focus control unit is terminated. The focus control device according to claim 3 .

5. The processor: causing the focus control unit to perform focus control in each of the plurality of focus regions; apply focus control to the focus region in which the focus evaluation value by the focus control unit is maximized among the plurality of focus regions; The focus control device of claim 1 .

6. an optical axis of the first imaging unit and an optical axis of the second imaging unit are parallel to each other; The processor: when the distance of the first subject is equal to or greater than a second threshold, the parallax information is set to zero, and the second position information is calculated based on the first position information, the angle of view and image size of the first imaging unit, and the angle of view and image size of the second imaging unit. The focus control device of claim 1 .

7. the first position information is coordinate information in the infrared image of a representative point of the first subject included in the infrared image, and the second position information is coordinate information in the visible light image of a representative point of the first subject included in the visible light image; The focus control device of claim 1 .

8. a display for displaying a visible light image; the processor causes the display device to superimpose a first indicator indicating the determined focal region on the visible light image; The focus control device of claim 1 .

9. the first indicator is a frame line surrounding the focus area; The focus control device of claim 8 .

10. The processor: when a plurality of subjects included in the infrared image are detected based on temperature information obtained from the infrared image, one of the plurality of subjects is determined to be the first subject, and another subject is determined to be a second subject, and third position information of the second subject in the infrared image is acquired; converting the third position information into fourth position information of the second object in the visible light image; a second indicator indicating the area of ​​the second subject is displayed superimposed on the visible light image displayed on the display device based on the fourth position information; The focus control device of claim 8 .

11. A user interface is provided for setting priorities when multiple subjects are detected, When the processor detects the plurality of objects, the processor determines the first object and the second object in accordance with the set priority order. The focus control device of claim 10.

12. the first index is a frame line indicating the focus area, and the second index is a frame line indicating the area of ​​the second subject; The focus control device of claim 10.

13. The frame line of the first index and the frame line of the second index have different line types or colors. The focus control device of claim 12.

14. When the determined focal region does not exist in the visible light image, the processor notifies the user that the focal region does not exist. The focus control device of claim 1 .

15. When the determined focal region does not exist in the visible light image, the processor superimposes a third indicator on the visible light image displayed on the display device, the third indicator indicating the direction in which the focal region exists, or notifies the user of the direction in which the focal region exists by voice. The focus control device of claim 8 .

16. The third indicator includes a symbol or a character indicating a direction in which the focal region exists.

16. The focus control device of claim 15.

17. an exposure control unit that controls brightness of a subject imaged by the first imaging unit; the processor outputs area information indicating the focal area or the focal area and its peripheral area to the exposure control unit; the exposure control unit controls exposure based on the area information. The focus control device of claim 1 .

18. An imaging device comprising the focus control device according to claim 1.

19. A focus control method for a focus control device including a focus control unit that performs focus control when capturing an image and a processor, comprising: acquiring, from the first imaging unit, a visible light image obtained by imaging in a wavelength region including the visible light region; acquiring an infrared image from a second imaging unit having parallax with respect to the first imaging unit, the infrared image having parallax with respect to the first imaging unit and obtained by capturing an infrared image in a wavelength region including an infrared region; acquiring first position information of a first object based on temperature information of the object included in the infrared image; converting the first position information into second position information of the first object in the visible light image; determining a focus region in which the focus control unit performs focus control based on the second position information; are executed by the processor, When the distance from the second imaging unit to the first object is unknown, the processor In the converting step, a plurality of pieces of second position information for the same first subject are calculated based on the first position information, the angle of view and the image size of the first imaging unit, the angle of view and the image size of the second imaging unit, and information related to the parallax; In the determining step, a plurality of focal regions on the visible light image are determined based on the plurality of pieces of second position information. Focus control method.

20. A program that causes a computer to execute the focus control method according to claim 19.

21. A non-transitory computer-readable recording medium having the program according to claim 20 recorded thereon.