Zoom control device, control method of zoom control device, and recording medium

Abstract

This invention relates to a zoom control device, a control method for the zoom control device, and a recording medium. A subject detection unit of a camera device detects a subject image from an image. An automatic zoom control unit performs zoom control based on the size of the subject detected by the subject detection unit. The automatic zoom control unit automatically selects a specific composition from multiple compositions and sets a reference size for controlling the zoom ratio of the subject based on the selected composition and the size and position of the detected subject. Processing is performed to determine a scene using information including the detection results from the subject detection unit; composition selection processing is performed on the determined scene; and a composition is selected from compositions of the upper body of the subject, a full-body composition, a composition of the subject's face, and compositions of multiple people.

Description

[0001] (This application is a divisional application of the application filed on December 21, 2015, with application number 201910935028.7 and entitled "Zoom Control Device, Control Method for Zoom Control Device and Recording Medium".) Technical Field

[0002] This invention relates to a technique for detecting a subject image from an image and controlling the zoom ratio. Background Technology

[0003] In the prior art, there exist camera devices with the following functions: a function for driving a zoom lens and performing optical zoom (hereinafter referred to as optical zoom) and a function for magnifying a portion of the captured image and performing electronic zoom (hereinafter referred to as electronic zoom). These camera devices also have an automatic zoom function (automatic magnification) that automatically changes the zoom ratio based on the detection information of the subject. The optical zoom operation and the electronic zoom operation are collectively referred to as zoom operation below.

[0004] Japanese Patent Application Publication No. 2012-95019 discloses the structure of a camera equipped with an auto-zoom function that constantly maintains the size of the subject. Japanese Patent Application Publication No. 2012-95019 discloses a method for the photographer to select a composition from multiple options such as "face close-up," "half-body close-up," and "full-body" in the auto-zoom function. In auto-zoom control that maintains the size of the subject image, the size of the subject image used as a reference for zoom control is determined based on the selected composition. For example, when the photographer selects the composition "full-body," a zoom operation is performed so that the full-body image of the person being photographed is included in the frame.

[0005] However, according to Japanese Patent Application Publication No. 2012-95019, the photographer needs to pre-select the desired composition in the auto zoom control. Therefore, whenever the shooting scene or the state of the subject (movement or number, etc.) changes, the photographer needs to adjust the settings. If the composition setting is unsuitable for the shooting scene, it may be impossible to capture the subject at an appropriate size. For example, when setting a "face" composition in a scene shooting a moving or distant subject, the subject may extend beyond the frame. When setting a "full-body" composition in a scene shooting a stationary subject, the subject may appear too small in the composition.

[0006] When you want to use the same composition to photograph a specific subject, you need to change the settings every time the subject changes. For example, if you want to photograph a baby by taking a close-up of his or her "face" or a child by using a "full-body" composition, the photographer must manually change the composition settings when the subject changes. Summary of the Invention

[0007] This invention enables automatic zoom control using a composition suitable for the scene or the determined attributes of the subject, based on the subject's detection or authentication information, without requiring the photographer to make composition choices.

[0008] According to one aspect of the present invention, a zoom control device is provided, comprising: a subject detection component for detecting a subject image from an image; a composition selection component for automatically selecting a composition from a plurality of compositions based on information relating to the subject image detected by the subject detection component; a setting component for setting a reference size of the subject image for controlling the zoom ratio based on the composition selected by the composition selection component and the size and position of the subject image detected by the subject detection component; and a control component for controlling the zoom ratio based on the reference size set by the setting component and the size of the subject images successively detected by the subject detection component.

[0009] According to one aspect of the present invention, a zoom control device is provided, comprising: a subject detection unit for detecting a subject image from an image; a registration unit for associating image information related to the subject and attribute information related to the subject, and registering the image information and the attribute information; an authentication unit for authenticating a subject based on a correlation value between a feature quantity of the subject image detected by the subject detection unit and a feature quantity of the subject image obtained from the image information of the subject registered by the registration unit; and a composition selection unit for selecting from a plurality of compositions. The system automatically selects a composition from the candidates; a setting component is used to set a reference size for the subject image used to control the zoom ratio based on the composition selected by the composition selection component and the size and position of the subject image detected by the subject detection component; and a control component is used to control the zoom ratio based on the reference size set by the setting component and the size of the subject images detected successively by the subject detection component, wherein the composition selection component changes the candidate composition that can be selected based on attribute information related to the subject certified by the certification component.

[0010] According to another aspect of the present invention, a control method is provided in a zoom control device, the control method comprising: an acquisition step for acquiring information relating to a subject image detected from an image; a composition selection step for automatically selecting a composition from a plurality of compositions based on the information relating to the subject image; a setting step for setting a reference size of the subject image for controlling the zoom ratio based on the composition selected in the composition selection step and the size and position of the subject image indicated by the information relating to the subject image; and a control step for controlling the zoom ratio based on the reference size set in the setting step and the size of the subject image indicated by the information relating to the subject image acquired sequentially in the acquisition step.

[0011] According to another aspect of the present invention, a control method is provided in a zoom control device, the control method comprising: a registration step for associating image information related to a subject and attribute information related to the subject, and registering the image information and the attribute information; an acquisition step for acquiring information related to an image of a subject detected from an image; an authentication step for authenticating a subject based on a correlation value between a feature quantity of the subject image based on detection information related to the subject image and a feature quantity of the subject image obtained from the registered image information of the subject; and a composition selection step for selecting from multiple compositions. The process includes: automatically selecting a composition from the candidates; a setting step for setting a reference size of the subject image for controlling the zoom ratio based on the composition selected in the composition selection step and the size and position of the subject image indicated by information related to the subject image; and a control step for controlling the zoom ratio based on the reference size set in the setting step and the size of the subject image indicated by information related to the subject image obtained successively in the acquisition step, wherein, in the composition selection step, the candidate compositions that can be selected based on attribute information related to the subject authenticated in the authentication step are changed.

[0012] Other features of the invention will become apparent from the following description of typical embodiments (with reference to the accompanying drawings). Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an example of the structure of a digital camera according to an embodiment of the present invention.

[0014] Figure 2 This is a diagram illustrating the relationship between focal length and the position of the focusing lens for each subject distance.

[0015] Figure 3 This is an illustration showing the process used to prevent the subject (object) from going out of the frame.

[0016] Figures 4A-4C This is an illustration showing the process used to prevent the subject (person) from going out of the frame.

[0017] Figure 5A and 5B This is an illustration showing the process used to prevent the size of the subject (person) from changing within the frame.

[0018] Figures 6A-6E This diagram illustrates composition settings when the subject is a person.

[0019] Figure 7 This is a flowchart illustrating the processing flow of the automatic zoom function.

[0020] Figure 8 This is a flowchart used to illustrate the subject search process.

[0021] Figure 9A and 9B This is a diagram used to illustrate the search termination region in the subject search process.

[0022] Figure 10A and 10B It is a flowchart used to explain the specified processing of the subject.

[0023] Figure 11 This is a flowchart illustrating another example of specified processing for the subject.

[0024] Figure 12 This is a flowchart used to illustrate the baseline size setting process.

[0025] Figure 13 This is a flowchart used to explain the calculation and processing of screen size.

[0026] Figure 14 This is a diagram illustrating the transformation of automatic composition judgment in the "Auto" composition setting.

[0027] Figures 15A-15C It is a diagram showing the position and size of the outermost subject.

[0028] Figure 16 This is a flowchart illustrating the process used to calculate the baseline size in the "Automatic" layout settings.

[0029] Figure 17 This is a flowchart illustrating the process used to calculate the ratio of the subject's position in the horizontal direction.

[0030] Figure 18 This is a flowchart illustrating the process used to calculate the ratio of the subject's position in the vertical direction.

[0031] Figure 19 This is a table showing the number of faces used to estimate the location of a subject.

[0032] Figure 20 This is a flowchart used to illustrate the automatic zoom control process.

[0033] Figure 21 This is a flowchart used to explain the zoom operation.

[0034] Figure 22 This is a transformation diagram illustrating the automatic composition judgment in the "automatic" composition setting according to the second embodiment.

[0035] Figure 23 It is a table showing the facial proportions based on age when authenticating a subject.

[0036] Figure 24 This is a diagram illustrating the transition of automatic composition judgment in the "Auto" composition setting during subject authentication.

[0037] Figure 25 It is a table showing the initial composition and the conditions for compositional transitions during subject authentication.

[0038] Figure 26 This is a table showing the estimated number of faces in the body position when authenticating a subject.

[0039] Figure 27 It is a table showing the composition options for various categories of subjects when authenticating a subject. Detailed Implementation

[0040] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating an example of the structure of a digital camera 100, which has an automatic zoom function and includes a zoom control device according to this embodiment.

[0041] The lens group is held in the lens barrel 101. The zoom lens 102 moves along the optical axis of the lens barrel 101 to adjust the focal length and optically change the viewing angle (moving the zoom position). The focusing lens 103 moves along the optical axis of the lens barrel 101 to focus. The image stabilizing lens (image blur correction lens) 104 corrects image blur caused by camera shake, etc. The aperture and shutter 105, which adjust the amount of light, are used for exposure control. In this embodiment, the digital camera 100 is an imaging device in which the lens barrel 101 and the camera body are integrated, but the present invention is not limited thereto. This embodiment can also be applied to imaging systems configured to include a camera body and an interchangeable lens detachably mounted on the camera body.

[0042] Image sensor 106 generates an image signal by receiving light passing through lens barrel 101 and converting the image of the subject into an electrical signal via photoelectric conversion. Image sensor 106 is, for example, a charge-coupled device (CCD) type or complementary metal-oxide-semiconductor (CMOS) type image sensor. The image signal generated by image sensor 106 is input to image processing circuit 107 and undergoes various processing such as pixel interpolation and color conversion. The processed image data is stored in image memory 108. Image memory 108 is a storage device such as dynamic random access memory (DRAM) or static random access memory (SRAM).

[0043] The display unit 109 is configured to include a thin-film transistor (TFT) driven liquid crystal display (LCD) and to display captured images (image data) or specific information (e.g., shooting information). The electronic viewfinder (EVF) function, which allows the photographer to align their viewpoint, can be provided through the display of information related to the captured image, such as live view.

[0044] The aperture and shutter drive unit 110 calculates exposure control values ​​(aperture value and shutter speed) based on brightness information obtained through image processing in the image processing circuit 107, and drives the aperture and shutter 105 based on the calculation results. This performs automatic exposure (AE) control. The anti-shake lens drive unit 111 calculates the amount of deflection applied to the digital camera 100 based on deflection detection information using an angular velocity sensor such as a gyroscope sensor. Based on this calculation result, the anti-shake lens 104 is driven to counteract (reduce) the amount of deflection applied to the digital camera 100.

[0045] The focusing lens drive unit 112 drives the focusing lens 103. In this embodiment, the digital camera 100 performs autofocus (AF) control based on a contrast-based method. That is, based on the focusing information (contrast evaluation value) of the shooting optical system obtained through image processing in the image processing circuit 107, the focusing lens drive unit 112 drives the focusing lens 103 to focus the subject. However, the present invention is not limited to this, but instead of the contrast-based method, a phase-difference AF method can be used as AF control. Furthermore, AF control can be performed according to multiple methods, such as a combination of contrast-based and other methods.

[0046] The zoom lens drive unit 113 drives the zoom lens 102 according to the zoom operation instruction. The operation unit 117 includes a zoom lever or zoom button, which serves as a zoom operation component used by the photographer to instruct the camera to zoom. The system control unit 114 performs the following control: detects the operation amount and operation direction of the zoom operation component used for the zoom instruction operation, calculates the zoom drive speed or zoom drive direction, and moves the zoom lens 102 along the optical axis according to the calculation result.

[0047] Image data generated during the shooting operation is transmitted to the recording unit 116 via the interface (I / F) unit 115 for storage. This image data is recorded in one or both of an external recording medium and a non-volatile memory 118. The external recording medium is, for example, a memory card installed in the digital camera 100. The non-volatile memory 118 is a storage medium built into the digital camera 100. The memory 118 stores not only program data or image data, but also settings information of the digital camera 100 or information related to the magnification position in the auto-zoom function, which will be described below.

[0048] In addition to the zoom operation components described above, the operation unit 117 also includes a release switch for indicating the start of shooting and an auto-zoom operation switch for indicating the start or end of the auto-zoom function. Signals from the operation unit 117 are sent to the system control unit 114.

[0049] The system control unit 114 includes a computing device such as a central processing unit (CPU). The system control unit 114 controls the digital camera 100 as a whole by sending control commands to each unit in response to the photographer's actions. The system control unit 114 executes various control programs stored in the memory 118, such as programs for controlling the image sensor 106, AE / AF control, and zoom control (including automatic zoom processing).

[0050] In order to maintain focus when changing the viewing angle through optical zoom, when the lens barrel 101 is a back-focusing type, the focusing lens 103 needs to be moved to the appropriate focal position according to the position of the zoom lens 102. This control is called computer zoom (CZ) control and is performed by the CZ control unit 119 in the system control unit 114. Figure 2 This is a diagram illustrating the relationship between the focal length of a zoom lens and the focal point position for various subject distances. Figure 2 In this embodiment, the relationship between the focal length of the zoom lens and the focused point position is represented by a graph as a data table for each distance to the subject. In this embodiment, this table is called the focusing cam table. Figure 2In the graph, the horizontal axis represents the focal length corresponding to the zoom position, and the vertical axis represents the focal point position. Next to each curve is an example of the distance from the digital camera 100 to the subject (subject distance).

[0051] During AF control, the system control unit 114 performs a scanning operation by controlling the focusing lens drive unit 112 and moving the focusing lens 103 within a predetermined range. Using known methods, the focal position, such as a contrast evaluation value obtained during the scanning operation, is detected. Referring to a focusing cam table, the zoom position and focal position at this time are used to measure the subject distance.

[0052] Next, the control related to the automatic zoom function in the system control unit 114 will be explained. For example... Figure 1 As shown, the system control unit 114 includes a CZ control unit 119, an electronic zoom control unit 120, an automatic zoom control unit (hereinafter referred to as an AZ control unit) 122, and a subject detection unit 123.

[0053] The digital camera 100 has both optical zoom and electronic zoom functions, and a CZ control unit 119 and a zoom lens drive unit 113 are used to control the optical zoom. During zoom operation, the CZ control unit 119 detects the zoom position of the zoom lens 102 for each predetermined control cycle. Then, the CZ control unit 119 controls the focusing lens 103 to track the focusing cam table at the subject distance measured in the AF control corresponding to the detected zoom position. In this way, optical zoom operation can be performed while maintaining focus.

[0054] On the other hand, the electronic zoom control unit 120 and the image memory 108 are used to control electronic zoom. The electronic zoom control unit 120 implements the electronic zoom function by cropping data within the object area from the image data sent to the image memory 108. The electronic zoom control unit 120 achieves smooth electronic zoom display by gradually increasing the cropped range according to the frame rate period of the image captured by the image sensor 106 while the display unit 109 displays the image.

[0055] The subject detection unit 123 detects the desired subject region from the image data stored in the image memory 108. In this embodiment, a subject detection method (face detection processing or color detection processing) for detecting a subject (face of a person or object, etc.) based on facial information or color information included in the image data will be described.

[0056] Face detection processing is a process used to detect facial regions in image data using known algorithms. For example, subject detection unit 123 extracts feature values ​​from a rectangular portion of the image data and compares these feature values ​​with pre-prepared facial feature values. Then, if the correlation value between the two features exceeds a predetermined threshold, subject detection unit 123 determines that the portion is a facial region. By repeating this determination process while changing combinations of the size, position, and angle of the portion, various facial regions present in the image data can be detected.

[0057] In the color detection process, processing is performed to store color information of the subject region specified according to the subject designation method described below as feature colors. When the subject being detected is an object (an "object" other than a person), color detection processing is performed. For example, color difference (RY or BY) signals and RGB signals or luminance signals (Y signals) as output signals from the image processing circuit 107 are used as color information. In the case of detecting a subject, the subject detection unit 123 divides the image data into multiple partial regions and calculates the average value of luminance and color difference for each partial region. The subject detection unit 123 compares the pre-stored feature color information with the color information of each region during subject detection and sets partial regions where the difference between luminance and color difference is equal to or less than a predetermined amount as candidates for subject regions. The following processing is performed: a group of adjacent partial regions among the candidate subject regions are set as the same color region, and regions with the same color region within a predetermined size range are set as the final subject regions.

[0058] In this embodiment, the subject detection unit 123 will be described as having the functions of registering information related to the subject and authenticating the subject, but the registration unit and the authentication unit can be set up independently. That is, the registration unit performs the process of acquiring image information of the subject and registering the image information in the storage unit in association with the attribute information of the subject. The subject authentication unit performs correlation calculations to correlate the feature values ​​of the detected subject image with the feature values ​​of the subject image obtained from the pre-registered image information of the subject, and performs authentication processing for the subject based on the correlation values.

[0059] Next, the authentication methods (face authentication processing and object authentication processing) for determining the attributes of the detected subject in the subject detection unit 123 will be described. In the face authentication processing, a process is performed to pre-register a facial image of a person who is the subject of the photograph and the attributes of that face (name, age, date of birth, gender, and initial composition, etc.). A pattern matching process is performed on all registered facial images and the facial images detected by the face detection processing using a known algorithm. A process is performed to authenticate the registered facial image with the highest correlation value as the detected face. If the correlation value of all registered facial images is less than a predetermined value, the detected face is determined to be an unregistered face. Therefore, the attributes of the person being photographed can be specified. On the other hand, in the object authentication processing, a process is performed to store images of objects other than the person being photographed as template images in the memory 118. For example, there are methods for pre-storing representative subjects such as airplanes, trains, birds, and the moon as template images, or methods for registering images taken by the photographer as template images. To improve detection performance, processing is performed to store images of the same object as multiple image data points from different angles or using different colors. Furthermore, processing is performed to register the attribute information of the object corresponding to the template image (type, judgment information related to whether the object is in motion or stationary, and composition settings, etc.). In addition to the color information of the object region detected in the color detection process, shape information or texture information is also pattern matched against the registered template images. Processing is performed to authenticate the image with the highest correlation value as the detected object. If the correlation value of all template images is less than a predetermined value, the detected object region is judged as an unregistered object. Therefore, the attributes of the object to be photographed can be specified.

[0060] The subject detection unit 123 uses the subject distance information measured by the CZ control unit 119 and the focal length information of the zoom lens 102 to calculate the size of the subject area in the image data. The posture detection unit 124 detects the posture of the digital camera 100 (e.g., upright position / handle up / handle down) based on information from the accelerometer sensor. The shake detection unit 125 determines the deflection state of the digital camera 100 based on angular velocity information from the gyroscope sensor, etc. If the deflection amount (detected value) applied to the gyroscope sensor, etc., is equal to or greater than a predetermined amount (threshold), the shake detection unit 125 determines that the camera is in a handheld state; if the deflection amount is less than the predetermined amount, it determines that the camera is fixed on a tripod, etc. The sensors of the detection unit used to acquire control information of the anti-shake lens drive unit 111 can also be configured to function as accelerometer sensors and gyroscope sensors for detecting posture and shaking.

[0061] Next, an overview of the auto zoom function according to this embodiment and the AZ control unit 122 will be described. In cameras without an auto zoom function, for example, when the subject moves out of the frame while the photographer is framing in telephoto mode and waiting to take a picture, the following operation is required.

[0062] First, the photographer zooms in by manipulating the zoom mechanism to narrow down the focus and then searches for the subject. Once the subject is located, the photographer zooms in again to adjust the perspective until the desired angle is achieved. For example, if the subject's image size changes due to movement, the photographer must manipulate the zoom mechanism to resize the image.

[0063] In contrast, in cameras equipped with auto zoom, the photographer, after setting the auto zoom function, selects the desired subject by operating an operation on a touch panel or similar surface. When auto zoom is enabled, the camera automatically zooms in so that the selected subject has a predetermined size near the center of the image. Besides touch panel operation, methods for selecting the subject include, for example, methods for selecting a subject near the center of the frame when the photographer operates a specific button, or methods for automatically selecting a main subject from subjects detected by the camera.

[0064] The subject detection unit 123 calculates the position or size of the subject area specified from the image memory 108 in the image data. By continuously processing the sampled image data whenever the image is displayed as a live view, the movement of the subject can be tracked. When an image of the tracked subject is detected in the zoom-out area described below, or when the detected image of the subject becomes larger than a predetermined size, the AZ control unit 122 initiates a zoom-out operation. That is, the AZ control unit 122 instructs the CZ control unit 119 or the electronic zoom control unit 120 to zoom out along the wide-angle direction. When a subject is detected in the zoom-in area and the subject image is contained within a predetermined size range, a zoom-in operation towards the telephoto side is performed. Through this process, the photographer can operate the camera without worrying about zoom operation, ensuring that the desired image of the subject is contained within the frame. Even when the subject exceeds the frame, the zoom ratio is automatically changed. Therefore, aiming the viewpoint can be made simpler.

[0065] Next, refer to Figures 3-5B This explains the starting conditions for scaling down or scaling up operations. Figure 3 This is an illustration showing the process used to prevent the subject (object) from going out of the frame. Figures 4A-4C This is an illustration showing the process used to prevent the subject (person) from going out of the frame.

[0066] exist Figures 3-4C In the diagram, frames 300a and 300b are first tracking frames (hereinafter referred to as object tracking frames) used to track the subject (object), and frames 400a to 400f are second tracking frames (hereinafter referred to as face tracking frames) used to track the subject (a person's face). In the following description, for applications to people or objects, in some cases, the object tracking frame and the face tracking frame will be collectively referred to as the subject tracking frame. The subject tracking frame is displayed on the electronic viewfinder screen (EVF screen) of the display unit 109, surrounding the subject image, allowing the photographer to view the designated subject. The position and size of the subject tracking frame on the screen are calculated by the subject detection unit 123 based on facial and color information and are updated periodically.

[0067] Reference Figure 3 This is to explain the processing used to prevent the subject (airplane) from going out of the frame. Figure 3 The reduced area ZO (hereinafter referred to as the ZO area) in the left image represents the area further outward compared to a predetermined ratio of the overall viewpoint (overall image) displayed by the EVF. For example, the center point of the image is 0% and the overall image is 100%, and it is assumed that the boundary of the ZO area is set at 80% of the overall image. In this case, the area of ​​80% to 100% of the overall image is the ZO area. When a part of the object tracking frame 300a enters the ZO area, the AZ control unit 122 controls the start of the reduction operation. The AZ control unit 122 stores the zoom ratio (corresponding to the magnified viewpoint) before the zoom movement in the memory 118. The target zoom ratio or zoom speed during the reduction operation is preset according to the size or movement speed of the subject image. The target zoom ratio or zoom speed can be appropriately calculated according to the size or movement speed of the subject image. The reduction operation is performed according to the target zoom ratio or zoom speed. In this way, the subject can be effectively prevented from going out of the frame.

[0068] Figure 3 The right figure shows the diagram from... Figure 3 The viewpoint shown in the left image is the viewpoint when a zoom-out operation corresponding to a predetermined zoom change is performed. Figure 3The magnified area ZI (hereinafter referred to as the ZI area) shown in the right figure represents the area further inward than the predetermined ratio of the magnified angle of view (the angle before zooming out) 301 in the subject search state. For example, the center point of the image is 0% and the magnified angle of view 301 is 100%, and it is assumed that the boundary of the ZI area is set at 70% of the magnified angle of view 301. In this case, the area of ​​0% to 70% of the entire magnified angle of view 301 is the ZI area. At this time, for example, when the zoom ratio is 1 / 2, the size of the magnified angle of view 301 is 50% of the entire image. Therefore, the boundary of the ZI area is 70% × (1 / 2) = 35% of the entire image. The ZI area can also be regarded as the area of ​​0% to 35% of the entire image. When the photographer changes the direction of the camera so that the object tracking frame 300b is contained in the ZI area, the AZ control unit 122 performs control to start the magnification operation.

[0069] Next, refer to Figures 4A-4C This section explains the processing to prevent the subject (person) from going out of the frame. When the subject is a person and part of the face tracking frame enters the ZO area, a zoom-out operation begins. When the face tracking frame is contained within the ZI area, a zoom-in operation is performed. The case of a person differs from the case of an object. Therefore, since the direction of the subject's movement can be predicted to some extent, the ZO and ZI areas are set according to the area in the predicted direction of movement. When shooting handheld, the subject may go out of the frame due to camera shake, etc. However, if the subject goes out of the frame due to camera shake, the photographer can bring the subject back into the frame by performing a framing operation. Here, if the ZO area is set at the top of the frame and the person is positioned near the center for shooting, the face tracking frame enters the ZO area, and the subject may unintentionally go out of the frame. Therefore, when the subject is a person in a handheld shooting position (the photographer is holding the camera), a ZO area is not set at the top of the frame, considering the photographer's framing operation.

[0070] Thus, in this embodiment, when the subject detection unit 123 detects a face, the AZ control unit 122 changes the ZO and ZI regions based on the camera's posture or shake state. The camera's posture is detected by the posture detection unit 124, and the shake state is determined based on the detection results obtained by the shake detection unit 125. The detection results obtained by the shake detection unit 125 indicate whether the camera is in a handheld state. The following will refer to... Figures 4A-4C Let me explain the situation in detail.

[0071] Figure 4AThe diagram shows the ZO and ZI zones set when the photographer holds the camera in an upright position. In this shooting scenario, if the subject moves horizontally and goes out of the frame, the position of the subject within the frame moves horizontally (long side) relative to the upright frame. Therefore, the ZO and ZI zones are configured as vertical strips (short side) relative to the upright frame. The ZO zone is set as a vertical strip at both ends of a horizontally elongated rectangular frame. The ZI zone is set as a vertical strip in the center of the frame. In this case, when the face tracking frame 400a enters the ZO zone, the AZ control unit 122 determines that zooming in has begun and performs a zoom-in operation corresponding to a predetermined zoom ratio. When the face tracking frame 400b is included in the ZI zone, the AZ control unit 122 determines that zooming in has begun and performs a zoom-in operation corresponding to a predetermined zoom ratio until the zoom returns to the original position. By setting the ZO and ZI zones in this way, it is possible to effectively prevent the subject (person) from going out of the frame.

[0072] Figure 4B The diagram illustrates the ZO and ZI zones set when the camera's posture changes in the same shooting scene and the photographer positions the camera in a vertical position with the handle down or up. In this case, the ZO and ZI zones are configured as vertical strips along the long side of the frame relative to the vertical position. That is, the ZO zone is set as a vertical strip at both ends of the horizontal direction of the long rectangular frame. The ZI zone is set as a vertical strip in the center of the frame. In this case, when the face tracking frame 400c enters the ZO zone, the AZ control unit 122 determines that zooming has begun and performs a zoom-down operation corresponding to a predetermined zoom ratio. When the face tracking frame 400d is included in the ZI zone, the AZ control unit 122 determines that zooming has begun and performs a zoom-up operation corresponding to a predetermined zoom ratio until the zoom returns to the original position. By setting the ZO and ZI zones in this way, horizontal movement of the subject can be detected and the subject (person) can be effectively prevented from going out of the frame.

[0073] Figure 4CThe diagram shows the ZO and ZI regions set when the camera detection unit 125 is in a fixed state. A fixed state means the camera is fixed on a tripod or similar device, and the subject is unlikely to move out of the frame due to camera shake. However, if a zoom operation is performed near the center of the image without the subject within the frame, the subject may move out of the frame due to the zoom operation. Therefore, the ZO region is set around the perimeter of the image, and the ZI region is set further inwards compared to the zoom angle. In other words, the ZO region is set as a rectangular strip located at the corners of the image in both the vertical and horizontal directions. The ZI region is set as a rectangle located in the center of the image. In this case, when the face tracking frame 400e enters the ZO region, the AZ control unit 122 determines that zooming has begun and performs a zoom operation corresponding to a predetermined zoom ratio. When the face tracking frame 400f is included in the ZI region, the AZ control unit 122 determines that zooming has begun and performs a zoom operation corresponding to a predetermined zoom ratio until the zoom returns to the starting position.

[0074] In this embodiment, as described above, the ranges of the ZO and ZI regions are dynamically changed according to changes in the camera's posture or shooting state (handheld / fixed). This effectively prevents the subject from going out of frame while preventing accidental operation caused by camera shake, etc. Furthermore, the ZO or ZI region can be changed according to either the camera's posture or the shooting state (handheld / fixed), or only one of the ZO and ZI regions can be changed.

[0075] Next, a zoom operation for maintaining the subject image relative to the screen size within a predetermined range will be described. In this embodiment, control (size maintenance control) is performed such that the size of the subject image is maintained within a predetermined range relative to the reference size by automatically performing a zoom operation when the detected size of the subject image exceeds a predetermined multiple of the reference size. Figure 5A and 5B This diagram illustrates the processing used to maintain the size of the image of the subject (person) within the frame. Figures 6A-6E It is an explanatory diagram of the composition of the subject (person).

[0076] First, refer to Figures 6A-6E This explains the setting of the range (composition) within which the subject being tracked is contained in the frame. Figure 6A This example illustrates the image display when the composition setting is "Manual". In "Manual" setting, the photographer views the image of a person in the frame while simultaneously changing the size of the face being tracked by manually zooming using the zoom lever. This current size of the subject image is stored as a reference size in memory 118. Figure 6B This example illustrates the screen display when the composition setting is "Face". In the "Face" composition setting, the size at which the face fits within the frame is calculated based on the camera's pose or the face's orientation, and this reference size is stored in memory 118. Similarly, Figure 6C This example shows the screen display when the composition setting is "upper body". Figure 6D This example illustrates the screen display when the composition setting is "full-body". The reference size is calculated in such a way that the reference size is the size set on each screen, and the reference size is stored in memory 118.

[0077] Figure 6E This example illustrates the screen display when the composition setting is "Auto". In the "Auto" setting, the AZ control unit 122 determines an appropriate composition based on factors such as the position of the subject within the frame, the size of the subject, the number of subjects, or the camera's posture. A reference size is calculated to achieve the determined composition, and this reference size is stored in the memory 118. The following will refer to... Figure 12 This will illustrate the method used to calculate the baseline size.

[0078] The photographer can change the composition settings by operating the left and right buttons on the shooting screen of the operation unit 117 or by selecting from the settings menu. When the photographer changes the composition settings, the AZ control unit 122 updates the composition-related information stored in the memory 118. Figures 6A-6E Examples of composition settings for "Manual," "Face," "Upper Body," "Full Body," and "Auto" are shown when the subject is a person, but the composition settings are not limited to these. Only a portion of the five illustrated compositions can be set, or other compositions may be included. When the subject is an object, for example, "Manual," "Large," "Medium," "Small," and "Auto" compositions can be set.

[0079] refer to Figure 5A and 5B In zoom operations used to maintain the proportion of the subject image relative to the frame within a predetermined range, the case where the composition setting is "manual" will be used as an example. Figure 5A This example illustrates the automatic zoom operation performed by the camera when the subject (person) is close to the camera. This zoom operation is used to reduce the size of the subject image relative to the frame within a predetermined ratio. Figure 5A and 5B In the image, face tracking frames 500a to 500f are displayed to enclose the facial region, which is used as the feature area of ​​a person as a subject. Therefore, the size of the face tracking frames is described here as the size of the subject.

[0080] Figure 5A The left figure shows the viewpoint when the subject is specified according to the subject specification method described below. The size of the face tracking frame 500a when the subject is specified is stored in the memory 118 as the reference subject size (reference size). Figure 5A The central figure shows the zoom ratio relative to Figure 5A The left image shows the perspective when the subject approaches the camera without changing its state. For example, the starting size of the zoom-out operation is set to 150% of the size of the face tracking frame 500a, which serves as the reference subject size. When the relationship between the subject tracking frames (face tracking frames) is "face tracking frame 500b > face tracking frame 500a × 150%", that is, when the tracking frame changes by a predetermined amount or more relative to the reference size, the AZ control unit 122 determines that the zoom-out operation has started.

[0081] Figure 5A The right figure shows from Figure 5A The field of view of the central image 501 is reduced to a predetermined zoom ratio, and the face tracking frame 500c is also reduced. Here, the predetermined zoom ratio is set to 1 / 1.5, taking into account the rate of change (150%) of the face tracking frame size relative to the reference subject size at the start of the zoom-out operation. Then, as the subject gets closer to the camera, further zooming towards the wide-angle side is performed so that the subject image can continue to be contained within the predetermined ratio. Therefore, the photographer can focus on operating the release switch.

[0082] In contrast. Figure 5B This example illustrates the automatic zoom operation performed by the camera when the subject moves away from the camera. This zoom operation is used to magnify the subject image relative to the frame within a predetermined range. Figure 5B The left figure shows the viewpoint when the subject is specified according to the subject specification method described below. The size of the face tracking frame 500d when the subject is specified is stored in memory 118 as the reference subject size (in the case where the composition setting is "manual").

[0083] Figure 5B The central figure shows the zoom ratio relative to Figure 5BThe left image shows the perspective when the subject moves away from the camera while the state remains unchanged. For example, a size of 50% of the size of the face tracking frame 500d, which serves as the reference subject size, is set as the starting size for the magnification operation. If the relationship between the face tracking frames is "face tracking frame 500e < face tracking frame 500d × 50%", it is determined that the tracking frame has changed by a predetermined amount relative to the reference size. If the determination condition is met and the face tracking frame 500e is included within the ZI region, the AZ control unit 122 determines that the magnification operation has begun. Here, the ZI region is defined relative to the ZI region. Figure 5B The central view is magnified compared to the predetermined zoom level of 502, which is set to be more inward.

[0084] Figure 5B The right figure shows from Figure 5B The central view is magnified to a predetermined zoom ratio (corresponding to viewpoint 502) and a face tracking frame 500f. Here, taking into account the rate of change (50%) of the face tracking frame size relative to the reference subject size at the start of the zoom operation, the predetermined zoom ratio is set to 1 / 0.5x.

[0085] exist Figure 3 When the subject is an object and in Figures 4A-4C When the subject is a person, the techniques used to prevent the subject from going out of the frame are explained. Figure 5A and 5B When the subject is a person, the process described involves ensuring that the size of the subject image is within a predetermined range relative to the aspect ratio of the frame. Even when the subject being tracked is an object, the same procedure applies as when the subject is a person; zoom operation can be initiated to determine and control the movement. Figure 5A and 5B The size of the subject image shown is maintained. When the composition settings are other than "Manual," it is consistent with... Figure 5A and 5B Similarly, it automatically performs zoom operations based on each reference size.

[0086] Next, refer to Figures 7-27 This will explain how the automatic zoom function works. Figure 7 This is a flowchart illustrating an overall processing example of the autofocus function. In the following text, unless otherwise explicitly stated, it is assumed that the autofocus function is performed based on instructions from system control unit 114 (AZ control unit 122).

[0087] In S700, the AZ control unit 122 first determines the operating status of the auto zoom operation switch of the operation unit 117. If the auto zoom operation switch is pressed, the process proceeds to S701. If the auto zoom operation switch is not pressed, the determination process in S700 is repeated. In S701, subject search processing is performed. Further, the subject designation processing in S702, the reference size setting processing in S703, and the auto zoom control in S704 are performed sequentially.

[0088] Immediately after the start of the auto zoom function processing, or when the subject search button on the operation unit 117 is pressed ("Yes" in S705), the subject search processing of S701 is performed. (Refer to...) Figure 8 Let's explain the subject search processing performed in step S701 during zoom ratio control. When the subject is far from the camera, even if the subject image is within the frame, the subject detection unit 123 may fail to detect it if the subject's feature area is too small. In such cases, automatic zoom based on subject information may not be possible. Therefore, subject search processing is performed to perform magnification up to a predetermined focal length while simultaneously detecting the subject.

[0089] In S800, a process is performed to determine whether the subject search button was pressed immediately after the start of auto zoom or during auto zoom. If the determination result in S800 is positive ("Yes"), the process proceeds to S801. If the determination result is negative ("No"), the process proceeds to S803. In S801, a process is performed to obtain the subject search end focal length. The subject search end focal length is the focal length at which the zoom magnification ends when the search for a subject image in the image ends. Hereinafter, the subject search end focal length is referred to as the search end focal length. In this embodiment, immediately after the start of auto zoom, the search end focal length is set to 85mm using a 35mm size conversion. This focal length is generally considered a suitable standard lens focal length when shooting people. Even if there is no subject (person) or the face cannot be detected (e.g., the person is facing backward), the subject detection can be set to standby state at an appropriate angle after the search ends. Here, the focal length is not limited to 85mm. Regardless of the current zoom ratio, other focal lengths corresponding to a predetermined (fixed) zoom ratio (e.g., 50mm or 100mm) can be used. The photographer can change the search end focal length from the menu settings, or can change it depending on the shooting mode or the subject to be photographed. When processing transitions from S801 to S802, the focal length at the start of the mode is compared with the search end focal length, and it is determined whether the focal length is on the wide-angle side or the telephoto side relative to the search end focal length. If the focal length at the start of auto zoom is on the telephoto side relative to the search end focal length ("No" in S802), the subject search processing ends. Conversely, if the focal length at the start of auto zoom is on the wide-angle side relative to the search end focal length ("Yes" in S802), processing proceeds to S805.

[0090] Conversely, if it is determined in S800 that a subject has been found through the operation of the subject search button, processing for obtaining the search end focal length is performed in S803. In this embodiment, it is assumed that the search end focal length when the subject search button is operated is a telephoto focal length that is 2.5 times the focal length at which the operation is performed. Here, this magnification is only an example, and other magnifications can be used. The photographer can change the search end focal length from the menu settings, or can change the search end focal length according to the shooting mode or the subject to be photographed. The process proceeds from S803 to S804, and compares the search end focal length obtained in S803 with the focal length at the far end (telephoto end), and determines whether the search end focal length is on the wide-angle side or the telephoto side relative to the focal length at the telephoto end. If the search end focal length is on the telephoto side relative to the focal length at the telephoto end, the subject search process ends. If the search end focal length is on the wide-angle side relative to the focal length at the telephoto end, the process proceeds to S805. In S805, the AZ control unit 122 sets the zoom change amount corresponding to the search end focal length obtained in S801 or S803 in the CZ control unit 119 or the electronic zoom control unit 120, and instructs to perform a magnification operation.

[0091] Next, in S806, it is determined whether the current focal length has reached the search end focal length. If the current focal length has reached the search end focal length, the process proceeds to S809. After the zoom operation stops, the subject search process ends. Conversely, if it is determined in S806 that the current focal length has not reached the search end focal length, the process proceeds to S807. S807 is used to determine whether the subject detection unit 123 has detected a subject. If it is determined in S807 that a subject has been detected, the process proceeds to S808. If it is determined that no subject has been detected, the process returns to S806. In S808, it is determined whether the detection position of the subject is within the predetermined search end area of ​​the screen. It is assumed that the predetermined search end area is determined based on a threshold (position determination reference value) preset according to the subject. If the detection position of the subject detected in S807 is within the predetermined search end area of ​​the screen, the process proceeds to S809, the zoom operation stops, and the subject search process ends. Conversely, if it is determined in S808 that the detection position of the subject is not within the predetermined search end area on the screen, the process returns to S806 and the determination process of S806 to S808 continues.

[0092] Here, we will refer to Figure 9A and 9BLet's explain the predetermined search end area described in S808. When photographing a specific subject (person or object), in some cases, the subject is positioned near the center of the frame. If a subject that is not the subject is detected at the periphery of the frame and the subject search process ends, the subject search process may end before the subject is detected. In this case, the automatic zoom function may not be able to be performed according to the photographer's intention. Therefore, in this embodiment, the search end area is set as the area used for determination only when a subject is detected in the central area of ​​the frame (which is the specific area) and in the area near that area, thereby ending the subject search process. Figure 9A In the example shown, the search end area 900a is set when the subject is an object (airplane). When the subject is an object, the search end area for the subject is set to a predetermined proportion (e.g., 70%) relative to the center of the frame in any direction (vertical, horizontal, or vertical). On the other hand, in... Figure 9B The example shown is a search termination area 900b set when the subject is a person. When the subject is a person, the search termination area is set such that it includes 100% of the area in the upper part of the frame from the center of the frame, and also includes a predetermined proportion (e.g., 70%) of the area from the center of the frame in other directions (right, left, or down). When the subject is a person, the face tracking frame 902 of the subject may be positioned in the upper part of the frame. For this reason, even if a subject is detected in the upper part of the frame, the search termination area is set to end the subject search process.

[0093] Next, refer to Figure 10A and 10B To explain Figure 7 The subject designation process is shown in step S702. Figure 10A This is a flowchart illustrating an example of a user specifying a desired subject using the touch panel included in the operation unit 117. In this example, it is assumed that the user specifies the subject by touching the image displayed on the display unit 109. In S1000, the AZ control unit 122 determines whether the touch panel has been pressed. If the touch panel has been pressed, the process proceeds to S1001. If the touch panel has not been pressed, the determination process in S1000 is repeated.

[0094] In S1001, the AZ control unit 122 acquires information related to the location touched by the user (touch location). Subsequently, in S1002, the AZ control unit 122 notifies the subject detection unit 123 of the touch location, and the subject detection unit 123 performs face detection near the touch location. If a face is detected near the touch location in S1002, it is determined that the main subject is a person, and the process proceeds to S1003. Conversely, if no face is detected near the touch location in S1002, it is determined that the main subject is an object other than a person, and the process proceeds to S1004.

[0095] In S1003, the AZ control unit 122 performs control to store facial information of a person being automatically tracked in the memory 118. This specific facial information includes information related to the size of the face, the detection position of the face, and the orientation of the face when the subject is specified. In the case of a camera with facial authentication functionality, identification information such as an authentication ID is also stored in the memory 118. In S1004, the AZ control unit 122 performs control to store the characteristic color near the touch position as color information of the automatically tracked object in the memory 118. This specific color information includes information related to the characteristic color of the subject when it is specified, its brightness, chromaticity value, or the size or center position of the same color area. In the case of a camera with object authentication functionality, identification information such as an authentication ID is also stored in the memory 118. In the following description, facial information and color information are collectively referred to as subject information (including subject size and subject detection position).

[0096] After storing the subject information in memory 118 in S1003 or S1004, the process proceeds to S1005 and the subject tracking frame is displayed. In S1005, the AZ control unit 122 controls the display unit 109 to center the subject detection position and display a subject tracking frame (object tracking frame or face tracking frame) corresponding to the size of the subject. Afterward, the subject designation process ends.

[0097] Thus, the subject detection unit 123 in the system control unit 114 detects a subject at or near the position specified by the photographer on the display unit 109. Then, the AZ control unit 122 causes the display unit 109 to display a subject tracking frame. According to... Figure 10A The example of this processing demonstrates how to simply specify the subject the photographer wants to track using an intuitive method. However, the method of specifying the subject is not limited to this one. See also... Figure 10B Let me illustrate another method.

[0098] Figure 10BThis is a flowchart illustrating an example of the process when the user uses a switch (subject designation switch) different from the automatic operation switch included in the operation unit 117 to designate a subject. In S1006, the AZ control unit 122 first displays a frame used as a reference for subject designation near the center of the screen on the display unit 109. The photographer adjusts the camera orientation so that the image of the subject he or she wants to track using the frame as a reference is contained near the center of the screen. Next, in S1007, the AZ control unit 122 determines whether the subject designation switch has been pressed. If the subject designation switch has been pressed, the process proceeds to S1008. If the subject designation switch has not been pressed, the determination process of S1007 is repeated in the standby state.

[0099] In S1008, the subject detection unit 123 performs face detection near the center of the frame. If a face is detected near the center of the frame, the main subject is identified as a person and the process proceeds to S1009. Conversely, if no face is detected near the center of the frame, the main subject is identified as an object other than a person and the process proceeds to step S1010. Since the detection is performed in S1009... Figure 10A The same process as S1003 and the same as in S1010. Figure 10A The process is the same as in S1004, so a description of these processes will be omitted. After the subject information is stored in memory 118 in S1009 or S1010, the process proceeds to S1011. Then, the subject tracking frame (object tracking frame or face tracking frame) is displayed and the subject designation process ends.

[0100] Thus, in Figure 10B In the processing example, the subject detection unit 123 detects a subject at or near the center of the screen displayed on the display unit 109. The AZ control unit 122 causes the display unit 109 to display a subject tracking frame indicating the position of the subject. Even in cameras without operating components such as a touch panel, the user can easily specify the subject.

[0101] Figure 11 This is a flowchart illustrating an example of a process for automatically selecting a subject to be tracked from faces detected when the auto-zoom operation switch included in the operation unit 117 is pressed. In S1100, the subject detection unit 123 first performs face detection on the entire frame to determine whether a subject's face is detected. If only one person's face is detected on the entire frame, it is determined that the main subject is a person and the process proceeds to step S1101. Conversely, if no face is detected on the frame, the process proceeds to S1105.

[0102] In step S1101, if the number of people whose faces are detected is only one person, the subject detection unit 123 selects that face as the primary face. If multiple faces are detected, the subject detection unit 123 selects from these faces as the primary face of the subject to be tracked. As a criterion for selecting the primary face, for example, there are methods for preferentially selecting faces whose detection location is closer to the center of the frame. If multiple faces are located in approximately the same position, there are methods for selecting the larger face as the primary face. If the camera has a face authentication function, there is a method where, if there are faces registered for authentication, the face registered for authentication is preferentially set as the primary face.

[0103] The process proceeds to S1101-S1102. Then, after the AZ control unit 122 stores facial information related to the main face in the memory 118, the process proceeds to S1103 and the face tracking frame is displayed on the screen. Next, in S1104, a face change determination is performed. That is, when multiple faces are detected, the automatically selected main face may not be the face the photographer desires. For this reason, the photographer can change the main face. At this time, if the photographer presses the switch (auto-zoom operation switch, etc.) of the operation unit 117 and specifies the desired face, the following process is performed: the main face is changed to another face among the faces detected by the face tracking frame that was not selected as the main face. If the main face is changed in S1104, the process returns to S1102 and the facial information to be stored in the memory 118 is updated. In S1103, the face tracking frame is changed to the size and detection position of the newly selected main face.

[0104] Conversely, if the main subject is determined to be an object other than a person, the processing proceeds from S1100 to S1105. Then, the AZ control unit 122 controls the storage of characteristic colors near the center of the screen as color information of the automatically tracked object in the memory 118. Next, in S1106, after displaying the object tracking frame on the screen, the subject designation process ends. Figure 11 The basic processes in S1102, S1105, S1103 (S1106) are respectively related to Figure 10A The processing in S1003, S1004 and S1005 is the same.

[0105] Thus, in Figure 11In the processing example, the subject detection unit 123 in the system control unit 114 performs face detection on the entire image displayed on the display unit 109. When multiple faces are detected, the AZ control unit 122 controls the display unit 109 to display a subject tracking frame indicating the position of a first face that is the primary subject among the multiple faces. When the subject changes from the first face to a second face, the AZ control unit 122 controls the display unit 109 to display a subject tracking frame indicating the position of the second face. The photographer can easily specify the desired subject with fewer operations.

[0106] In the subject designation process, if the camera includes both the touch panel included in the operation unit 117 and the subject designation switch, they can be applied simultaneously. Figure 10A Methods and Figure 10B Both methods. Figure 7 After determining the change in the subject in S706 as shown, it can be used Figure 10A Alternatively, the 10B method can be used to specify the subject. After the subject search process in S701, it can also be utilized... Figure 11 The method for specifying the subject can be used. The subject specification processing applied according to this processing flow can be changed. In the subject specification processing ( Figure 7 In the case of the end of S702), processing enters Figure 7 The S703 is used for processing to set the size of the tracked subject to a reference size.

[0107] Reference Figures 12-19 To explain Figure 7 The S703's normal reference size setting process is shown. Additionally, refer to... Figures 23-27 This explains the baseline size setting process when a pre-registered subject is detected and authenticated through subject authentication processing. Figure 12 This is a flowchart illustrating an example of the reference size setting process when the subject is a person. In S1200, the AZ control unit 122 first determines whether the composition setting stored in the memory 118 is "manual" as described above. If the composition setting is "manual," the process proceeds to S1201. If the composition setting is not "manual," the process proceeds to S1203.

[0108] In S1201, the AZ control unit 122 performs subject detection and judgment. If the subject detection unit 123 detects that the subject is a person, the AZ control unit 122 determines whether a face has been detected. If the subject is an object, the AZ control unit 122 determines whether the same characteristic color has been detected. The judgment process in S1201 is repeated until a subject is detected. If a subject is detected, the process proceeds to S1202. In S1202, the AZ control unit 122 sets the size of the detected subject image as a reference size and stores this size in the memory 118. Then, the process ends. Therefore, when the composition setting is "manual," the size of the initially detected subject image (detection size) at the start of the reference size setting process is stored as the reference size in the memory 118.

[0109] In S1203, the AZ control unit 122 calculates the image size based on the camera's pose information determined by the pose detection unit 124. This image size is then used to calculate the reference size in S1211, S1704, S1705, S1806, and S1807, which will be described below. (Refer to...) Figure 13 To illustrate the image size calculation process, in this embodiment, we will illustrate the case where the image size of a still image is set to "long side × short side = 640 × 480" and the image size of a moving image is set to "long side × short side = 640 × 360". However, the image size is not limited to these.

[0110] In S1300, the posture detection unit 124 determines whether the photographer is holding the camera in an upright position or a vertical position. If the photographer is holding the camera in an upright position, the process proceeds to S1301. If the photographer is holding the camera in a vertical position, the process proceeds to S1306. In S1301, processing is performed to set the size of the long side 640 to the horizontal screen size. Next, in S1302, processing is performed to determine whether the current setting mode is a mode that can simultaneously capture still images and motion videos (still image / motion image simultaneous shooting mode). The still image / motion image simultaneous shooting mode is a mode in which the photographer presses the release switch of the operation unit 117 during the shooting of motion images to simultaneously capture still images, or the framing operation immediately preceding the capture of a still image is automatically recorded as a motion image. If, in S1302, it is determined that the current setting mode is a simultaneous still image / moving image shooting mode, a process is performed to compare the size of the shorter side of the moving image with the size of the longer side of the still image and set the smaller size as the vertical screen size. This is because, by automatically zooming based on the smaller screen size, the subject is prevented from exceeding the frame from any angle of the still image or the moving image. Therefore, in this embodiment, when the current setting mode is a simultaneous still image / moving image shooting mode, the process proceeds to S1304 to set 360, which is the size of the shorter side of the moving image, as the vertical screen size. After that, the screen size calculation process ends.

[0111] If, in S1302, it is determined that the current setting mode is not a simultaneous still image / moving image shooting mode, the process proceeds to S1303. In S1303, it is determined whether a moving image is being recorded or a still image is being captured. If it is determined that a moving image is being recorded, the process proceeds to S1304. Conversely, if it is determined that no moving image is being recorded, the process proceeds to S1305. In S1305, the 480° section, which is the size of the shorter side of the still image, is set to the vertical screen size.

[0112] If it is determined in S1300 that the photographer is keeping the camera in a vertical position, then the processing steps S1306 to S1310 are performed. The processing in S1306 to S1310 involves switching between the horizontal and vertical image sizes as described in S1301 to S1305. In other words, since "horizontal" and "vertical" can be used interchangeably in the above description, detailed explanations will be omitted. In this embodiment, the image size for still images is described as 4:3 (see...). Figure 15A and 15B And the aspect ratio of the moving images is 16:9 (see...). Figure 15CExamples are provided, but the image size can be changed according to the aspect ratio. The aspect ratio of the image of a person relative to the frame can be set to constant, regardless of the aspect ratio. An example of different image sizes between still and moving images is illustrated in the simultaneous still / moving image shooting mode, but it is also possible to simultaneously shoot multiple still or moving images of different sizes.

[0113] exist Figure 12 If the screen size is determined in S1203, the process proceeds to S1204. If the composition setting stored in memory 118 is one of "face," "upper body," or "full body," in S1204 to S1209, the AZ control unit 122 determines the ratio of the face image size to the screen size (called the face ratio) based on the composition setting. According to this embodiment, the face ratio is the ratio of the length of the face tracking frame in the vertical direction relative to the screen, and may also be the ratio of the length of the face tracking frame in the horizontal direction relative to the screen, or an area ratio, etc. Specifically, for example, in S1204, it is determined whether the composition setting is "face." If the composition setting is "face" (yes in S1204), the face ratio is determined to be 27% (S1207). If the composition setting is not "face," the process proceeds to S1205 to determine whether the composition setting is "upper body." If the composition setting is "upper body" (yes in S1205), the face ratio is determined to be 16% (S1208). If the composition setting is not "upper body", the process proceeds to S1206 to determine if the composition setting is "full body". If the composition setting is "full body" ("yes" in S1206), the facial ratio is determined to be 7% (S1209). After the facial ratio is determined in steps S1204 to S1209, the process proceeds to S1210.

[0114] In S1210, the AZ control unit 122 calculates the reference size by multiplying the face ratio determined in S1204 to S1209 by the vertical screen size calculated in S1203. This is a method of calculating the reference size based on the face ratio relative to the screen size in the vertical direction, assuming the subject (person) is standing. If the orientation of the face can be determined using the subject detection unit 123, the orientation of the screen size can be determined based on the vertical orientation of the face on the screen. Furthermore, after processing to store the calculated reference size in the memory 118, the reference size setting process ends. Therefore, when the composition setting is one of "face," "upper body," or "full body," the size with a predetermined face ratio corresponding to the composition setting relative to the screen size is set as the reference size.

[0115] When the detected subject is a pre-registered face, a more suitable baseline size for the subject can be calculated by adjusting the facial proportions according to the subject's age. Figure 22 This illustrates a face ratio set based on the age of the subject authenticated through facial recognition processing. Regarding the subject's age, there are methods that use the registered age information as an attribute of the subject without any alteration. Furthermore, in the case of a registered birth date, the subject's current age can be calculated based on the birth date and the current date. In this embodiment, when the detected subject is not authenticated (unregistered face), the same face ratio as when the subject is a child aged 2 to 12 years is set. When the composition setting is "Face," the face ratio is set to 27%, regardless of the subject's age. When the composition setting is either "Upper Body" or "Full Body," the face ratio is changed according to the subject's age. For example, when the subject is an infant aged 0 to 1 year, the face ratio is set to a value larger than the value for unauthenticated subjects. When the subject is an adult aged 13 years or older (in terms of body shape), the face ratio is set to a value smaller than the value for unauthenticated subjects. Therefore, when the composition is set as "upper body" or "full body", an appropriate size can be obtained so that the image size of the subject does not become significantly larger or smaller compared to the frame size.

[0116] If the composition setting in S1206 is not "full-body," meaning it's determined to be "automatic," then processing proceeds to S1211. Then, the AZ control unit 122 automatically determines an appropriate composition based on the shooting status, the number of subjects, or the movement of the subjects. (Refer to...) Figure 14 and 24 This section explains the automatic composition judgment and composition selection processing of the S1211 when the composition setting is "Auto". The "Auto" composition setting is a mode that simplifies composition matching by judging the shooting scene and causing the camera to automatically select an appropriate composition (automatic composition selection).

[0117] Figure 14This is a transformation diagram of the automatically selected composition in this embodiment. In this embodiment, a composition selection process is performed to select an appropriate composition from five compositions: upper body 1400, full body 1401, face 1402, multiple people 1403, and manual 1404, based on the shooting scene. In the composition matching operation, when shooting still images, the subject image needs to be set so that the subject image has an appropriate size immediately before shooting. On the other hand, when shooting moving images, the subject image needs to be set so that the subject image generally has an appropriate size during shooting, such as continuous tracking of the subject or a subject that is close to stopping. When shooting moving images, image data is recorded during zoom-in or zoom-out operations based on the change in composition. Therefore, video can be shot effectively by changing the composition at the appropriate time. Thus, the appropriate composition differs between still images and moving images. Therefore, processing is performed during still image framing 1405 and during moving image recording 1406 to change the selected composition according to the shooting state. In other words, in this embodiment, the still image shooting preparation state is exemplified as the first operation state, and the moving image recording state is exemplified as the second operation state. A composition is selected from the upper body 1400, full body 1401, and multiple people 1403 as a selection candidate composition for the first composition group during still image framing 1405. A composition is selected from the upper body 1400, full body 1401, and face 1402 as a selection candidate composition for the second composition group during moving image recording 1406. That is, the compositions for the upper body 1400 and full body 1401 are common to both the first and second composition groups, the compositions for the multiple people 1403 are only included in the first composition group, and the composition for the face 1402 is only included in the second composition group. Depending on the specifications of the camera equipment, more than three operation states and multiple composition groups (including compositions selectable in each operation state) corresponding to these operation states can be set.

[0118] Here, the criteria for determining the shooting scene and the changes in each composition will be explained. When the automatic composition determination process S1211, with the composition set to "Auto," begins, the upper body 1400 is set to the initial composition state. The composition of the upper body 1400 is selected when a single subject that will be stopped during still image framing 1405 or during moving image recording 1406 is determined to be the shooting scene. In this scenario, the composition close to the subject can be appropriately set by framing the upper body of the subject within the frame.

[0119] If movement of the subject is detected in the upper body composition 1400, or if the distance between the camera and the subject is determined to be greater than a predetermined distance D1 (first threshold), the composition changes to a full-body composition 1401. The full-body composition 1401 is selected during still image framing 1405 or during moving image recording 1406 in scenarios where a moving subject or a distant subject (i.e., a subject that may easily extend beyond the frame) is being photographed. In such scenarios, the subject is tracked within the full-body composition to ensure it does not extend beyond the frame as much as possible. If the subject in the full-body composition 1401 stops for more than a predetermined time, and the distance between the camera and the subject is determined to be less than a predetermined distance D2 (second threshold), processing is performed to return the composition to the upper body composition 1400. The predetermined time is a pre-set judgment standard time.

[0120] The following describes a method for determining whether a subject is moving or stationary. If at least one of the changes in position or size of the subject detected by the subject detection unit 123 and the amount of jitter detected by the jitter detection unit 125 during the subject detection state is greater than a predetermined amount, the subject being photographed is determined to be moving. If both the changes in position or size of the subject detected by the subject detection unit 123 and the amount of jitter detected by the jitter detection unit 125 during the subject detection state are less than a predetermined amount for a predetermined time or longer, the subject being photographed is determined to be stationary. It is assumed that the duration of the detection state is measured using a measuring timer.

[0121] Furthermore, a method for calculating the distance between the camera and the subject will be explained. First, the focal length used as a reference and the subject distance related to the subject size are measured in advance, and the measurement results are stored in memory 118. The subject distance is calculated by performing the calculation "Subject distance = (Reference subject distance × Focal length × Reference subject size) / (Reference focal length × Detection size)" on the reference value stored in memory 118. For example, assuming that a subject image with a detection size of 40 pixels is detected using a focal length of 120mm, given a reference subject distance of 2m, a reference focal length of 24mm, and a reference subject size of 20 pixels, the subject distance is calculated as "(2m × 120mm × 20pix) / (24mm × 40pix) = 5m".

[0122] Next, the composition when multiple people are detected as subjects in the frame will be explained. During still image framing 1405, if multiple people are detected as subjects in the frame within an upper-body 1400 or full-body 1401 composition, processing is performed to change the composition to multiple people 1403. The number of subjects detected at this time is stored in memory 118. The composition of multiple people 1403 is the composition selected during still image framing 1405 when it is determined that the shooting scene is a group photo where all multiple subjects are included from a predetermined angle. In the composition state of multiple people 1403, processing is performed to determine whether the number of detected subjects has changed. If the number of detected subjects increases, new subjects enter the frame, thereby immediately updating the number of subjects stored in memory 118 and changing the composition. Conversely, if the number of detected subjects decreases, for example, if one subject is horizontally oriented, there is a possibility that it cannot be detected temporarily. Therefore, if the composition is changed immediately, there is a possibility that the subject may not be detected and may extend beyond the frame. Therefore, if the number of subjects decreases, it is determined whether this state has lasted for a predetermined time. If this state has lasted for a predetermined time, the number of subjects stored in memory 118 is updated and the composition is changed. Furthermore, if the number of subjects is a single person, processing is performed to return the composition to the upper body 1400.

[0123] During motion picture recording 1406, if a composition is set to include all subjects from the perspective, and subjects other than the subject being photographed are in motion, there is a concern about accidental zooming in response to those subjects. Therefore, if multiple people are detected as subjects during motion picture recording 1406, processing is performed to set only the main subject as the subject and maintain either the upper body 1400 or the full body 1401 as the composition. If a face is detected near the center of the frame in the upper body 1400 composition during motion picture recording 1406 and remains there for more than a predetermined time, processing is performed to change the composition to face 1402. The face 1402 composition is selected when it is determined during motion picture recording 1406 that the photographer is focused on the subject's face. In such a scenario, the face of the subject being focused on can be photographed at a larger size by setting a composition that is closer to the face than the upper body. By changing the zoom speed to a significantly lower or higher speed compared to the normal speed during the magnification operation caused by a change in composition, more effective video can be captured. Furthermore, in the composition of face 1402, if a face is detected near the periphery of the frame for a predetermined time or longer, or if it is determined that the photographer has changed the framing, processing is performed to return the composition to the upper body 1400. In this embodiment, the periphery of the frame is exemplified as a specific area, and a measurement timer is used to measure the duration for which the position of the subject's face image detected by the subject detection unit 123 is in the periphery. As a method for determining a change in framing, if the subject detection unit 123 does not detect a subject and the amount of shake detected by the shake detection unit 125 is greater than a predetermined amount, it is determined that the photographer has changed the framing.

[0124] During still image framing 1405, the photographer sometimes continues to wait to take a picture with the same composition until the direction or expression of the subject's face becomes the desired state. In this case, when setting a composition close to the face, there is a possibility that the composition is different from the composition the photographer expects. Therefore, even if a face is detected near the center of the frame in the upper body 1400 composition during still image framing 1405 and remains there for more than a predetermined time, processing is performed to maintain the composition of the upper body 1400.

[0125] Next, the composition when changing the shooting state at the start / end of motion picture recording will be explained. The compositions of upper body 1400 and full body 1401 are selectable in both cases during still image framing 1405 and during motion picture recording 1406. Therefore, in these compositions, the original composition is maintained even when the shooting state changes. On the other hand, the compositions of multiple people 1403 and face 1402 are selected in only one case, either during still image framing 1405 or during motion picture recording 1406. Therefore, if motion picture recording begins with a composition of multiple people 1403 during still image framing 1405, or ends with a composition of face 1402 during motion picture recording 1406, a change to the common composition of upper body 1400 is considered. However, there is a concern in this situation: if zooming begins along with the start or end of motion image recording, the photographer might be annoyed if an image is recorded in zoom mode at the beginning of a motion image or during still image framing. Therefore, in cases where the shooting state changes in this situation, a process is performed to temporarily change the composition to manual 1404. Manual 1404 composition sets the size of the main subject image initially detected after the composition change to a reference size. Therefore, zooming does not begin immediately due to the change in shooting state, and the photographer does not experience discomfort. Even if the subject begins to move, the size of the subject image can be maintained. Furthermore, if a predetermined time has elapsed during manual 1404 composition, a process is performed to return the composition to upper body 1400.

[0126] Figure 24 This diagram shows the transformation when the subject is identified as a registered face. (Towards...) Figure 14 Common compositions are assigned the same reference numerals, and detailed descriptions of these reference numerals will be omitted. During still image framing 1405, the candidate compositions are selected as upper body 1400, full body 1401, face 1402, and multiple people 1403, and a composition is selected. Figure 25The initial state and composition change conditions for compositional shift based on the age of the subject authenticated in the face authentication process are shown. In this embodiment, if the age of the authenticated subject is less than a predetermined age, and a face is detected near the center of the frame for a predetermined time even during still image framing 1405, processing is performed to change the composition from upper body 1400 to face 1402. Specifically, if the subject is a 0-year-old infant and the photographer frames the face from the center of the frame, a composition close to the face is selected. That is, if it is determined that the subject is a relatively still infant (different from the infant's age in the face ratio setting described above, only 0 years old is set as the composition judgment condition), the composition is changed so that a still image can be captured by close-up of the face. Since the predetermined time for changing the composition from upper body 1400 to face 1402 is set shorter compared to other ages, it is easy to select the composition of face 1402.

[0127] When the subject is a child aged 1 to 12, it is assumed that there are multiple shooting scenes involving movement and play, and the composition is set to easily select a full-body 1401. Therefore, when the subject is a child, the initial composition setting is set to full-body 1401. This indicates that the judgment distance of the subject from D1 or D2 is shorter than the judgment distance for other ages, thus even when the distance to the subject is short, the upper body 1400 is changed to full-body 1401.

[0128] As mentioned above, in Figure 12 When the composition setting in S1211 is "Auto", the appropriate composition is automatically determined based on the shooting status, the number of subjects, or the movement of the subjects. The zoom can be adjusted to a more appropriate angle by changing the composition selected based on the age of the subject. The data representing the composition after the determination is stored in memory 118 and processed in S1212.

[0129] In S1212, it is determined whether the composition determined in S1211 has changed relative to the preceding composition. Figure 14The process proceeds to S1213 to determine whether the selected composition has changed or whether the number of subjects has changed in the case of multiple people (1403). If the composition has changed, the process proceeds to S1213 to determine whether the selected composition is a manual composition (1404). If the selected composition is a manual composition (1404), the process proceeds to S1201 to perform processing to store the size of the initially detected subject image as a reference size in memory (118). If the selected composition is not a manual composition (1404), the process proceeds to step S1215 to calculate the reference size in the composition setting "Auto". If it is determined in S1212 that the composition has not changed, the process proceeds to S1214 to determine whether the subject image has moved within the frame. As a method for calculating the amount of movement of the subject image within the frame, the reference size is set in S1215, and then the center position of the face tracking frame immediately following the zoom operation up to the angle of view relative to the reference size is stored in coarse initial position (118). The distance from the position stored in memory 118 to the center position of the detected face tracking frame is calculated as the movement amount. The center positions of the face tracking frames can be added together in each predetermined period, and the movement amount within the frame can be calculated. If the calculated movement amount is equal to or greater than a predetermined amount, it is determined that the subject has moved. By standardizing the predetermined amount using the size of the detected face tracking frame (setting a predetermined multiple of the face tracking frame size), the actual movement amount of the subject can be set to approximately the same regardless of the size of the subject image within the frame. If it is determined in S1214 that the subject has moved within the frame, the process proceeds to S1215 to calculate the reference size in the "Auto" composition setting. If it is determined in S1214 that the subject has not moved within the frame, the reference size setting process ends without needing to update the reference size.

[0130] The following will be referenced Figures 15A-19 This explains the baseline size calculation process in the "Auto" composition setting. When the composition setting is one of "Face," "Upper Body," or "Full Body," in... Figure 12In S1210, a reference size is calculated to ensure a constant ratio between the face and the frame size. That is, the same reference size is calculated regardless of the subject's position on the frame. Therefore, if the composition is changed to enlarge the subject image when it is located near the periphery of the frame, there is a possibility that the subject may extend beyond the frame. When calculating the reference size using this method, the photographer needs to frame the subject near the center of the frame. Therefore, this method is unsuitable, for example, when the subject is offset from the center of the frame and is shot along with the background. When multiple subjects are present, another method is needed to calculate the reference size. Therefore, when the composition setting is "automatic," the size of the subject image is determined based on the composition determined in S1211, and a process is performed to calculate the reference size regardless of the detected subject's position so that the subject is contained within the frame.

[0131] Figures 15A-15C It is a graph showing the relationship between the size of the frame and the position of the subject closest to the edge of the frame among multiple subjects. Figure 15A and 15B Examples of still images are shown, and Figure 15C Examples of moving images are shown, illustrating multiple people as subjects. Figure 16 This is a flowchart illustrating the overall process for calculating the baseline size in the "Auto" layout settings.

[0132] exist Figure 16 In S1600, the first step is to calculate the subject position ratio Rh in the horizontal direction. The subject position ratio Rh in the horizontal direction is a predetermined proportion related to the horizontal screen size, calculated as the ratio of the distance from the center of the screen to a horizontal position near the shoulder of the nearest subject (hereinafter referred to as the shoulder position and represented by Xsmax) × 2. The predetermined proportion is, for example, 80% or 90% of the horizontal screen size, and varies depending on the number of people arranged horizontally in S1703, which will be explained below. Figure 15A Among subjects 1500a, 1501a, and 1502a, subject 1500a is the closest to the perimeter from the center of the frame in the horizontal direction. (Refer to...) Figure 17 The flowchart illustrates the process used to calculate the subject position ratio Rh in the horizontal direction.

[0133] In step S1700, the shoulder position of the subject (represented by Xs) is calculated based on the center position or size of the face tracking frame. Assuming the center of the frame is the origin (0,0), the center position of the face tracking frame is (Xc,Yc), and the size of the face tracking frame is S. When Ns is the number of face tracking frames located within a distance from the center position of the face tracking frame to the shoulder position, the shoulder position Xs on the periphery of the frame can be calculated as "Xs = |Xc| + S × Ns". Figure 19 This shows an example of Ns settings. According to... Figure 12 The value of Ns is changed based on the composition determined in S1211. For example, in the composition of face 1402, Ns is set to 1 (in this case, the horizontal position near the ear instead of the shoulder position). In other cases ("upper body", "full body" and "multiple people"), Ns is set to 2.

[0134] After calculating the shoulder position Xs of the subject, the process proceeds to S1701. In S1701, if the maximum shoulder position Xsmax is not stored in memory 118, the shoulder position Xs calculated in S1700 is stored as the maximum shoulder position Xsmax. If the maximum shoulder position Xsmax is stored in memory 118, the shoulder position Xs calculated in S1700 is compared with the maximum shoulder position Xsmax. If the shoulder position Xs is greater than the maximum shoulder position Xsmax, processing is performed to update the maximum shoulder position Xsmax in memory 118 to the shoulder position Xs. After the processing to update the maximum shoulder position Xsmax, the process proceeds to S1702. In S1702, it is determined whether the processing to calculate the shoulder positions Xs of all subjects and update the maximum shoulder position Xsmax has ended. If the calculation of shoulder positions Xs and the update of the maximum shoulder position Xsmax have not ended, the process returns to S1700. Once the determination of the maximum shoulder position Xsmax of all subjects is completed and the update of the maximum shoulder position Xsmax is completed, the process proceeds to S1703.

[0135] In S1703, a process is performed to determine the number of people arranged as subjects in the horizontal direction (the number of people arranged in the horizontal direction). This is because, in cases where the number of subjects is large, such as in a group photo, a composition is set where all subjects are concentrated in the frame, and in cases where the number of subjects is small, a composition is set where the remaining space around the edges of the frame is left blank. As a method for counting the number of people arranged in the horizontal direction, if the face tracking frames overlap in the vertical direction (the vertical direction of the frame), the overlapping subjects are counted as one person. For example, if four people are detected as subjects in the frame, and the face tracking frames of two of these four people overlap in the vertical direction of the frame, the number of people is counted as three. In S1703, the number of people arranged in the horizontal direction is compared with a threshold (e.g., two people). If the number of people arranged in the horizontal direction is determined to be equal to or less than two, the process proceeds to S1704. If the number of people arranged in the horizontal direction is determined to be equal to or greater than three, the process proceeds to S1705. In S1704 and S1705, the subject position ratio Rh in the horizontal direction is calculated. The subject position ratio Rh in the horizontal direction in S1704 is calculated as the ratio of the maximum shoulder position Xsmax×2 to 80% of the horizontal frame size. The subject position ratio Rh in the horizontal direction in S1705 is calculated as the ratio of the maximum shoulder position Xsmax×2 to 90% of the horizontal frame size. After S1704 or S1705, the processing for calculating the subject position ratio Rh ends.

[0136] Next, in Figure 16 In step S1601, a process is performed to calculate the subject position ratio Rv in the vertical direction. The subject position ratio Rv in the vertical direction is a predetermined proportion of the vertical frame size, calculated as twice the distance from the center of the frame to the vertical position of the head of the nearest subject (hereinafter referred to as head position) or to the vertical position of the subject's body (hereinafter referred to as body position). The predetermined proportion is, for example, 90% of the vertical frame size. Hereinafter, Yhmax is used to represent the head position relative to the nearest subject from the center of the frame, and Ybmix is ​​used to represent the body position relative to the nearest subject from the center of the frame. Figure 15B Among subjects 1500b, 1501b, and 1502b, subject 1500b has its head closest to the periphery of the frame relative to the center in the vertical direction, while subject 1501b has its body closest to the periphery of the frame relative to the center. (Refer to...) Figure 18 The flowchart illustrates the process used to calculate the subject position ratio Rv in the vertical direction.

[0137] exist Figure 18In S1800, the subject's head position (represented by Yh) is calculated based on the center position or size of the subject's face tracking frame. The head position Yh can be calculated as "Yh = Yc + S × Nh". Nh represents the number of face tracking frames located within the distance from the center of the face tracking frame to the head position. Figure 19 The example setting for Nh is shown below. For the value of Nh, it is related to... Figure 12 Regardless of the composition determined in S1211, Nh = 1.5 is set. After calculating the head position Yh of the subject, the process proceeds to S1801.

[0138] In S1801, if the maximum value of the head position Yhmax is not stored in memory 118, a process is performed to store the head position Yh calculated in S1800 as the maximum head position Yhmax. If the maximum head position Yhmax is stored in memory 118, the head position Yh calculated in S1800 is compared with the maximum head position Yhmax. If the head position Yh is greater than the maximum head position Yhmax, a process is performed to update the maximum head position Yhmax in memory 118 to the head position Yh. After the update process, the process proceeds to S1802.

[0139] Next, in S1802, the body position of the subject (represented by Yb) is calculated based on the center position or size of the subject's face tracking frame. The body position Yb can be calculated as "Yb = Yc - S × Nb". Nb represents the number of face tracking frames located within the distance from the center position of the face tracking frame to the body position. Figure 19 and 26 The image shows an example of Nb settings. Figure 26 An example setting for Nb is shown when the detected subject is a registered face. For example, in the case of an uncertified subject and in the case of a composition of face 1402, Nb is set to 1.5. Based on... Figure 12The value of Nb is changed based on the composition determined in S1211. For example, in the case of a face 1402 composition, Nb is set to 1.5. In the case of an upper body 1400 composition, Nb is set to 5. In the case of a full body 1401 composition, Nb is set to 10. In the case of a multiple people 1403 composition, Nb is set to 3.5. In this setting, the center of the image is set at the chest in face 1402, below the waist in upper body 1400, at the legs in full body 1401, and near the waist in multiple people 1403. When the composition is one of upper body 1400, full body 1401, or multiple people 1403, the value of Nb is changed according to the age of the subject. In this embodiment, when the detected subject is not certified, the same Nb as that for children aged 2 to 12 years is set. When the subject is an infant aged 0-1 years, Nb is set to a value smaller than that of an uncertified subject. When the subject is an adult aged 13 years or older (in terms of body size), Nb is set to a value larger than that of an uncertified subject. Therefore, when the selected composition is upper body 1400, full body 1401, or multiple people 1403, it is possible to reduce the occurrence of the subject appearing significantly larger or smaller than the frame size. After calculating the body position Yb of the subject, processing proceeds to S1803.

[0140] In S1803, if the minimum body position Ybmin is not stored in memory 118, a process is performed to store the body position Yb calculated in S1802 as the minimum body position Ybmin. If the minimum body position Ybmin is stored in memory 118, the body position Yb calculated in S1802 is compared with the minimum body position Ybmin. If the body position Yb is less than the minimum body position Ybmin, a process is performed to update the minimum body position Ybmin in memory 118 to the body position Yb. After this update process, the process proceeds to S1804.

[0141] In S1804, the head position Yh and body position Yb of all subjects are calculated, and it is determined whether the processing for updating the maximum head position Yhmax and minimum body position Ybmin has ended. If the updating of the maximum head position Yhmax and minimum body position Ybmin has not ended, the processing returns to S1800. If the determination of the maximum head position Yhmax and minimum body position Ybmin of all subjects has ended, and the updating of the maximum head position Yhmax and minimum body position Ybmin has ended, the processing proceeds to S1805. In S1805, the absolute value of the maximum head position Yhmax is compared with the absolute value of the minimum body position Ybmin, and processing is performed to determine which value is located closer to the periphery of the frame. If it is determined that the maximum head position Yhmax is located closer to the periphery of the frame, the processing proceeds to S1806. If it is determined that the minimum body position Ybmin is located closer to the periphery of the frame, the processing proceeds to S1807. In S1806 and S1807, the ratio Rv of each subject position in the vertical direction is calculated. In S1806, the subject position ratio Rv in the vertical direction is calculated as the ratio of the absolute value of the maximum head position Yhmax × 2 to 90% of the vertical frame size. In S1807, the subject position ratio Rv in the vertical direction is calculated as the ratio of the absolute value of the minimum body position Ybmin × 2 to 90% of the vertical frame size. After S1806 or S1807, the processing for calculating the subject position ratio Rv in the vertical direction ends.

[0142] Subsequently, Figure 16In step S1602, the subject position ratio Rh in the horizontal direction is compared with the subject position ratio Rv in the vertical direction. Therefore, the position with the largest ratio among the predetermined ratios of the image size in each direction and the ratios of the distances from the center of the image to each position of the subject (i.e., the position closest to the edge of the image) can be determined. These positions are the maximum shoulder position Xsmax, the maximum head position Yhmax, and the minimum body position Ybmin. If the subject position ratio Rh in the horizontal direction is determined to be greater than the subject position ratio Rv in the vertical direction, the process proceeds to step S1603. If the subject position ratio Rh in the horizontal direction is determined to be equal to or less than the subject position ratio Rv in the vertical direction, the process proceeds to step S1604. In S1603 and S1604, a reference size is calculated based on the detected size of the main subject and the ratio of the subject position closest to the edge of the image determined in S1602. The reference size is calculated in S1603 as "detection size / subject position ratio Rh in the horizontal direction", and in S1604 as "detection size / subject position ratio Rv in the vertical direction". That is, in S1603, a zoom operation is performed so that the main subject image has a reference size obtained by multiplying the size of the main subject image by "1 / subject position ratio Rh in the horizontal direction". In S1604, a zoom operation is performed so that the main subject image has a reference size obtained by multiplying the size of the main subject image by "1 / subject position ratio Rv in the vertical direction". In this way, the subject can be positioned according to the perspective of the composition determined in S1211 without exceeding the frame.

[0143] exist Figure 12When the composition is set to "automatic" in S1215, the position of the subject closest to the periphery of the screen in the horizontal and vertical directions is determined based on the composition determined in S1211, and a reference size is calculated so that the subject position is contained within the screen. Therefore, regardless of the position of the subject image on the screen, the subject can be contained at an appropriate angle without exceeding the frame. The reference size can be calculated using the same process regardless of whether the number of subjects is singular or plural. In this embodiment, the direction of the screen size used to calculate the reference size is determined assuming that the person used as the subject is standing. The present invention is not limited to this. As long as the subject detection unit 123 can determine the direction of the face, the direction of the screen size can be determined based on the vertical direction of the face on the screen. Furthermore, an example of composition determination or reference size calculation when the subject is a person has been described, but the present invention is also applicable when the subject is an object. However, in this case, instead of "face", "upper body", "full body" and "multiple people", it is assumed that the composition to be selected is "large", "medium", "small" and "multiple". Based on these compositions, the number of object tracking frames can be set when calculating the subject's position (corresponding to Ns, Nh, and Nb in the case of people). When the subject can be authenticated through object authentication processing, the composition can be changed based on the authenticated subject. For example, as... Figure 27 As shown, when the subject is a moving object such as an airplane, train, or bird, the motion of the subject is determined using the method described above for judging whether the subject is moving or stationary. If the subject is determined to be moving, the composition is set to "small". If the subject is a moving object and is determined to be stationary, the composition is set to "medium". If the subject is a stationary object such as the moon, the composition is set to "large". In this way, even when the subject is an object other than a person, the composition can be changed according to the authentication result.

[0144] exist Figure 12 If the baseline size setting process is complete, then processing will proceed to... Figure 7 S704. In S704, the AZ control unit 122 performs automatic zoom control based on the subject information detected by the subject detection unit 123 and the reference size stored in the memory 118. The following will refer to... Figure 20Let's explain the autofocus control. After the autofocus control ends, the process proceeds to S705. Then, the AZ control unit 122 determines whether the photographer has given an instruction to search for a subject. That is, the AZ control unit 122 determines whether the photographer has pressed the subject search button on the operation unit 117. If the subject search button has been pressed, the process returns to S701 and performs the subject search process. Conversely, if the subject search button has not been pressed, the process proceeds to S706. In S706, it is determined whether the subject being autofocused has been changed through operation of the touch panel of the operation unit 117 or the subject designation switch, etc. If the subject has changed, the process returns to S702 to perform the subject designation process. Conversely, if the subject has not changed, the process proceeds to S707.

[0145] In step S707, it is determined whether the composition settings have been changed by operating the left and right buttons of the operation unit 117 from the shooting screen. If the composition settings have changed, the data of the composition settings stored in the memory 118 is updated, and then the process returns to S703 to perform reference size setting processing. Conversely, if the composition settings have not changed, the process proceeds to S708. In S708, it is determined whether the auto zoom operation switch of the operation unit 117 has been pressed. If it is determined that the auto zoom operation switch has been pressed, the auto zoom function ends. Conversely, if it is determined that the auto zoom operation switch has not been pressed, the process proceeds to step S709. In S709, processing is performed to determine whether the composition settings stored in the memory 118 are "auto". If the composition settings are "auto", the process returns to S703 to perform reference size setting processing including auto composition determination. Conversely, if the composition settings are other than "auto", the process returns to S704 to continue auto zoom control.

[0146] Next, refer to Figure 20 The flowchart below illustrates the auto zoom control process of S704. In S2000, the AZ control unit 122 first determines whether the subject detection unit 123 has detected a subject. If no subject is detected in S2000, the auto zoom control ends. If a subject is detected in S2000, the process proceeds to S2001. The determination in steps S2001 to S2003 is the auto zoom start determination process used to prevent the subject from going out of the frame. That is, as... Figures 3-4C This process is performed to prevent the subject from going out of the frame. In S2001, the AZ control unit 122 determines whether the subject tracking frame related to the subject being tracked has entered the ZO region. This ZO region is related to... Figure 3 The left image and Figures 4A-4CThe ZO region corresponds to this. If, in S2001, the subject tracking frame enters the ZO region (i.e., there is a high probability that the subject will exceed the frame), the process proceeds to S2004 to begin a zoom-out operation. Here, the zoom-out operation corresponds to the zoom-out operation used to prevent the subject from exceeding the frame. After the zoom-out operation, the auto-zoom control ends.

[0147] Conversely, if in S2001 the subject tracking frame does not enter the ZO area, i.e., the camera captures the subject near the center of the frame, the process proceeds to S2002. In S2002, it is determined whether the preceding zoom operation was a zoom-out operation performed because the subject tracking frame entered the ZO area in S2001; that is, whether the preceding zoom operation was a zoom-out operation used to prevent the subject from going out of the frame. If in S2002 it is determined that the operation occurred after a zoom-out operation used to prevent the subject from going out of the frame, the process proceeds to S2003. Conversely, if in S2002 it is determined that the operation did not occur after a zoom-out operation used to prevent the subject from going out of the frame, the process proceeds to S006.

[0148] In S2003, the AZ control unit 122 determines whether the subject tracking frame related to the subject being tracked is contained within (included in) the ZI region. Here, the ZI region is... Figure 3 The right image and Figures 4A-4C The ZI region corresponds to this. If it is determined in S2003 that the subject tracking frame is not contained within the ZI region, the auto zoom control process ends. Conversely, if it is determined that the subject tracking frame is contained within the ZI region, the process proceeds to S2005. That is, if the camera captures the subject such that its size is near the center of the frame and within the field of view of the zoom return position, a zoom operation begins in S2005. Here, the zoom operation corresponds to the zoom operation used to prevent the subject from exceeding the frame. After the zoom operation, the auto zoom control ends.

[0149] In this embodiment, in order to make the control to prevent the subject from going out of the frame and the size maintenance control in auto zoom compatible, the subject is captured near the center of the frame due to the control to prevent the subject from going out of the frame, and size maintenance control can be performed. Therefore, after the zoom-out operation used for the control to prevent the subject from going out of the frame, the auto zoom processing (S2006 to S2010 below) for constantly maintaining the subject size as described below is not performed. In other words, when the control to prevent the subject from going out of the frame is performed, size maintenance control is not performed until the zoom-in operation used for the control to prevent the subject from going out of the frame is completed.

[0150] Next, the processing in S2006 to S2010 will be explained. If, in S2002, it is determined that no operation was performed after the shrinking operation used to prevent the subject from exceeding the frame, the process proceeds to S2006. The AZ control unit 122 compares the detected size of the subject with the size obtained by multiplying a reference size by a predetermined ratio (represented by N1, where N1 > 1). The process will utilize... Figure 12 The size of the subject, as indicated by the information related to the reference subject set in S1202, S1210, and S1215, is compared with the size of the subject detected in S2000. If the size of the subject detected in S2000 is greater than N1 times the size of the reference subject, that is, if the ratio of the subject image to the frame exceeds a predetermined value, the process proceeds to S2009. In S2009, the AZ control unit 122 begins a zoom-out operation. Here, this zoom-out operation corresponds to the zoom-out operation used for size maintenance control. After the zoom-out operation, the auto zoom control ends.

[0151] Conversely, in S2006, if the size of the subject detected in S2000 is equal to or less than N1 times the size of the reference subject, the process proceeds to S2007. In S2007, the AZ control unit 122 compares the size of the subject in the information related to the reference subject with the size of the subject detected in S2000. If the detected size of the subject is less than the size of the reference subject multiplied by a predetermined ratio (represented by N2, where N2 < 1) (less than N2 times), that is, if the ratio of the subject image to the frame is less than a predetermined value, the process proceeds to S2008. Conversely, if the detected size of the subject is equal to or greater than N2 times the size of the reference subject, the auto zoom control ends.

[0152] In S2008, the AZ control unit 122 determines whether the subject tracking frame related to the subject being tracked is contained within (including) the ZI region. This is because, when the subject is at the periphery of the frame, the zoom operation prevents the subject from exceeding the frame. Here, the ZI region is... Figure 5B This corresponds to the ZI region shown in the central diagram. In S2008, if it is determined that the subject tracking frame is not contained within the ZI region, the autofocus control process ends.

[0153] Conversely, if it is determined in S2008 that the subject tracking frame is contained within the ZI region, the process proceeds to S2010. In S2010, the AZ control unit 122 initiates a zoom operation. Here, this zoom operation corresponds to the zoom operation used for size maintenance control. Thus, in this embodiment, to prevent the subject from exceeding the frame even during the zoom operation used for size maintenance control, the subject image is contained within the ZI region before the zoom operation begins. After the zoom operation, the auto zoom control ends.

[0154] Next, refer to Figure 21 Let me explain the zoom operation. Figure 21 It is used for explanation Figure 20 The flowcharts for the zoom-out or zoom-in operations in S2004, S2005, S2009, and S2010 are as follows: First, in S2100, the AZ control unit 122 retrieves the zoom change amount (the change in zoom magnification) from the memory 118. In the case of a zoom-out operation used to prevent the subject from exceeding the frame, the zoom change amount is set based on information related to the detected subject. Specifically, in the zoom-out operation used to prevent the subject from exceeding the frame ( Figure 20 In S2004, as the size of the subject decreases, the reduction amount is set to be relatively small. Therefore, it is possible to prevent the subject from becoming too small and undetectable due to the reduction operation. Considering the minimum detectable size of the subject, the reduction operation may not be possible if the subject is smaller than a predetermined size. In the zoom operation used to prevent the subject from exceeding the frame, the zoom ratio before the reduction operation begins is stored in memory 118. The zoom ratio is set so that it is the same as the zoom ratio before the reduction operation begins.

[0155] The shrinking operation used in size maintenance control Figure 20 In S2009), a reduction amount (1 / N1 times) corresponding to the predetermined N1 times used in the judgment of S2006 is set. Therefore, even if the subject cannot be detected, a minimum reduction operation can be performed until the size of the subject becomes the size of the reference subject. Similarly, in the magnification operation used for size maintenance control ( Figure 20 In S2010, an amplification change amount (1 / N2 times) corresponding to the predetermined N2 times used in the judgment of S2007 is also set.

[0156] In S2101, the AZ control unit 122 sets the zoom change amount obtained in S2100 in the CZ control unit 119 or the electronic zoom control unit 120 and gives an instruction to perform zoom processing. Next, in S2102, the AZ control unit 122 determines which zoom operation is currently in progress: the zoom operation used to prevent the subject from going out of the frame or the zoom operation used to maintain the size. If the zoom operation at the current moment is the zoom operation used to prevent the subject from going out of the frame (…), then… Figure 20 (S2004 and S2005), processing proceeds to S2105. Conversely, in the case where the zoom operation at the current moment is a zoom operation used for size maintenance control ( Figure 20 (S2009 and S2010), then proceed to S2103.

[0157] In S2103, the AZ control unit 122 determines whether the subject detection unit 123 has detected a subject. If a subject is detected, the process proceeds to S2104. If no subject is detected, the process proceeds to S2105. In S2104, the size of the subject, represented by information related to a reference subject, is compared with the size of the subject detected in S2103. If, as a result of the comparison, it is determined that the size of the subject detected in S2103 and the size of the reference subject are not within a predetermined ratio range (within a predetermined change amount), the process returns to S2102, and the zoom operation determination continues. If, through the zoom operation, the size of the subject detected in S2103 and the size of the reference subject are within a predetermined ratio range, the process proceeds to S2106. In S2106, the AZ control unit 122 stops the zoom operation and then ends the zoom operation process.

[0158] If, in S2102, it is determined that the zoom operation is a zoom operation used to prevent the subject from going out of the frame, or if, in S2103, no subject is detected during a zoom operation used for size maintenance control, the process proceeds to S2105. In S2105, based on the zoom change amount obtained in S2100, the AZ control unit 122 determines whether zoom processing corresponding to a predetermined zoom change amount has been performed for each zoom operation. If no zoom processing with the predetermined zoom change amount has been performed, the process returns to S2102 to continue. Conversely, if zoom processing with the predetermined zoom change amount has been performed, the process proceeds to S2106, and the AZ control unit 122 stops zooming during operation and ends the zoom operation processing.

[0159] In this embodiment, the shooting scene is determined based on factors such as the movement of the subject, the number of subjects, the detection time of the subjects, or the distance from the camera to the subjects. The camera automatically selects an appropriate composition based on the determined scene. Based on the composition selected through composition selection processing, a reference size for setting the subject image is performed, and a zoom operation is conducted to ensure that the detected subject image has the reference size. According to this embodiment, automatic zoom can be performed without the photographer needing to select a composition, allowing an appropriate composition to be selected based on the shooting scene determined by the camera's detection information.

[0160] In this embodiment, the camera automatically selects a more appropriate composition based on the subject's attribute information obtained from the subject's authentication results. Based on the composition selected through composition selection processing, a reference size for setting the subject image is determined, and a zoom operation is performed so that the detected subject image has the reference size. Specifically, by changing the facial ratio according to age in the calculation of the person's reference size, a zoom operation can be performed to set a more appropriate size. In this embodiment, a case where the composition selection or facial ratio is changed according to the person's age is illustrated, but the invention is not limited to this. For example, a structure can be implemented that registers an initial composition or facial ratio for each person (name). An example of classifying age into infants (0 years or 0-1 years), children (under 12 years old), and adults (13 years and older) is illustrated. Age can be set in more detail, or it can be changed according to gender. In this embodiment, automatic zoom control can be performed using a composition suitable for the subject's attributes determined based on the subject's authentication information, without the need to select a composition for the subject.

[0161] Second Embodiment

[0162] Next, refer to Figure 22 The second embodiment of the present invention will now be described. Figure 22 Show Figure 12 The automatic composition setting in S1211 is related to the automatic composition judgment and processing. Figure 14 Examples of transformation diagrams with different compositions. Towards and Figure 14 Common compositions will be assigned the same reference numerals, and detailed descriptions of these compositions will be omitted.

[0163] In the first embodiment, an example of determining the shooting scene when photographing a person different from the photographer as the subject is described. For convenience, this mode is referred to as the normal mode. In some cases, the camera is pointed at the photographer himself or her to photograph one or more people, including the photographer or her, as the subject. The mode suitable for this type of shooting is called the self-portrait mode. In the self-portrait mode, effective portrait photography can be performed by adjusting the angle suitable for self-portrait via automatic zoom control and applying background blur effects or skin-smoothing filters, etc.

[0164] In composition matching, in normal mode, the subject image needs to be sized appropriately depending on whether the subject is stationary or moving. Conversely, in Selfie mode, even when the subject is stationary, the size needs to be adjusted based on whether the photographer is holding the camera or has lowered it. For example, when the photographer is holding the camera, the shot should include more of the face in the field of view. When the photographer has lowered the camera, the shot should include the entire body in the field of view. Thus, the appropriate composition differs between normal mode and Selfie mode.

[0165] Therefore, in this embodiment, the process for performing... Figure 22 The process of changing the selected composition between the normal mode 2200 and the selfie mode 2201. That is, the normal mode is exemplified as the first mode and the selfie mode is exemplified as the second mode. A composition is selected from the upper body 1400, full body 1401, and multiple people 1403, which are selection candidates belonging to the first composition group of the normal mode 2200. A composition is selected from the selfie face 2202 and selfie full body 2203, which are selection candidates belonging to the second composition group of the selfie mode 2201.

[0166] Here, the judgment conditions for the shooting scene and the changes in composition will be explained. When the automatic composition judgment process S1211 is started with the composition setting "Auto", the upper body 140° is set as the initial composition. When the mode is Normal Mode 220°, the change is... Figure 14 The same transition occurs during still image framing at 1405. Therefore, a description of this transition will be omitted.

[0167] When the mode changes from normal mode 2200 to Selfie mode 2201, the shake detection unit 125 determines whether the camera is being held by the photographer's hand based on the amount of deflection applied to the gyroscope sensor, etc. When the mode changes to Selfie mode 2201, and the deflection is equal to or greater than a predetermined amount, thus determining that the camera is being held by the photographer's hand, the composition changes to self-portrait face 2202. The composition of self-portrait face 2202 is a close-up shot of the photographer's face taken using the camera held by the photographer's hand. Even when shooting multiple people including subjects, the zoom is adjusted to include as many faces as possible within the field of view.

[0168] In selfie mode 2201, the same composition is selected whether the photographer is the only subject or multiple people are subjects. In other words, the composition is... Figure 19 The number of faces in the subject position estimation shown is set to the same value. As mentioned above, the number of faces in the subject position estimation is... Figure 12 In the "automatic" reference size calculation process S1215 of the composition setting, the number of tracking frames (Ns, Nh, Nb) starting from the center position of the face tracking frame used to estimate the shoulder position Xs, head position Yh, and body position Yb of the subject is calculated. Specifically, by setting different numbers of tracking frames Ns up to the shoulder position and Nb up to the body position according to the selected composition, the same reference size is calculated to set the composition. In the first embodiment, when photographing a person's subject using a "face" composition, the number of tracking frames starting from the center position (Xc, Yc) of the face tracking frame is Ns = 1 for the shoulder position and Nb = 1.5 for the body position, and a reference size is calculated to set the area near the subject's ears or chest as the range conceived by the viewing angle. When shooting multiple subjects using a "multiple people" composition, the number of tracking frames starting from the center position (Xc, Yc) of the face tracking frame is Ns = 2 for the shoulder position and Nb = 3.5 for the body position. A reference size is calculated to set the area near the shoulders or waist of all subjects to the range encompassed by the viewing angle. On the other hand, in the case of a self-portrait 2202, regardless of the number of subjects, the number of tracking frames is set to Ns = 1 for the shoulder position and Nb = 1.5 for the body position, the same as for the "face". Therefore, a reference size is calculated to set the area near the ears or chest of all subjects to the range encompassed by that viewing angle, and the zoom is adjusted to a suitable viewing angle for a self-portrait when the photographer is holding the camera.

[0169] When the composition is changed to a selfie face 2202, the number of detected subjects is stored in memory 118. In the composition state of the selfie face 2202, processing is performed to determine whether the number of detected subjects has changed. If the number of detected subjects increases, the number of subjects stored in memory 118 is immediately updated and the reference size is changed. If the number of subjects decreases after this state has lasted for a predetermined time, the number of subjects stored in memory 118 is updated and the composition is changed. This is because, in the case of multiple people 1403, where one subject is horizontally oriented and temporarily undetectable, the subject is prevented from exceeding the frame by immediately changing the composition.

[0170] When the mode changes from normal mode 2200 to Selfie mode 2201, the shake detection unit 125 determines whether the camera is placed (fixed) on a table or the like based on the amount of deflection applied to the gyroscope sensor, etc. If the mode changes to Selfie mode 2201 and the deflection is less than a predetermined amount, thus determining that the camera is lowered, the composition changes to full-body Selfie 2203. The composition of full-body self-timer 2203 is the composition when the camera is lowered and the photographer's entire body is captured. When photographing multiple people including the photographer, the zoom is adjusted to allow the field of view to encompass the entire body of all subjects. In full-body Selfie 2203, regardless of the number of subjects, the shoulder position Ns = 2 and the body position Nb = 10 are set as the same number of tracking frames as for "full-body". Therefore, the shoulders and legs of all subjects are calculated to a reference size that is within the range of the field of view, and the zoom is adjusted to a suitable field of view for Selfie with the camera lowered. When the composition changes to full-body Selfie 2203, the number of detected subjects is stored in the memory 118. When the number of subjects changes, such as in a full-body selfie 2203 or a face selfie 2202, the number of subjects stored in the memory 118 is updated and the reference size is changed.

[0171] In this embodiment, the shooting scene determined based on the set mode is changed, and the camera automatically selects a composition more suitable for each mode. This embodiment illustrates a case where an appropriate composition is selected from multiple composition selection candidates within a single mode, but only one composition is the selection candidate corresponding to that mode. Furthermore, normal mode and Selfie mode are illustrated as modes, but the invention is not limited to these modes. Besides normal mode and Selfie mode, for example, embodiments can be implemented such as selecting a "full-body" composition of the participant in sports mode or selecting a composition of the baby's "face" in baby mode.

[0172] In this embodiment, automatic zoom control can be performed without the photographer needing to select a composition, so that an appropriate composition can be selected based on the shooting scene determined by the set mode and the camera's detection information.

[0173] Other embodiments

[0174] The present invention can also be implemented by supplying a program that implements one or more functions of the above embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be implemented by circuitry (e.g., an ASIC) that implements one or more functions.

[0175] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0176] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be interpreted in the broadest sense to include all such modifications and equivalent structures and functions.

[0177] This application claims priority to Japanese Patent Application No. 2014-260109, filed on December 24, 2014; Japanese Patent Application No. 2015-202330, filed on October 13, 2015; and Japanese Patent Application No. 2015-202331, filed on October 13, 2015, the entire contents of which are incorporated herein by reference.

Claims

1. A zoom control device, comprising: A subject detection component for detecting a subject from multiple images; Setting components are used to set a reference size for controlling the zoom ratio of a subject, wherein the reference size changes at least in response to a determination of whether the subject is moving or stationary; as well as A control unit is configured to control the zoom ratio based on a reference size set by the setting unit and the size of the subject detected by the subject detection unit from each of a plurality of sequential images. Specifically, when a moving subject is detected, the control unit controls the zoom level to make the moving subject in the image appear smaller, and The control component controls the zoom ratio so that when the subject is detected near the center of the image for a predetermined period of time or more, the zoom ratio is adjusted to make the size of the subject in the image larger.

2. The zoom control device according to claim 1, wherein, The setting component sets a first reference size when the subject is moving, and sets a second reference size larger than the first reference size when the subject is stopped.

3. The zoom control device according to claim 1, wherein, The setting component also changes the reference size based on the distance between the zoom control device and the subject, which is determined based on the size of the subject detected by the subject detection component and information related to the zoom ratio controlled by the control component.

4. The zoom control device according to claim 1, wherein, Set multiple shooting modes, and The setting component also changes the reference size depending on whether a selfie mode is set.

5. The zoom control device according to claim 1, further comprising: A registration component is used to associate image information related to a subject with attribute information related to the subject, and to register the image information and the attribute information; as well as An authentication component is used to authenticate a subject based on the correlation value between the feature quantity of the subject detected by the subject detection component and the feature quantity of the subject obtained from the image information of the subject registered in the registration component.

6. The zoom control apparatus of claim 5, wherein, The setting component also changes the base size in response to the authentication result of the authentication component.

7. The zoom control device according to claim 5, wherein, The attribute information includes one or more pieces of information related to the subject, such as the person's age, date of birth, gender, and initial scene composition, as well as... The setting component changes the reference size based on one or more pieces of information, such as the age, date of birth, gender, and initial scene composition of the person, which are attribute information related to the subject.

8. The zoom control device according to claim 5, wherein, The attribute information includes information related to the subject such as the type of object, whether it is a moving or stationary object, and one or more pieces of information in the initial scene composition. The setting component changes the reference size based on the type of object (as attribute information related to the subject), the determination information of whether it is a moving or stationary object, and one or more pieces of information in the initial scene composition.

9. The zoom control device according to claim 1, further comprising: A scene composition selection component is used to automatically select a scene composition from multiple scene compositions, based at least on a determination of whether the subject is moving or stationary. The setting component is configured to change the reference size based on the scene composition selected by the scene composition selection component, and The base size changes according to the scene composition selected by the scene composition selection component.

10. The zoom control device according to claim 9, wherein, The scene composition selection component also changes the scene composition to be selected based on the number of subjects detected by the subject detection component.

11. The zoom control device according to claim 9, wherein, The setting component is configured to set the reference size based on the scene composition selected by the scene composition selection component and the size and position of the subject detected by the subject detection component.

12. The zoom control apparatus of claim 10, wherein, The setting component is configured to set the reference size based on the scene composition selected by the scene composition selection component and the size and position of the subject detected by the subject detection component.

13. The zoom control device according to claim 9, wherein, The various scene compositions differ in the ratio of the size of the subject to the size of the frame.

14. The zoom control device according to claim 1, wherein, If the change in the size or position of the subject and the shake of the zoom control device are both less than a predetermined amount for a predetermined time, the setting component determines that the subject is stopped.

15. The zoom control apparatus of claim 1, wherein, The zoom ratio is controlled such that the size of the subject detected by the subject detection component from each of a plurality of sequential images is within a predetermined ratio range relative to the reference size.

16. A control method performed in a zoom control device, the control method comprising: The subject detection step is used to detect subjects from each of multiple images; A setting step is used to set a reference size for the subject used to control the zoom ratio, wherein the reference size changes at least in response to a determination of whether the subject is moving or stationary; as well as A control step is used to control the zoom ratio based on the reference size set in the setting step and the size of the subject detected in each of the sequentially multiple images in the subject detection step. Wherein, when a moving subject is detected, the zoom ratio is controlled to make the moving subject in the image smaller, and The zoom ratio is controlled such that when the subject is detected near the center of the image and remains there for a predetermined period of time, the zoom ratio is adjusted to make the size of the subject in the image larger.

17. A non-transitory recording medium storing a control program for a zoom control device, the control program causing a computer to perform steps of a control method performed by the zoom control device, the control method comprising: The subject detection step is used to detect subjects from each of multiple images; A setting step is used to set a reference size for the subject used to control the zoom ratio, wherein the reference size changes at least in response to a determination of whether the subject is moving or stationary; as well as A control step is used to control the zoom ratio based on the reference size set in the setting step and the size of the subject detected in each of the sequentially multiple images in the subject detection step. Wherein, when a moving subject is detected, the zoom ratio is controlled to make the moving subject in the image smaller, and The zoom ratio is controlled such that when the subject is detected near the center of the image and remains there for a predetermined period of time, the zoom ratio is adjusted to make the size of the subject in the image larger.

Images