Imaging device, image processing method, program

The imaging device dynamically adjusts composition by using a composition data storage unit and cropping calculation unit to address the issue of unnatural images caused by subject size changes, ensuring smooth transitions and optimal framing.

JP2026182295APending Publication Date: 2026-11-19SONY GROUP CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2025078412
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing imaging devices with automatic tracking framing functions apply the same composition regardless of changes in subject size due to distance, leading to unnatural images.

Method used

The imaging device includes a composition data storage unit storing multiple data sets with varying subject-to-frame size ratios and a cropping calculation unit that determines the subject size in the frame, selecting and setting a cropping frame based on these data to adjust composition dynamically.

Benefits of technology

This approach ensures seamless adjustment of composition according to subject size changes, resulting in more natural and desirable images during automatic tracking framing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2026182295000001_ABST
    Figure 2026182295000001_ABST
Patent Text Reader

Abstract

This system enables cropping that changes composition depending on the size of the subject. [Solution] The imaging device includes a composition data storage unit that stores multiple composition data according to the ratio of the size of the subject to the size of the frame, and a cropping calculation unit that determines the size of the subject in the frame of the captured image data, sets a cropping frame based on the multiple composition data selected from the composition data storage unit according to the ratio of the size of the subject to the size of the frame of the captured image data, and performs cropping processing on the frame of the captured image data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to an imaging apparatus, an image processing method, and a program, and particularly relates to cutout processing for a captured image.

Background Art

[0002] Among various imaging apparatuses such as digital cameras and smartphones equipped with a shooting function, some have a function of cutting out a part of a captured image and recording the same. In particular, processing of tracking a target subject within a moving image frame and cutting out an image of the target subject is known. Patent Document 1 below discloses a technology for tracking a target subject.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As an automatic tracking framing function in an imaging apparatus, there is a function of cutting out and recording a range such as a bust shot from each frame of a captured moving image by allowing a user to pre-specify a composition such as a bust shot or a full-body shot. However, with this function, the same composition is applied even when the size of the subject in the image changes due to a change in the distance to the subject during imaging, which may result in an unintended or unnatural image for the user.

[0005] In view of this, the present technology has an object of obtaining an unnatural cutout image when tracking and cutting out a specific subject as in the automatic tracking framing function.

Means for Solving the Problems

[0006] The imaging device according to this technology includes a composition data storage unit that stores multiple composition data with different ratios between subject size and frame size, and a cropping calculation unit that determines the subject size in the frame of the captured image data, sets a cropping frame based on the multiple composition data selected from the composition data storage unit according to the ratio between the subject size and frame size of the captured image data, and performs cropping processing on the frame of the captured image data. In other words, multiple composition data sets are prepared according to the ratio of the subject size to the frame, and the cropping frame is set from the multiple composition data sets corresponding to the subject size and frame of the actual captured image data. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram of the imaging device according to an embodiment of this technology. [Figure 2] This is a block diagram of the cropping process configuration for automatic tracking framing according to the embodiment. [Figure 3] This is an explanatory diagram of the composition data table of the embodiment. [Figure 4] This is an explanatory diagram of the composition data table of the embodiment. [Figure 5] This is a flowchart of the extraction process in the embodiment. [Figure 6] This is a flowchart of the subject size determination process in the embodiment. [Figure 7] This is an explanatory diagram of the interpolation process and cropping frame setting for the composition data in the embodiment. [Modes for carrying out the invention]

[0008] The embodiments will be described below in the following order. <1. Imaging System Configuration> <2. Extraction Process Configuration> <3. Processing Example> <4. Summary and Variations>

[0009] In this disclosure, "image" includes both video and still images. While the embodiments primarily assume video capture, the technology of this disclosure can also be applied to still image capture.

[0010] <1. Imaging System Configuration> Figure 1 shows an example of the configuration of the imaging device 1 according to this embodiment. The imaging device 1 can be implemented in the form of a digital camera, or as a smartphone or tablet device equipped with imaging capabilities.

[0011] The imaging device 1 comprises a control unit 30, an imaging unit 31, a display unit 32, an operation unit 33, a sensor unit 34, an image processing unit 35, a storage unit 36, and a memory unit 37. Note that the configuration in Figure 1 is merely an example showing the essential parts, and it may also include components not shown, nor is it necessary to include all of the components shown.

[0012] The imaging unit 31 includes a lens unit, an image sensor that receives incident light corresponding to one or more lenses of the lens unit, and a signal processing circuit for the photoelectric conversion performed by the image sensor. The lens unit in the imaging unit 31 includes an optical system such as a zoom lens, a focus lens, and an aperture mechanism, as well as a drive mechanism for these optical elements. Light from the subject is guided by this optical system and focused onto the image sensor.

[0013] The image sensor in the imaging unit 31 is configured as, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type. The image sensor performs processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion of the received light, and further performs A / D (Analog / Digital) conversion processing. The captured image data as digital data is then output to the image processing unit 35.

[0014] The image processing unit 35 is configured as, for example, an image processing processor. The image processing unit 35 performs processing on captured image data that is captured by the imaging unit 31 and input thereto, such as Y / C development processing, codec processing, and file generation processing. Image data of moving images and still images output through the processing by the image processing unit 35 is stored in a storage medium by the storage unit 36. Note that, as will be described later, the image processing unit 35 also performs clipping processing as automatic tracking framing.

[0015] The display unit 32 includes a display panel and a display drive circuit therefor, and performs various types of display for a user. The display panel is configured by a liquid crystal display (LCD) panel, an organic electro-luminescence (EL) panel, or the like.

[0016] In accordance with an instruction from the control unit 30, the display unit 32 causes various types of display to be executed on a display screen. For example, the display unit 32 displays a preview image (monitor image) at the time of shooting, a reproduced image, or displays various icons, operators, and the like as a user interface.

[0017] The operation unit 35 collectively denotes input devices for a user to perform various operation inputs. Specifically, the operation unit 33 includes physical operators such as a shutter button, a touch panel that detects operations on operators (buttons, icons, sliders, etc.) displayed on the display unit 32, and the like. A user's operation is detected by the operation unit 33, and a signal corresponding to the input operation is sent to the control unit 30.

[0018] The sensor unit 34 denotes various sensors mounted on the imaging apparatus 1. As the sensor unit 34, for example, an IMU (inertial measurement unit) is mounted, and for example, angular velocities can be detected by three-axis angular velocity (gyro) sensors for pitch, yaw, and roll, and acceleration can be detected by an acceleration sensor. In addition, the sensor unit 34 may be equipped with an illuminance sensor. Furthermore, the sensor unit 34 may also be equipped with a distance measuring sensor, such as a TOF (Time Of Flight) sensor. Various types of information detected by the sensor unit 34, such as illuminance information, IMU data, and distance measurement data, are supplied to the control unit 30 and the image processing unit 35.

[0019] The memory unit 36 ​​performs processing to record image files (content files) such as still image data and video data, as well as attribute information of image files, thumbnail images, etc., on a recording medium such as a non-volatile memory. The actual configuration of the memory unit 36 ​​can be diverse. For example, the memory unit 36 ​​may function as a circuit unit that performs recording and playback on flash memory built into the imaging device 1, or as a card recording and playback circuit that performs recording and playback access on a memory card (e.g., portable flash memory) that can be attached to or removed from the imaging device 1. Furthermore, the memory unit 36 ​​may be implemented as an HDD (Hard Disk Drive) or SSD (Solid State Drive) built into the imaging device 1.

[0020] The control unit 30 is composed of a processor (arithmetic processing unit) such as a CPU (Central Processing Unit). The memory unit 37 stores information used by the control unit 30 for processing. This memory unit 37 may include, for example, ROM (Read Only Memory), RAM (Random Access Memory), or flash memory. The memory unit 37 may be a memory area built into the microcomputer chip that constitutes the control unit 30, or it may be composed of a separate memory chip.

[0021] The RAM in the memory unit 37 is used as a workspace for the CPU of the control unit 30 during various data processing operations, and is used for the temporary storage of data, programs, etc. The ROM and flash memory (non-volatile memory) in the memory section 37 are used to store the OS (Operating System) for the CPU of the control unit 30 to control each part, as well as application programs and various setting information for the processor of the control unit 30 or the image processing unit 35 to perform various operations.

[0022] The control unit 30 controls the entire imaging device 1 by executing programs stored in the ROM or flash memory of the memory unit 37. For example, the control unit 30 controls the optical system of the imaging unit 31, such as zoom, focus, and aperture adjustment; controls the shutter speed of the image sensor; performs signal processing on captured image data; performs imaging and recording operations in response to user operations; and plays back recorded image files.

[0023] Although the image processing unit 35 and the control unit 30 are shown as separate components in the diagram, they may be formed in a single processor.

[0024] <2. Extraction Process Configuration> Figure 2 shows the automatic tracking framing processing function in the image processing unit 35. Specifically, Figure 2 shows the configuration of the process that automatically tracks the subject in the captured image data and crops it with an appropriate composition to generate a recorded image.

[0025] For automatic tracking and framing, the image processing unit 35 includes a cropping calculation unit 10 and a buffer unit 17. The image processing unit 35 can also refer to a composition data storage unit 15 formed in internal or external memory.

[0026] The captured image data Idt is image data captured by the imaging unit 31 and processed in the image processing unit 35 after the preceding predetermined processing. As captured image data Idt, each frame of the video data during video capture becomes the target of processing by the extraction calculation unit 10. The buffer unit 17 buffers the captured image data Idt for each frame to synchronize processing timing, taking into account the delay caused by the processing of the cropping calculation unit 10. Note that the buffer unit 17 is not required, as a slight discrepancy between the cropping frame set by the cropping calculation unit 10 and the frame to be cropped is acceptable.

[0027] The cropping calculation unit 10 includes a size determination unit 11, a frame setting unit 12, and a cropping unit 14.

[0028] The size determination unit 11 performs a process to determine the size of the subject in the frame of the input image data Idt. The information that forms the basis of the subject size is the position and size of the face, the position of the human skeletal points, etc., obtained by human body detection technology, and the size determination unit 11 estimates the subject size from this information. The subject size determined by the size determination unit 11 is obtained for composition setting, that is, setting the cropping range, and does not require the precise area of ​​the pixel region of the subject person within the frame. It is sufficient to know roughly how much area the subject person occupies within the frame range. For this reason, the subject size can be determined as the area of ​​the body frame, the area of ​​the face frame, etc., as described later.

[0029] The frame setting unit 12 performs the process of setting the cropping frame. Specifically, the frame setting unit 12 selects one or more composition data from the composition data storage unit 15 according to the size ratio of the subject size determined by the size determination unit 11 and the frame frame size of the captured image data Idt, and performs the process of setting the cropping frame based on the selected one or two composition data.

[0030] The composition data storage unit 15 stores multiple composition data as a composition data table. Each of the multiple composition data is a composition data corresponding to the ratio of the subject size to the frame size. For example, the processor acting as the image processing unit 35 includes a composition data storage unit 15 that stores multiple composition data FDs in its internal memory. Alternatively, the composition data storage unit 15 may be provided in the memory unit 37, allowing the image processing unit 35 to sequentially access the composition data FDs in the memory unit 37.

[0031] Figures 3 and 4 show eight composition data entries stored in the composition data storage unit 15. Note that eight is merely an example for illustrative purposes. Figures 3 and 4 also show composition data for, for example, when a user holds the imaging device 1 horizontally and captures an image with a horizontal frame. In addition, composition data tables applicable to other orientations, such as "vertical orientation with a vertical frame," may also be stored in the composition data storage unit 15.

[0032] Figures 3 and 4 show eight composition data sets, from composition data FD1 to composition data FD8. For explanatory purposes, when composition data FD1 to composition data FD8 are not distinguished, they will be referred to as "composition data FD".

[0033] The composition data FD is data in which the size of the cropping frame 50 and the representative position within the frame are set for the frame frame 55, that is, the overall frame of one frame. The representative position can be any information that indicates a specific position within the cropping frame 50, for example, the central coordinates of the subject area or specific skeletal coordinates. The following explanation will use an example where the representative position is the central coordinates. For example, the cropping frame 50 set by the central coordinates and size will indicate the composition of the image cropped from the frame. Composition data FD1 to FD8 are data for cropping frames 50, each with different sizes and central coordinates, as shown in the diagram. These cropping frames 50 are pre-configured with appropriate compositions, sizes, and central coordinates based on the ratio of subject size to frame size. In other words, each composition data FD is data in which a cropping frame 50 is set according to the size of the subject (subject size) within the frame 55.

[0034] Composition data FD1 through FD8 represent data where the size of the subject within the frame gradually increases. Composition data FD1 is set to a cropping frame of 50 that includes the entire body of the subject, in the smallest possible case. The composition data FD2 also includes the entire body of the subject, and a cropping frame of 50 is set, with the frame size slightly enlarged according to the size of the subject. In composition data FD3 and FD4, a cropping frame of 50 is set, which includes the area from the knees or waist to the top of the head, depending on the size of the subject. In composition data FD5, FD6, and FD7, the cropping frame of the bust-up area is set to 50, depending on the size of the subject. The composition data FD8 has a cropping frame of 50 set to primarily crop the face area, depending on the size of the subject.

[0035] As described above, each composition data FD has a cropping frame 50 set to produce an appropriate composition according to the ratio of the subject size to the size of the frame 55, that is, the proportion of the subject that occupies within the frame 55. The size of the cropping frame 50 refers to, for example, its size in the x and y coordinate directions, while the center coordinate is the coordinate value where the center point of the cropping frame 50 is located within the x and y coordinates of the frame.

[0036] The composition data FD1, FD2, FD3, FD4, and FD5 use the area of ​​the body frame 51 as the subject size, and the size and central coordinates of the cropping frame 50 are set according to the ratio of the area of ​​the frame 55 to the area of ​​the body frame 51. The body frame 51 is a frame set from the skeletal points of the body, etc.

[0037] Composition data FD6, FD7, and FD8 are used when the body frame 51 cannot be determined, and the area of ​​the face frame 52 is used as the subject size. The size of the cropping frame 50 and the central coordinates are set according to the ratio of the area of ​​the frame 55 to the area of ​​the face frame 52.

[0038] As described above, multiple composition data FDs are stored in the composition data storage unit 15. From among these composition data FDs, the frame setting unit 12 selects one or two composition data FDs according to the ratio of the subject size to the frame size in the current image data Idt.

[0039] The size determination unit 11 determines that the subject size in the frame of the current captured image data Idt is also the area of ​​the body frame 51 and face frame 52 mentioned above. Therefore, the frame setting unit 12 calculates the ratio of the subject size to the frame size in the frame of the current captured image data Idt and selects the composition data FD that matches the ratio.

[0040] However, in reality, it is rare for any of the eight composition data FDs to have matching ratios, and in most cases, the frame setting unit 12 selects two composition data FDs with similar ratios. By performing linear interpolation using the two composition data FDs, composition data (hereinafter referred to as "interpolated composition data FDc") suitable for the subject size in the frame of the current captured image data Idt can be generated.

[0041] Composition data FD, or interpolated composition data FDc, with matching ratios in this way, contains information about a cropping frame 50 that is adapted to the subject size in the frame of the current image data Idt. In other words, it contains information about the size and central coordinates of the cropping frame 50 adapted to the subject size in the frame of the current image data Idt.

[0042] Based on the size and central coordinate information of the cropping frame 50, the frame setting unit 12 generates information for the cropping frame 50c that will actually be used for cropping the frame of the current captured image data Idt, and provides it to the cropping unit 14. The extraction unit 14 reads frames corresponding to the information of the supplied extraction frame 50c from the buffer unit 17 and performs extraction processing on the frame to be processed using the information of the supplied extraction frame 50c.

[0043] The cropping calculation unit 10 performs the above-described processing of the size determination unit 11, frame setting unit 12, and cropping unit 14 for each frame of the captured image data Idt. The resulting extracted image becomes the captured image data IdtF for an automatically tracking and framing video. This captured image data IdtF is then stored as a video file in, for example, the storage unit 36.

[0044] <3. Processing Example> Specific processing examples are explained in Figures 5, 6, and 7. Figure 5 shows an example of processing performed by the image processing unit 35 for each frame of the captured image data Idt, using the functions of the cropping calculation unit 10 described above.

[0045] In step S101, the image processing unit 35 determines the size of the subject in the currently input frame. For explanatory purposes, the current frame will be referred to as "frame FRc". An example of the process in step S101 is shown in Figure 6.

[0046] In step S120, the image processing unit 35 determines the area of ​​the subject person within frame FRc. In this case, it detects the position of the face and points of the human skeleton to determine the pixel area in which the subject person is captured.

[0047] In step S121, the image processing unit 35 branches the process depending on whether a valid body frame 51 can be determined within the frame frame 55. For example, by detecting points of the human skeleton, it is possible to determine which part of the person is captured in the image. For example, if the whole body or the area from the knees to the top of the head, which corresponds to the body frame 51, is captured, then the body frame 51 is obtained.

[0048] In this case, the image processing unit 35 proceeds to step S122 and sets the body frame 51 within frame FRc. The face frame 52 may also be set at the same time. Then, in step S123, the image processing unit 35 calculates the size (area) of the body frame 51 as the subject size.

[0049] For example, if the image processing unit 35 determines in step S121 that a body frame 51 cannot be obtained within frame FRc because only the chest to the top of the head of the person is captured in the image, the image processing unit 35 proceeds to step S124 to set a face frame 52 within frame FRc. Then, in step S125, the image processing unit 35 calculates the size (area) of the face frame 52 as the subject size.

[0050] After the processing shown in Figure 6, the image processing unit 35 proceeds to step S102 in Figure 5. In step S102, the image processing unit 35 calculates the ratio of the subject size to the frame size. This size ratio refers to either the ratio of the body frame 51 to the frame FRc, or the ratio of the face frame 52 to the frame FRc.

[0051] In step S103, the image processing unit 35 selects from multiple composition data FDs stored in the composition data storage unit 15 a composition data FD with a matching size ratio, or two composition data FDs with similar size ratios.

[0052] If a composition data FD with a matching aspect ratio exists and is selected, the image processing unit 35 proceeds to step S106 and obtains information about the cropping frame 50 in that composition data FD. That is, information about the size and central coordinates of the cropping frame 50.

[0053] On the other hand, if no composition data FD with a matching size ratio exists and two composition data FDs with similar size ratios are selected, the image processing unit 35 proceeds to step S105. The image processing unit 35 then generates interpolated composition data FDc by linear interpolation of the two composition data FDs and obtains information about the cropping frame 50 in the interpolated composition data FDc. That is, information about the size and central coordinates of the cropping frame 50.

[0054] This will be explained in Figure 7. Let's assume that composition data FD3 and FD4 were selected as composition data FDs with similar size ratios.

[0055] First, weights w1 for composition data FD3 and w2 for composition data FD4 are calculated from the difference in size ratios in the frame FRc being processed for each of the size ratios of composition data FD3 and composition data FD4. Then, by applying interpolation processing that reflects the above weights w1 and w2 to the size SZ3 of the cropping frame 50 of the composition data FD3 and the size SZ4 of the cropping frame 50 of the composition data FD4, the size SZc of the cropping frame 50 of the interpolated composition data FDc is obtained. Similarly, by applying interpolation processing that reflects the above weights w1 and w2 to the central coordinates 53(x3,y3) of the crop frame 50 in composition data FD3 and the central coordinates 53(x4,y4) of the crop frame 50 in composition data FD4, the central coordinates 53(xc,yc) of the crop frame 50 of the interpolated composition data FDc can be obtained. This allows the interpolated composition data FDc to be calculated as shown in the figure. In other words, the size and central coordinate information of the crop frame 50 as the interpolated composition data FDc can be obtained.

[0056] In step S107 of Figure 5, the image processing unit 35 sets the cropping frame 50c for the frame FRc based on the size and central coordinates of the cropping frame 50 as interpolated composition data FDc obtained in step S105, or the size and central coordinates of the cropping frame 50 obtained in step S106. The size and central coordinates obtained from the above interpolated composition data FDc or selected composition data FD are based on the pre-set position of the human subject and differ from the position of the human subject in the current frame FRc. In other words, there is a discrepancy between the coordinates of a specific position of the human subject and the central coordinate 53. A specific position of the human subject refers to, for example, the position of a particular body part that you want to be the center of the human subject. Therefore, as shown in Figure 7, the information for the cropping frame 50c is set by moving the central coordinate 53 according to the position of the person subject in frame FRc.

[0057] In step S108 of Figure 5, the image processing unit 35 performs cropping on frame FRc based on the information of the cropping frame 50c that matches frame FRc. This outputs the captured image data IdtF as frame FRc.

[0058] <4. Summary and Variations> According to the above embodiments, the following effects can be obtained.

[0059] The imaging device 1 of this embodiment includes a composition data storage unit 15 that stores multiple composition data FDs corresponding to the ratio of the subject size to the frame size, and a cropping calculation unit 10 in the image processing unit 35, etc. The cropping calculation unit 10 determines the subject size in the frame of the captured image data Idt, selects two composition data FDs from the composition data storage unit 15 according to the ratio of the subject size to the frame size of the captured image data Idt, sets a cropping frame 50c based on the multiple composition data FDs, and performs cropping processing on the frame of the captured image data Idt. This enables cropping that automatically adjusts the composition according to the size of the subject, resulting in more desirable automatically tracked and framed images. For example, even if the subject moves and its apparent size changes, the image smoothly transitions to the optimal composition. Also, if the subject becomes smaller and the frame size is relatively increased to maintain the resolution of the recorded image, the image changes to an optimal composition appropriate for the larger size. As a result, the video content extracted with automatic tracking framing is recorded with a more favorable composition for the user.

[0060] In this embodiment, an example was described in which the cropping frame 50c is set based on two composition data FDs, but this is just one example, and it is also possible to set the cropping frame 50c based on three or more composition data FDs. For example, by using spline interpolation to draw a cubic curve from four composition data FDs with similar size ratios, the inflection points are smoothly connected, making it suitable for application in this technology.

[0061] In this embodiment, the composition data FD is data that includes information on the size of the cropping frame 50 and its representative position (e.g., the center coordinates). Therefore, the central coordinates and size of the cropping frame 50 suitable for the currently processed frame FRc can be determined from multiple composition data FDs. This allows the cropping frame 50 to be set according to the subject size of each frame.

[0062] In one embodiment, the cropping calculation unit 10 selects multiple composition data FDs with a size ratio close to the ratio of the subject size to the frame size of the captured image data from the composition data FD stored in the composition data storage unit 15, and sets an interpolated composition data FDc including the representative position and size of the cropping frame 50 by linear interpolation using the multiple composition data FDs. Therefore, the ratio of the subject size to the frame size is used to generate interpolated composition data FDc from, for example, two composition data FDs that are close to the frame of the current captured image data. This interpolated composition data FDc is composition data corresponding to the size ratio of the current frame, that is, data containing information about the central coordinates and size of the cropping frame 50. By using interpolated composition data FDc in this way, it is not necessary to unnecessarily increase the number of composition data FDs stored in the composition data storage unit 15, and the size of the composition data table can be reduced. Alternatively, interpolation may be performed using three or more composition data FDs, including two composition data FDs with similar size ratios.

[0063] In one embodiment, the cropping calculation unit 10 sets a cropping frame 50c according to the difference between a representative position in the interpolated composition data FDc and a specific position coordinate of the subject in the frame of the captured image data, and then performs the cropping process. In other words, using the information about the central coordinates and size of the cropping frame 50 in the interpolated composition data FDc obtained through interpolation processing, which is suitable for the current frame, the cropping frame 50c is set to have central coordinates and size that match the position of the subject in the current frame. This allows the cropping process to be performed with a cropping frame 50c that is suitable for the position and size of the subject in the current frame.

[0064] In this embodiment, the captured image data Idt is a video, and the cropping calculation unit 10 determines the subject size for each frame, selects multiple composition data FDs from the composition data storage unit 15 according to the ratio of the subject size to the frame size of the captured image data Idt, sets the cropping frame 50c based on the multiple composition data FDs, and performs the cropping process. This enables automatic tracking framing, where the cropping frame 50c changes seamlessly according to the size and position of the subject.

[0065] In one embodiment, the composition data storage unit 15 is shown to store multiple composition data FDs, including a composition data FD that defines a cropping frame 50 including the entire body of the subject, a composition data FD that defines a cropping frame 50 that crops the area from the chest to the face of the subject, and a composition data FD that defines a cropping frame 50 that crops the area around the face of the subject. By preparing composition data FDs with different cropping ranges (full body, bust-up, face, etc.) depending on the size ratio of the subject to the frame size, selecting a composition data FD will automatically select a composition including the full body, a bust-up composition, a close-up of the face, etc. This means the composition type is automatically selected based on the subject size of the current frame.

[0066] In one embodiment, the cropping calculation unit 10 determines the size of the body frame 51 set for the subject person as the subject size in the frame of the captured image data Idt, and calculates the size ratio of the body frame 51 to the frame frame 55. For example, if a frame 51 containing the subject's body is available, the size ratio between the body frame 51 and the frame size 55 is determined. Based on this size ratio, the composition data FD is selected, allowing the composition data FD including the entire body to be applied.

[0067] In one embodiment, if the cropping calculation unit 10 cannot set a body frame 51 for the subject person in the frame of the captured image data Idt, it determines the size of the face frame 52 set for the subject person as the subject size and calculates the size ratio of the face frame 52 to the frame frame 55. If the subject's body frame 51 cannot be captured, composition data FD is selected according to the size ratio of the face frame 52 and the frame frame 55, allowing for the application of composition data FD for bust shots or close-ups of the face.

[0068] Although this explanation assumes video capture, the captured image data Idt may also be a preview image used for still image capture. Furthermore, the captured image data Idt may also be a still image. While we have explained automatic tracking and framing using an example with a person as the subject, this technology can also be applied to subjects other than humans, such as animals.

[0069] The program of this embodiment is a program that causes an information processing device such as a CPU or DSP, or a device including such a device, to perform processes like those shown in Figures 5 and 6.

[0070] In other words, the program of this embodiment determines the size of the subject in the frame of the captured image data Idt, sets a cropping frame 50c based on a plurality of composition data FDs selected from a plurality of composition data FDs with different ratios of subject size to frame size in the captured image data, according to the ratio of subject size to frame size in the captured image data, and causes the image processing unit 35 to execute the process of cropping the captured image data to the frame using the set cropping frame 50c.

[0071] The functionality of this embodiment is realized when such a program is implemented in the image processing unit of a digital camera or included in, for example, a camera application on a smartphone.

[0072] Such programs can be pre-recorded in storage media such as HDDs (hard disk drives) built into computer devices, or in ROM (remote-controlled memory) within microcomputers with CPUs. Alternatively, the data can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to installing such programs from removable storage media to personal computers, they can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.

[0073] Furthermore, such a program is suitable for providing the imaging device 1 of the embodiment to a wide range of users. For example, by downloading the program to mobile terminal devices such as smartphones and tablets, imaging devices, mobile phones, personal computers, game consoles, etc., these smartphones, etc., can be made to function as the imaging device 1 of this disclosure.

[0074] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0075] Furthermore, this technology can also be configured as follows. (1) A composition data storage unit that stores multiple composition data with different ratios of subject size to frame size, The system includes: a cropping calculation unit that determines the size of the subject in the frame of the captured image data, sets a cropping frame based on a plurality of composition data selected from the composition data storage unit according to the ratio of the size of the subject in the captured image data to the size of the frame, and performs cropping processing on the frame of the captured image data. Imaging device. (2) The aforementioned composition data includes information on the size of the cropping frame and its representative position. The imaging device described in (1) above. (3) The cropping calculation unit selects multiple composition data from the composition data stored in the composition data storage unit, where the ratio of the subject size to the frame size is close to the ratio of the subject size to the frame size of the captured image data. Using the selected multiple composition data, it sets interpolated composition data that includes the representative position and size of the cropping frame through linear interpolation. The imaging device described in (1) or (2) above. (4) The cropping calculation unit sets a cropping frame according to the difference between the representative position in the interpolated composition data and the specific position coordinates of the subject in the frame of the captured image data, and then performs the cropping process. The imaging device described in (3) above. (5) The aforementioned captured image data is video data, The cropping calculation unit determines the subject size for each frame, selects multiple composition data from the composition data storage unit according to the ratio of the subject size to the frame size of the captured image data, sets a cropping frame based on the multiple composition data, and performs the cropping process. An imaging device as described in any of (1) to (4) above. (6) The aforementioned composition data storage unit includes: Composition data consisting of a cropped frame that includes the entire body of the subject, Compositional data that uses a cropping frame to cut out the area from the chest to the face of the subject, and Composition data that uses a cropping frame to cut out the area around the subject's face, Multiple composition data, including this, are stored. An imaging device as described in any of (1) to (5) above. (7) The aforementioned extraction calculation unit, The size of the body frame set for the subject person is determined as the subject size in the frame of the captured image data, and the ratio of the size of the body frame to the size of the frame frame is determined. An imaging device as described in any of (1) to (6) above. (8) The aforementioned extraction calculation unit, If a body frame cannot be set for a subject in the frame of the captured image data, the size of the face frame set for the subject is determined as the subject size in the frame of the captured image data, and the ratio of the face frame size to the frame size is calculated. Determine the frame size ratio. The imaging device described in (7) above. (9) The image processing device, Determine the size of the subject in the frame of the captured image data, From among multiple composition data with different subject size and frame size ratios, the cropping frame is set based on the multiple composition data selected according to the subject size and frame size ratio of the captured image data. Perform cropping operations on the frame of the captured image data using the set cropping frame. Image processing methods. (10) Determine the size of the subject in the frame of the captured image data, From among multiple composition data with different subject size and frame size ratios, the cropping frame is set based on the multiple composition data selected according to the subject size and frame size ratio of the captured image data. The image processing device is instructed to perform the process of cropping the captured image data frame using the set cropping frame. program. [Explanation of Symbols]

[0076] 1. Imaging device 10 Extraction Calculation Unit 11 Size determination section 12 Frame setting section 13 Buffer section 14 Cutting section 15 Composition data storage unit 35 Image Processing Unit 37 Memory section 50,50c cutting frame 51 Body Frame 52 Face Frames 55 frame

Claims

1. A composition data storage unit that stores multiple composition data with different ratios of subject size to frame size, The system includes: a cropping calculation unit that determines the size of the subject in the frame of the captured image data, sets a cropping frame based on a plurality of composition data selected from the composition data storage unit according to the ratio of the size of the subject in the captured image data to the size of the frame, and performs cropping processing on the frame of the captured image data. Imaging device.

2. The aforementioned composition data includes information on the size of the cropping frame and its representative position. The imaging apparatus according to claim 1.

3. The cropping calculation unit selects multiple composition data from the composition data stored in the composition data storage unit, where the ratio of the subject size to the frame size is close to the ratio of the subject size to the frame size of the captured image data. Using the selected multiple composition data, it sets interpolated composition data that includes the representative position and size of the cropping frame through linear interpolation. The imaging apparatus according to claim 1.

4. The cropping calculation unit sets a cropping frame according to the difference between the representative position in the interpolated composition data and the specific position of the subject in the frame of the captured image data, and then performs the cropping process. The imaging device according to claim 3.

5. The aforementioned captured image data is video data, The cropping calculation unit determines the subject size for each frame, selects multiple composition data from the composition data storage unit according to the ratio of the subject size to the frame size of the captured image data, sets a cropping frame based on the multiple composition data, and performs the cropping process. The imaging apparatus according to claim 1.

6. The aforementioned composition data storage unit includes: Composition data with a cropped frame that includes the entire body of the subject, Compositional data that uses a cropping frame to cut out the area from the chest to the face of the subject, and Composition data that uses a cropping frame to cut out the area around the subject's face, Multiple composition data, including this, are stored. The imaging apparatus according to claim 1.

7. The aforementioned extraction calculation unit, The size of the body frame set for the subject person is determined as the subject size in the frame of the captured image data, and the ratio of the size of the body frame to the size of the frame frame is determined. The imaging apparatus according to claim 1.

8. The aforementioned extraction calculation unit, If a body frame cannot be set for a subject in the frame of the captured image data, the size of the face frame set for the subject is determined as the subject size in the frame of the captured image data, and the ratio of the face frame size to the frame size is calculated. Determine the frame size ratio. The imaging apparatus according to claim 7.

9. The image processing device, Determine the size of the subject in the frame of the captured image data, From among multiple composition data with different subject size and frame size ratios, the cropping frame is set based on the multiple composition data selected according to the subject size and frame size ratio of the captured image data. Perform cropping operations on the frame of the captured image data using the set cropping frame. Image processing methods.

10. Determine the size of the subject in the frame of the captured image data, From among multiple composition data with different subject size and frame size ratios, the cropping frame is set based on the multiple composition data selected according to the subject size and frame size ratio of the captured image data. The image processing device is instructed to perform the process of cropping the captured image data frame using the set cropping frame. program.

Citation Information

Patent Citations

  • Apparatus and method for tracking object

    JP2007328746A