Imaging device, imaging system, and control method

Abstract

When generating multiple videos with different extraction areas from a single video, it will be possible to perform image blur correction on each video. [Solution] The system includes setting means for setting information for generating first and second videos with different regions extracted from a reference video acquired using an imaging sensor; acquisition means for acquiring motion information while the reference video is being filmed; image stabilization amount calculation means for calculating first and second image stabilization amounts corresponding to the first and second videos, respectively, based on the motion information acquired by the acquisition means and the information for generating the first and second videos; and recording means for recording the information for generating the first and second videos and the first and second image stabilization amounts on a recording medium in relation to the reference video.

Description

Technical Field

[0001] The present invention relates to an imaging apparatus.

Background Art

[0002] Some imaging apparatuses such as cameras are equipped with an image blur correction function. There are various systems for this image blur correction function, and an example thereof is electronic image blur correction. The electronic image blur correction is a system that corrects positional deviation between a plurality of temporally consecutive images (between frames) by changing the position at which a partial region is read out from an image obtained using an imaging sensor, in accordance with shake of the camera. Further, the electronic image blur correction can be executed either before or after recording a captured image on a recording medium.

[0003] Patent Document 1 proposes a technology in which, when a video camera apparatus shoots a moving image, information related to shake is added to the moving image and recorded, and a video playback apparatus cuts out a partial region of the moving image based on the shake information recorded together with the moving image, so as to perform electronic image blur correction during moving image playback.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0005] However, Patent Document 1 does not give any consideration to the case of generating a plurality of moving images with different cut-out regions from one moving image.

[0006] The present invention has been made in view of the above-described problem, and an object thereof is to enable image blur correction for each of a plurality of moving images when generating the plurality of moving images with different cut-out regions from one moving image. [Means for solving the problem]

[0007] To achieve the above objective, the imaging apparatus according to the present invention comprises: an imaging sensor; setting means for setting information for generating first and second videos with different regions to be extracted from a reference video acquired using the imaging sensor; acquisition means for acquiring motion information when the reference video is being filmed; blur correction amount calculation means for calculating a blur correction amount based on the motion information acquired by the acquisition means; and recording means for recording the information for generating the first and second videos and the blur correction amount on a recording medium in relation to the reference video. The blur correction amount calculation means calculates first and second blur correction amounts corresponding to the first and second videos, respectively, based on the motion information and the information for generating the first and second videos, and the recording means records the first and second blur correction amounts as the blur correction amounts on the recording medium in relation to the reference video. [Effects of the Invention]

[0008] According to the present invention, when generating multiple videos with different cropping regions from a single video, image blur correction can be performed on each video. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the configuration of the imaging device according to the first embodiment of the present invention. [Figure 2] A diagram showing the relationship between recorded video and cropped video. [Figure 3] Diagram explaining the process of reducing the amount of image stabilization. [Figure 4] Diagram explaining the process involved in calculating the amount of image stabilization. [Figure 5] This figure shows an example of an image generation device that applies blur correction processing to a video recorded by an imaging device according to the first embodiment of the present invention. [Figure 6] A diagram showing the configuration of the imaging device in the second embodiment of the present invention. [Figure 7]This figure shows the configuration of the imaging device and image generation device in the third embodiment of the present invention. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the figures.

[0011] [First Embodiment] Figure 1 shows the configuration of an imaging device according to a first embodiment of the present invention. The imaging device, camera 100, has a function that records captured video (hereinafter referred to as "captured video") and, after recording, generates a video (hereinafter referred to as "cropped video") using a portion of the image area of ​​the captured video, which is a reference video. For example, the photographer sets information about the cropped video in advance to camera 100, and camera 100 records information related to the cropped video settings as metadata. The image generation device, which is a playback device or playback software, generates the cropped video based on the settings recorded in the metadata. Camera 100 is described as a lens-interchangeable camera with a detachable lens unit 1000, but it may also be a lens-integrated camera with the lens unit 1000 as an integral part. The lens unit 1000 includes an image stabilization lens that can move in a direction perpendicular to the optical axis, and the image stabilization lens can be driven by a known method, and is equipped with an optical image stabilization function.

[0012] The imaging sensor 101 is a CCD sensor or CMOS sensor, which converts the optical image formed by the lens unit 1000 into an electrical charge, accumulates an electrical charge, and then reads out that charge to supply an image signal consisting of multiple pixels to the signal processing unit 102.

[0013] The signal processing unit 102 performs noise reduction, gain adjustment, and AD conversion on the image signal read from the imaging sensor 101 to convert it into digital data, and then performs image processing such as pixel interpolation and color conversion. The signal processing unit 102 then supplies (outputs) the processed image data to the display control unit 103, encoding unit 105, image memory 108, and motion vector detection unit 109. In the following, image data corresponding to video will be referred to as video data, and otherwise simply as image data. Also, if it can be either a still image or a video, it will be referred to as an image.

[0014] The display control unit 103 processes the display of images based on image data output from the signal processing unit 102, various setting menus, and information such as crop frames (described later) on the display device 104. The display device 104 consists of a display device such as a liquid crystal display (LCD) and a driver unit that controls the display device. Note that the display device is not limited to a liquid crystal display; for example, it may use organic electroluminescence (organic EL).

[0015] The encoding unit 105 converts the video data output from the signal processing unit 102 into video data in a predetermined format such as MPEG, according to a compression encoding standard.

[0016] The recording control unit 106 records the video data output from the encoding unit 105 onto the recording medium 107, associating it with metadata. The recording medium 111 is a recording medium built into the camera 100, such as a hard disk or solid-state drive, or a removable recording medium such as an SD memory card.

[0017] Motion vector detection unit 109 detects a motion vector of an image based on the image data of the current frame supplied from signal processing unit 102 and the image data of the past frame temporarily stored in image memory 108. Examples of motion vector detection methods include a feature point extraction method that searches for feature points such as image edges and compares images around the feature points, and a block matching method that divides an image into a plurality of blocks and calculates the motion amount of the image for each block. For motion vector detection, the frame immediately preceding the current frame among the past frames may be used, or frames preceding the current frame by two or more frames may be used. Note that the motion vector detected by the block matching method or the like indicates the amount of motion per unit time of the frame rate, and integration processing is performed on the motion vector to obtain motion vector data in the same unit as the angle information described later, which is used as motion information for image blur correction.

[0018] Angular velocity sensor 110 is a sensor that detects the motion of camera 100 as an angular velocity, and the detected angular velocity data is supplied to unit conversion unit 111. Note that unit conversion unit 111 can acquire the angular velocity data from angular velocity sensor 110 via serial communication, but may also be configured to acquire the angular velocity data via an A / D converter.

[0019] Unit conversion unit 111 executes integration processing for converting angular velocity to angle, and processing for converting the angle to a pixel unit. The conversion from angle to pixel is calculated by the following equation (1). θp is the angle converted to pixel units, θ is the angle in radians, f is the focal length, and d is the pixel pitch of imaging sensor 101. θp=f×tan(θ) / d ···(1) The angle data converted to pixel units is output to motion information selection unit 112. As described above, the angle data converted to pixel units by equation (1) and the motion vector data can be handled as data of the same unit.

[0020] The motion information selection unit 112 selects motion information to output from the motion vector data supplied (output) from the motion vector detection unit 109 and the angle data supplied from the unit conversion unit 111. As an example of a method for selecting motion information to output, the determination condition is whether or not the lens unit 1000 attached to the camera 100 is capable of optical image stabilization. Whether or not the lens unit 1000 is capable of optical image stabilization may be determined based on whether or not the lens unit 1000 includes an image stabilization lens. Alternatively, it may be determined based on whether or not the lens unit 100 is configured to perform optical image stabilization with the image stabilization lens. Information regarding whether or not the lens unit 100 is capable of optical image stabilization may be obtained by the motion information selection unit 112 through communication between the camera 100 and the lens unit 1000 in a known manner. Alternatively, the motion information selection unit 112 may obtain this information based on user operation via the operation unit 116. If the lens unit 1000 is capable of optical image blur correction, the motion vector data output from the motion vector detection unit 109 is selected; if the lens unit 1000 is not capable of optical image blur correction, the angle data output from the unit conversion unit 111 is selected. The reason for this is explained below.

[0021] If the lens unit does not have an optical image stabilization function, image blur will occur in the video due to the vibration applied to the camera 100. In other words, the amount of image blur in the video at this time can be said to be equivalent to the value obtained by converting the angular velocity detected by the angular velocity sensor 110 into pixel-level angle information. On the other hand, if the lens has an optical image stabilization function, there will be a discrepancy between the amount of camera vibration and the amount of image blur due to the correction made by the optical image stabilization. In other words, the angular velocity sensor 110 will not be able to detect the amount of image blur. Therefore, when performing optical image stabilization, it is necessary to detect the image blur that occurs in the video using motion vectors. However, if the camera 100 can acquire the amount of correction performed by optical image stabilization, it can be said that the value obtained by subtracting the amount of blur correction performed by optical image stabilization from the information of the unit conversion unit 111 will be equivalent to the amount of image blur. Therefore, the difference between the output of the unit conversion unit 111 and the correction amount of optical image stabilization may be used as the output of the motion information selection unit 112.

[0022] The crop information acquisition unit 113 acquires crop information set by the user operating the operation unit 116 and supplies it to the blur correction amount calculation unit 114. The operation unit 116 has at least one of known operating members such as a button, dial, or touch sensor.

[0023] Here, we will explain the settings related to cropping (image extraction) of camera 100 using Figure 2. Figure 2 is a diagram showing the relationship between the recorded video and the cropped video.

[0024] For example, let's say the maximum video resolution that camera 100 can shoot is 6000 x 4000 pixels. Then, from this video, a 3840 x 2160 pixel area, which is the resolution of 4K UHDTV commonly used in television broadcasting, is extracted to generate cropped video 1. Separately from 4K UHDTV, a 1080 x 1920 pixel area, which is the vertical FullHD resolution often used for posting on social networking services (SNS) for promotional purposes, is extracted to generate cropped video 2.

[0025] Crop video 1 and crop video 2 have different amounts of remaining cropped footage (hereinafter referred to as "excess pixels") relative to the original recorded video. Electronic image stabilization uses excess pixels to correct positional shifts between frames, so the larger the area of ​​excess pixels, the greater the image stabilization that can be achieved. When both crop video 1 and crop video 2 are cropped from approximately the center, crop video 2 has a larger area of ​​excess pixels than crop video 1, allowing for greater image stabilization. In other words, when performing electronic image stabilization after image recording, it is preferable to add image stabilization information that takes excess pixels into account to each crop video.

[0026] Camera 100 allows the user to set the size and crop position of the cropped video before shooting the video via the control unit 116 using the settings menu displayed on the display device 104. The size and crop position of the cropped video correspond to the size and position of a portion of the video used in each frame of the cropped video.

[0027] In the example above, Size_X0 in Figure 2 is 6000 and Size_Y0 is 4000. Also, if the user sets the size of cropped video 1 as in the example above, Size_X1 will be 3840 and Size_Y1 will be 2160. Note that when setting the size of cropped video in the camera 100 settings menu, the user may set a numerical value, or the user may select from pre-defined items such as 4K UHDTV.

[0028] Regarding the method for setting the crop position, for example, as shown in Figure 2, the user may input the coordinates Pos_X1 and Pos_Y1 of the top left corner of the cropped video. Alternatively, if the display device 104 has a touch sensor, the top left corner coordinates of the cropped video may be set based on the position where the user touches the display device 104. In addition, when the size of the cropped video is set, a crop frame corresponding to the set size may be displayed on the display device 104, and the user may set the crop position by moving the position of the crop frame. The method of moving the position of the crop frame may be linked to user operation via the operation unit 116 or to touch operation on the display device 104. For cropped video 2, similar to cropped video 1, if the user sets the size of cropped video 2 as in the example above, a crop frame corresponding to 1080 x 1920 pixels will be displayed on the display device 104, and the crop position can be set according to the user's operation.

[0029] As described above, the crop information obtained from the crop information acquisition unit 113 includes information regarding the size and position of the cropped video. The information regarding the size of the cropped video corresponds to Size_X1, Size_Y1, Size_X2, and Size_Y2 in Figure 2, and the information regarding the position of the cropped video corresponds to Pos_X1, Pos_Y1, Pos_X2, and Pos_Y2 in Figure 2.

[0030] The image stabilization amount calculation unit 114 calculates the image stabilization amount based on the motion information supplied from the motion information selection unit 112 and the information supplied from the crop information acquisition unit 114.

[0031] As mentioned above, the smaller the size of the cropped area, the larger the area that can be used as surplus pixels, so image blur correction can be performed even for large shakes. The blur correction amount calculation unit 114 performs blur correction amount calculation processing that includes processing to reduce the blur correction amount as the size of the cropped area increases.

[0032] The process of reducing the amount of image blur by the image blur calculation unit 114 will be explained in detail below using Figure 3. In Figure 3, the horizontal axis shows the temporary image blur amount, and the vertical axis shows the cutoff frequency of the HPF (high-pass filter) for reducing the image blur amount. The temporary image blur amount is the value obtained by inverting the polarity of the output of the motion information selection unit 112. For example, if the amount of blur per unit time indicated by the motion information corresponds to +5 pixels on the X-axis coordinate, the temporary image blur amount will be -5 pixels. The HPF can output the temporary image blur amount while blocking the low-frequency components included in the temporary image blur amount, and by blocking the low-frequency components, it becomes more difficult to reach the correction limit during image blur correction. For example, the following method can be used to change the cutoff frequency of the HPF. The image blur calculation unit 114 allows you to set three HPF cutoff frequencies in advance. When the temporary image blur amount is smaller than the first threshold, the cutoff frequency FcMin is set, and as the temporary image blur amount increases, the cutoff frequency is set to a higher frequency value. Furthermore, the gradient for changing the cutoff frequency (the relationship between the change in the cutoff frequency and the change in the temporary image stabilization amount) is also predetermined. For temporary image stabilization amounts where the cutoff frequency is between FcMin and FcMid, the cutoff frequency is set based on the LowSlope gradient. For temporary image stabilization amounts where the cutoff frequency is between FcMid and FcMax, the cutoff frequency is set based on the HighSlope gradient. Note that the cutoff frequency increases in the order of FcMin, FcMid, and FcMax, and the HighSlope gradient is steeper than the LowSlope gradient.

[0033] Figure 3 shows the transition of the cutoff frequency in cropped video 1 in Fc graph 1. The first threshold ThLow1 is the threshold for whether or not the cutoff frequency should be greater than FcMin, and the second threshold ThHigh1 is the threshold for whether or not the cutoff frequency should be greater than FcMid. In other words, the first threshold ThLow1 is the temporary shake correction amount that is at the lower end of the slope LowSlope, and the second threshold ThHigh1 is the temporary shake correction amount that is at the lower end of the slope LowSlope. Here, the maximum value of the cutoff frequency is FcMax, and the cutoff frequency is maintained at FcMax if the temporary shake correction amount is greater than or equal to a certain value. When the temporary shake correction amount is between the first threshold ThLow1 and the second threshold ThHigh1, the cutoff frequency Fc is determined by the following equation (2). Fc = SlopeLow × (Temporary shake correction amount - ThLow1) + FcMin ... (2) Furthermore, when the amount of provisional image stabilization exceeds the second threshold ThHigh1, the cutoff frequency Fc is determined by the following equation (3). Fc = SlopeHigh × (Temporary shake correction amount - ThHigh1) + FcMid ... (3) Furthermore, in Figure 3, the transition of the cutoff frequency in cropped video 2 is shown in Fc Graph 2. In Fc Graph 2, the reference value for the cutoff frequency is the same as in Fc Graph 1 for cropped video 1, and the slopes of the HighSlope and LowSlope gradients are also the same as in Fc Graph 1. Fc Graph 1 and Fc Graph 2 differ in that the first threshold value and the second threshold value are different. In Fc Graph 2, the first threshold is ThLow2 and the second threshold is ThHigh2. The first thresholds, ThLow1 and ThLow2, and the second thresholds, ThHigh1 and ThHigh2, are calculated based on the surplus pixels that can be used for image blur correction. When the center coordinate position of cropped video 1 coincides with the center coordinate position of the captured video, the surplus pixels in cropped video 1 are 2160 pixels in the horizontal direction, which is obtained by subtracting the Siza_X1 of cropped video 1 (3840 pixels) from the 6000 pixels of the captured video. The surplus pixels on the right outer and left outer edges are 1080 pixels each. Similarly, if the center coordinate position of cropped video 2 coincides with the center coordinate position of the captured video, the extra pixels in cropped video 2 are 2460 pixels each on the right outer and left outer sides in the horizontal direction. If the temporary image stabilization amount that sets the HPF cutoff frequency to FcMax is the maximum stabilization amount (for example, a value 100 pixels smaller than the extra pixels), then cropped video 1 will have its cutoff frequency set to FcMax when the temporary image stabilization amount is 980 pixels or more. Then, cropped video 2 will have its cutoff frequency set to FcMax when the temporary image stabilization amount is 2060 pixels or more. In other words, the larger the size and position of the cropped area, and the more extra pixels there are, allowing for image stabilization for larger movements, the weaker the effect of the process that reduces the amount of image stabilization. In this embodiment, in order to simplify calculations, the gradient for changing the cutoff frequency is fixed and the threshold for changing the cutoff frequency from FcMin is changed. However, it is also possible to fix the threshold for changing the cutoff frequency from FcMin and change the gradient for changing the cutoff frequency according to the amount of extra pixels. For example, the amount of temporary image stabilization that results in FcMax can be increased by making the gradient of the cutoff frequency change gentler as the size and position of the cropped area with many excess pixels increases.Furthermore, when changing the cutoff frequency, the smallest value among the upper outer, lower outer, right outer, and left outer pixels should be used as the reference for the excess pixels. Also, if the cutoff frequency increases as the amount of temporary blur correction increases, instead of changing the cutoff frequency according to the slope of a straight line, the cutoff frequency may be changed in steps, or it may be changed according to a curve.

[0034] The image stabilization amount calculated by the image stabilization amount calculation unit 114 is supplied to the display control unit 103. As described above, the display control unit 103 displays frames corresponding to cropped video 1 and cropped video 2 on the recorded video, as shown in Figure 2, and moves the display position of the frames based on the image stabilization amount. Specifically, the initial position of the area to be cropped is determined according to the user's operation, but the display position of the frames is moved to correspond when the area to be cropped is moved from the initial position to cancel out image blur. By displaying frames corresponding to the cropped video in this way, the user can recognize the field of view when the cropped video is generated by a playback device after the recorded video is filmed, even at the time of filming.

[0035] The image stabilization amount calculated by the image stabilization amount calculation unit 114 is also supplied to the metadata generation unit 115. The metadata generation unit 115 outputs the supplied image stabilization amount to the recording control unit 106. The information output from the metadata generation unit 115 includes not only image stabilization-related information such as the image stabilization amount, but also crop information such as the size and position of the cropping area set by the user. The information output from the metadata generation unit 115 also includes camera-related data, which is information about shooting conditions such as focal length and exposure settings. This information is recorded on the recording medium 107 by the recording control unit 106 in association with the video data output from the encoding unit 105 as metadata for the recorded video.

[0036] Next, the processing of the image stabilization amount calculation unit 114 will be explained using Figure 4. Figure 4 is a diagram illustrating the processing related to the calculation of the image stabilization amount in the image stabilization amount calculation unit 114. This processing is performed for each frame of the captured video, and the motion vector detection processing in the motion vector detection unit 109 and the integration and pixel conversion processing of angular velocity data in the unit conversion unit 111 are also performed for each frame. The motion vector detection processing and pixel conversion processing may be performed at a speed (short period) faster than the frame rate of the captured video.

[0037] In step S101, the blur correction amount calculation unit 114 acquires motion information output from the motion information selection unit 112.

[0038] In step S102, the blur correction amount calculation unit 114 sets the target crop number, which is used to identify the crop video to be calculated, to its initial value of 1. The process is then repeated for each crop video. In this embodiment, the example will be explained using the generation of two crop videos, crop video 1 and crop video 2.

[0039] In step S103, the image stabilization amount calculation unit 114 compares the current target crop number with the set total number of cropped videos. If the target crop number is less than or equal to the total number of cropped videos, the process proceeds to step S104. If the target crop number exceeds the total number of cropped videos, this process is terminated. In this embodiment, since the total number of cropped videos is 2, the process from step S104 onwards is performed for cropped video 1 (target crop number 1) and cropped video 2 (target crop number 2), and this process is terminated when the target crop number becomes 3.

[0040] In step S104, the blur correction amount calculation unit 114 acquires the crop information output from the crop information acquisition unit 113. For example, when the target crop number is 1, Size_X1, Size_Y1, Pos_X1, and Pos_Y1 in Figure 2 are acquired as crop information.

[0041] In step S105, the image stabilization amount calculation unit 114 calculates the excess pixels. As described above, the calculation method is based on the difference between the size of the captured video and the size of the cropping area. However, it is not necessary to calculate the excess pixels every time this process is executed. The calculation of excess pixels may be performed when the user changes the crop information settings or when this process is executed in response to the first frame of the captured video. The calculation result may then be stored in a predetermined memory, and in step S105, the value of the excess pixels that was finally calculated (stored in memory) may be read out.

[0042] In step S106, the image stabilization amount calculation unit 114 sets the cutoff frequency of the HPF.

[0043] In step S107, the image stabilization amount calculation unit 114 performs image stabilization amount calculation processing. As described above, the image stabilization amount calculation unit 114 calculates a provisional image stabilization amount based on the motion information acquired in step S101, and then applies the HPF with the cutoff frequency set in step S106 to the calculated provisional image stabilization amount to calculate the image stabilization amount.

[0044] In step S108, the image stabilization amount calculation unit 114 outputs the image stabilization amount calculated in step S107 to the display control unit 103 and the metadata generation unit 115.

[0045] In step S109, the image stabilization amount calculation unit 114 increments the target crop number and proceeds to step S103. From there, the processes in steps S104 to S109 are executed until the target crop number exceeds the total number of cropped videos in step S103.

[0046] Through the processes described so far, an appropriate amount of image stabilization is calculated for each set crop video. Based on this amount of image stabilization, the size and position of the crop video displayed are changed, and the display position of the frame corresponding to the crop video is changed, and the crop information is recorded as metadata.

[0047] Next, using Figure 5, we will describe the configuration and operation of the playback device 200, which is an image generation device that generates cropped video 1 and cropped video 2. Note that it does not have to be a device dedicated to video playback, but may be software that runs on a general-purpose computer. Also, in the imaging system including the camera 100 and playback device 200 according to this embodiment, the camera 100 and playback device 200 may be mounted in separate housings, or the camera 100 and playback device 200 may be mounted in a single housing.

[0048] The recording medium 201 is assumed to contain video data to which metadata has been added by the camera 100. The recording control unit 202 reads the video data recorded on the recording medium 201 and supplies it to the decoding unit 203. The decoding unit 203 decodes the video data of various formats read from the recording medium 201. The metadata analysis unit 204 reads the metadata associated with the video data output from the recording control unit 202 and outputs information related to image blur correction and cropping information. The image deformation unit 205 generates cropped video 1 and cropped video 2 based on the information related to image blur correction and cropping information output from the metadata analysis unit 204. Specifically, the image deformation unit 205 sequentially cuts out images of a predetermined size from a predetermined cutting position based on the information related to image blur correction and cropping information for each of the multiple frame images that make up the decoded video data. The image deformation unit 205 outputs the thus generated cropped video 1 or cropped video 2 to the display device 206 via a cable or the like. The display device 206 includes an LCD or organic EL and displays the image acquired from the image deformation unit 205. In Figure 5, the playback device 200 is shown connected to the display device 206 via a cable or the like, but the playback device 200 and the display device 206 may be integrated into one unit. Alternatively, the playback device 200 may record the cropped video 1 or cropped video 2 it generates onto a recording medium, and the recording medium may be connected to the display device 206 to display the cropped video 1 or cropped video 2 on the display device 206.

[0049] As described above, in this embodiment, by adding the amount of image blur correction calculated according to the settings of the cropped video to the metadata for each cropped video, it is possible to perform image blur correction on each video when generating multiple videos with different cropping regions from a single video.

[0050] [Second Embodiment] Next, a second embodiment of the present invention will be described. Figure 6 is a diagram showing the configuration of the imaging device of the second embodiment of the present invention, and parts that are the same as those of the camera 100 of the first embodiment are given the same reference numerals as in Figure 1, and detailed explanations are omitted. The camera 300 is capable of selecting whether or not to perform image blur correction and subject tracking for each cropped video. For example, in Figure 2, image blur correction is performed for cropped video 1, and for cropped video 2, subject tracking is performed so that even if the person moves within the filmed video, the person remains in the center of the cropped video, with the aim of generating a video that extracts only a specific person. Subject tracking is achieved by changing the position from which the image is cut out from the filmed video according to the movement of the specific person.

[0051] The person detection unit 301 detects people from image data output from the signal processing unit 102. The person detection method can be any known method, such as face detection, head detection, torso detection, or skeleton detection. Furthermore, when multiple people are detected, the person detection unit 301 can identify each person based on information about their feature points. For example, it can detect feature points such as eyes, mouth, and ears from the image data and identify each person based on the arrangement of these feature points. While this embodiment describes a configuration for detecting people, the type of specific subject to be detected may include things other than people. For example, animals such as dogs, cats, horses, and birds may be used as detection targets, or vehicles such as cars and airplanes may be used. The method for detecting these targets is not particularly limited, and any known method, such as a method using deep learning technology, may be used.

[0052] The tracking target determination unit 302 determines the subject to be tracked from the recorded video based on the detection results of the person detection unit. Methods for determining the tracking target include, for example, linking it to user operation via the operation unit 116 or to touch operation on the display device 104. The person movement amount calculation unit 303 calculates the movement amount of the person who has been selected as the tracking target. The person's movement amount is the amount of change in the position of the subject being tracked within the recorded video from the time the tracking target was determined. Based on this movement amount, the crop setting determination unit 304 determines the crop position. As mentioned above, since the surplus pixels differ for each cropped video, the trackable amount also depends on the crop information. When the movement amount calculated by the subject movement amount calculation unit 303 reaches the trackable amount, the crop position at the time the movement amount reached the trackable amount is fixed as the crop position for the period during which the movement amount exceeds the trackable amount. Furthermore, the crop position control described above makes it easier to resume subject tracking when the amount of movement exceeds the tracking limit and then returns to within the tracking limit. However, if the amount of movement exceeds the tracking limit for a predetermined period of time, the crop position may be set as the center of the recorded video.

[0053] Furthermore, the crop setting determination unit 304 also performs the same processing as the blur correction amount calculation unit 114 in the first embodiment. Therefore, the crop setting determination unit 304 outputs information for identifying the crop position according to whether or not image blur correction and subject tracking are performed for each crop video. Specifically, it outputs the blur correction amount corresponding to the crop video to be image blur corrected and the movement amount corresponding to the crop video to be subject-tracked. The display control unit 103 and the metadata generation unit 115 perform the same processing as in the first embodiment based on the information output from the crop setting determination unit 304.

[0054] As described above, in this embodiment, metadata is added to identify the crop position according to the settings for whether or not to perform image blur correction and subject tracking in the cropped video, for each cropped video. This makes it possible to perform image blur correction or subject tracking on each video when generating multiple videos with different cropping regions from a single video.

[0055] [Third Embodiment] Next, a third embodiment of the present invention will be described. Figure 7 shows the configuration of the imaging device and playback device of the third embodiment of the present invention. In the first and second embodiments, the camera recorded information as metadata for identifying the crop position determined based on the excess pixels of the cropped video.

[0056] In contrast, in this embodiment, the camera records information for identifying the crop position as metadata without considering the excess pixels of the cropped video, and the playback device determines the crop position considering the excess pixels based on the metadata. With this configuration, the load of calculating information for identifying the crop position is reduced, and even when shooting video at a high frame rate, information for identifying the crop position can be added to the metadata for each cropped video.

[0057] In Figure 7, parts similar to those of the camera 300 in the second embodiment are given the same reference numerals as in Figure 6, and detailed explanations are omitted.

[0058] The motion information selection unit 401 or the person movement amount calculation unit 402 outputs the calculation result to the metadata generation unit. In other words, the metadata generation unit 404 outputs the calculation result, which has not been corrected to take into account excess pixels, as metadata to the recording control unit 106. The metadata generation unit 404 also outputs the settings for whether or not to perform image blur correction and subject tracking in each cropped video as metadata.

[0059] The crop information acquisition unit 403 outputs information related to cropping to the display control unit 405. In this embodiment, a frame corresponding to the size and position of the cropping area set by the user is displayed on the display device 104, but the size and position of the frame do not change unless the user changes the settings.

[0060] In the playback device 500, the metadata analysis unit 501 analyzes the metadata output from the recording control unit 202. The analyzed data, including the blur correction amount calculated by the motion information selection unit 401, the movement amount calculated by the person movement amount calculation unit 402, the crop information output by the crop information acquisition unit 403, and the setting for whether or not to perform image blur correction and subject tracking, is output to the image deformation amount calculation unit 502.

[0061] The image deformation amount calculation unit 502 calculates the excess pixels for each cropped video based on the information output from the metadata analysis unit 501, and performs the process of reducing the blur correction amount described in the first embodiment and the process of fixing the crop position described in the second embodiment. The image deformation amount calculation unit 502 outputs the processing results to the image deformation unit 205, and the image deformation unit 205 generates a cropped video based on the acquired information.

[0062] As described above, in this embodiment, the camera records information for identifying the crop position as metadata without considering the excess pixels of the cropped video, and the playback device determines the crop position considering the excess pixels based on the metadata. With this configuration, the load of calculating information for identifying the crop position is reduced, and even when shooting video at a high frame rate, information for identifying the crop position can be added to the metadata for each cropped video. Note that the metadata recording method described in the first and second embodiments and the metadata recording method described in the third embodiment may be switched depending on the shooting settings. For example, when shooting video at a low frame rate below a predetermined threshold, the metadata recording method described in the first and second embodiments may be executed, and when shooting video at a high frame rate faster than the threshold, the metadata recording method described in the third embodiment may be executed. The predetermined threshold can be set to, for example, 60fps.

[0063] As described above, the present invention has been detailed based on its preferred embodiments, but the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of its gist. For example, an acceleration sensor, a geomagnetic sensor, or other sensor besides an angular velocity sensor may be used to detect camera motion. Also, in the process of calculating the amount of image stabilization, a method of changing the HPF cutoff based on the excess pixels of the cropped video has been shown, but any method that correlates the size of the excess pixel area with the size of the image stabilization amount may be used, other than the example shown for HPF.

[0064] Furthermore, although the components inside the camera are described separately in the embodiments described above, a CPU consisting of one or more microcomputers may function as one or more components according to a control program stored in memory. For example, one or more CPUs inside the camera may function as at least one of a signal processing unit, a display control unit, an encoding unit, a recording control unit, a motion vector detection unit, a unit conversion unit, a motion information selection unit, a crop information acquisition unit, a blur correction amount calculation unit, and a metadata generation unit. Alternatively, one or more CPUs inside the camera may function as at least one of a person detection unit, a tracking target determination unit, a person movement amount calculation unit, and a crop setting determination unit. Similarly, although the components inside the playback device are described separately, a CPU consisting of one or more microcomputers may function as one or more components according to a control program stored in memory. For example, one or more CPUs inside the playback device may function as at least one of a recording control unit, a decoding unit, a metadata analysis unit, an image deformation unit, and an image deformation amount calculation unit.

[0065] Furthermore, it can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0066] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention.

[0067] This embodiment includes the following configuration.

[0068] (Composition 1) Image sensor and A setting means for setting information for generating first and second videos with different regions extracted from a reference video acquired using the aforementioned imaging sensor, An acquisition means for acquiring motion information while the aforementioned reference video is being filmed, A blur correction amount calculation means that calculates a blur correction amount based on the motion information acquired by the acquisition means, The system includes recording means for recording information for generating the first and second videos and the amount of image stabilization on a recording medium in relation to the reference video, The image stabilization amount calculation means calculates first and second image stabilization amounts corresponding to the first and second videos, respectively, based on the motion information and the information for generating the first and second videos. The recording means is characterized by recording the first and second image stabilization amounts as image stabilization amounts on the recording medium in relation to the reference video.

[0069] (Configuration 2) The imaging apparatus according to configuration 1, characterized in that the setting means sets information for generating the first and second videos, including information regarding the resolution of the first and second videos and information regarding the position from which to extract the first and second videos from the reference video.

[0070] (Composition 3) The imaging apparatus according to configuration 1 or 2, characterized in that the image stabilization amount calculation means reduces the provisional image stabilization amount corresponding to the motion information based on the information for generating the first and second videos, and calculates the first and second image stabilization amounts.

[0071] (Composition 4) The imaging apparatus according to configuration 3, characterized in that the blur correction amount calculation means reduces the temporary blur correction amount by changing the cutoff frequency of the high-pass filter applied to the temporary blur correction amount to the high-frequency side.

[0072] (Composition 5) The imaging apparatus according to configuration 4, characterized in that the blur correction amount calculation means changes the cutoff frequency according to the magnitude of the provisional blur correction amount.

[0073] (Composition 6) The imaging apparatus according to configuration 4 or 5, characterized in that the blur correction amount calculation means sets the cutoff frequency differently when calculating the first blur correction amount and when calculating the second blur correction amount.

[0074] (Composition 7) The system includes a detection means for detecting a specific subject from the aforementioned reference video, The setting means sets, as information for generating the first and second videos, whether or not to perform image blur correction based on the first blur correction amount when generating the first video and whether or not to perform subject tracking based on the detection result of the specific subject. The imaging device according to any one of configurations 1 to 6, characterized in that, when the recording means is set to perform subject tracking, it records information regarding the position of the subject, which is the detection result of the detection means, in association with the reference video on the recording medium.

[0075] (Composition 8) The imaging apparatus according to any one of configurations 1 to 7, characterized in that the recording means records the first and second blur correction amounts as metadata of the reference video on the recording medium.

[0076] (Composition 9) An imaging system including an imaging device and an image generation device, Image sensor and A setting means for setting information for generating first and second videos with different regions extracted from a reference video acquired using the aforementioned imaging sensor, An acquisition means for acquiring motion information while the aforementioned reference video is being filmed, A blur correction amount calculation means that calculates a blur correction amount based on the motion information acquired by the acquisition means, A recording means for recording the information for generating the first and second videos and the amount of image stabilization on a recording medium in relation to the reference video, The system comprises generation means for generating the first and second videos, The image stabilization amount calculation means calculates first and second image stabilization amounts corresponding to the first and second videos, respectively, based on the motion information and the information for generating the first and second videos. The recording means records the first and second image stabilization amounts as image stabilization amounts in relation to the reference video on the recording medium. The imaging system is characterized in that the generation means generates the first and second videos based on the information for generating the first and second videos and the first and second image stabilization amounts recorded on the recording medium. [Explanation of symbols]

[0077] 100 Cameras 101 Imaging sensor 106 Recording Control Unit 109 Motion vector detection unit 110 Angular velocity sensor 113, 403 Crop Information Acquisition Unit 114 Image stabilization amount calculation unit 115, 404 Metadata generation unit

Claims

1. Image sensor and A setting means for setting information for generating first and second videos with different regions extracted from a reference video acquired using the aforementioned imaging sensor, An acquisition means for acquiring motion information while the aforementioned reference video is being filmed, A blur correction amount calculation means that calculates a blur correction amount based on the motion information acquired by the acquisition means, The system includes recording means for recording information for generating the first and second videos and the amount of image stabilization on a recording medium in relation to the reference video, The image stabilization amount calculation means calculates first and second image stabilization amounts corresponding to the first and second videos, respectively, based on the motion information and the information for generating the first and second videos. The recording means is characterized by recording the first and second image stabilization amounts as image stabilization amounts on the recording medium in relation to the reference video.

2. The imaging apparatus according to claim 1, wherein the setting means sets information for generating the first and second videos, including information regarding the resolution of the first and second videos and information regarding the position from which to extract the first and second videos from the reference video.

3. The imaging apparatus according to claim 1, characterized in that the image stabilization amount calculation means reduces the provisional image stabilization amount corresponding to the motion information based on the information for generating the first and second videos, and calculates the first and second image stabilization amounts.

4. The imaging apparatus according to claim 3, characterized in that the blur correction amount calculation means reduces the temporary blur correction amount by changing the cutoff frequency of the high-pass filter applied to the temporary blur correction amount to the high-frequency side.

5. The imaging apparatus according to claim 4, characterized in that the blur correction amount calculation means changes the cutoff frequency according to the magnitude of the provisional blur correction amount.

6. The imaging apparatus according to claim 4, characterized in that the blur correction amount calculation means sets the cutoff frequency different when calculating the first blur correction amount and when calculating the second blur correction amount.

7. The system includes a detection means for detecting a specific subject from the aforementioned reference video, The setting means sets, as information for generating the first and second videos, whether or not to perform image blur correction based on the first blur correction amount when generating the first video and whether or not to perform subject tracking based on the detection result of the specific subject. The imaging apparatus according to claim 1, characterized in that, when the recording means is set to perform subject tracking, it records information regarding the position of the subject, which is the detection result of the detection means, in association with the reference video on the recording medium.

8. The imaging apparatus according to claim 1, characterized in that the recording means records the first and second blur correction amounts as metadata of the reference video on the recording medium.

9. An imaging system including an imaging device and an image generation device, Image sensor and A setting means for setting information for generating first and second videos with different regions extracted from a reference video acquired using the aforementioned imaging sensor, An acquisition means for acquiring motion information while the aforementioned reference video is being filmed, A blur correction amount calculation means that calculates a blur correction amount based on the motion information acquired by the acquisition means, A recording means for recording the information for generating the first and second videos and the amount of image stabilization on a recording medium in relation to the reference video, The system comprises generation means for generating the first and second videos, The image stabilization amount calculation means calculates first and second image stabilization amounts corresponding to the first and second videos, respectively, based on the motion information and the information for generating the first and second videos. The recording means records the first and second image stabilization amounts as image stabilization amounts in relation to the reference video on the recording medium. The imaging system is characterized in that the generation means generates the first and second videos based on the information for generating the first and second videos and the first and second image stabilization amounts recorded on the recording medium.

10. A control method for an imaging device equipped with an imaging sensor, A setting step of setting information for generating first and second videos with different regions extracted from a reference video acquired using the aforementioned imaging sensor, The acquisition step involves obtaining motion information while the aforementioned reference video is being filmed, A step to calculate the amount of image stabilization based on the motion information acquired in the acquisition step, The system includes a recording step of recording the information for generating the first and second videos and the amount of image stabilization on a recording medium in relation to the reference video, In the step of calculating the amount of image stabilization, based on the motion information and the information for generating the first and second videos, the first and second amounts of image stabilization corresponding to the first and second videos are calculated, The control method is characterized in that, in the recording step, the first and second image stabilization amounts are recorded on the recording medium in relation to the reference video as the image stabilization amount.

Images