Imaging device

Abstract

To provide an imaging apparatus that by real-time image processing, obtains a high-resolution motion picture with a smaller angle of view from the motion picture with a larger angle of view .SOLUTION: An imaging apparatus comprises an imaging element for capturing an optical image of a reference video with a first angle of view, and a control unit that has a first mode and a second mode as motion picture photographing modes and stores motion picture data of a video based on the optical image. The control unit performs a first storing process of storing motion picture data of the reference video using the imaging element in the first mode. The control unit performs a setting process of placing a recording area with a second angle of view in the reference video, the second angle of view being smaller than the first angle of view, and performs a second storing process of storing motion picture data of a recorded video in the recording area. In the first storing process and the second storing process, thinned-out driving, in which a part of read-out of a plurality of pixels in the optical image is thinned out, is performed. A second thinning-out rate of the thinned-out driving in the second storing process is set to be lower than a first thinning-out rate of the thinned-out driving in the first storing process.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] One embodiment of the present invention relates to an imaging device that captures a moving image, and more particularly to an imaging device that processes a moving image to obtain a moving image with a small angle of view from the moving image in real time.

Background Art

[0002] Shooting a video is performed by a shooting device such as a digital camera, for example. Currently, due to the high pixel count of imaging elements such as CMOS (Complementary Metal Oxide Semiconductor) sensors, it is possible to process a video, which is a moving image captured by one imaging element, and digitally cut out a part of the angle of view to obtain a moving image with a small angle of view.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

[0004] An object of one embodiment of the present invention is to provide an imaging device that obtains a moving image with high resolution and a small angle of view from a moving image with a large angle of view by real-time image processing.

Means for Solving the Problems

[0005] In order to achieve the above object, an embodiment of the present invention includes an image sensor that captures an optical image of a reference video with a first angle of view and has a pixel region in which a plurality of pixels are arranged in a first direction and a second direction intersecting the first direction, and a control unit that includes a first mode and a second mode as video imaging modes and records video data of a video based on the optical image. The control unit executes a first recording process of recording video data of the reference video using the image sensor in the first mode. In the second mode, the control unit executes a setting process of setting a recording region, which is a second angle of view smaller than the first angle of view, within the reference video, and a second recording process of recording video data of the recorded video in the recording region. In the first recording process and the second recording process, the control unit executes a thinning drive for thinning out a part of the reading of a plurality of pixels among the optical images. The control unit provides an imaging device in which a second thinning rate in the thinning drive of the second recording process is set lower than a first thinning rate in the thinning drive of the first recording process.

[0006] In the second mode, it is preferable that the control unit changes the second thinning rate of the recording region according to the change of the second angle of view. Due to the change of the second thinning rate of the control unit, the image sensor changes the input resolution of the recorded video of the recording region to be input to the control unit. When the input resolution is higher than the output resolution of the video data, it is preferable that the control unit reduces the input resolution to match the output resolution. When the input resolution is lower than the output resolution, it is preferable that the control unit complements the input resolution to match the output resolution.

[0007] An imaging lens having a zoom function is provided. The image sensor receives light transmitted through the imaging lens and captures an optical image of a reference video with a first angle of view. When the input resolution is lower than the set resolution, it is preferable that the control unit notifies the user about the zoom function. In the second mode, the control unit executes a setting process of setting a plurality of video regions with a second angle of view within the reference video, and a selection process of selecting a recording region from the plurality of video regions. It is preferable that the second recording process combines the recorded video before selection and the recorded video after selection in the selection process to record the video data.

[0008] The pixel includes a photoelectric conversion element and a switch section, and a pixel circuit including a plurality of first wirings extending in a first direction and a plurality of second wirings extending in a second direction is provided. The first wiring and the second wiring are electrically connected to the switch section. By the control section supplying a drive signal to the switch section through the first wiring, a video signal based on the photoelectric conversion element is output to the second wiring according to the drive signal. The control section executes thinning driving to divide the pixel region in the first direction into a driving region composed of pixels to which a drive signal is supplied to the switch section through the first wiring and a non-driving region other than the driving region. In the second recording process, it is preferable that the control section records the recorded video using the driving region and associates the non-driving region with a video region other than the recording region.

[0009] In the second recording process, the control section moves the position of the recording region in the reference video or changes the second angle of view of the recording region in accordance with the movement of the subject within the recording region. It is preferable that the control section changes the position or width of the driving region within the pixel region in accordance with the movement of the position of the recording region or the change of the second angle of view. It is preferable that the control section determines the second thinning rate based on the smallest second angle of view among the second angles of view of the respective plurality of video regions.

[0010] The control section executes a detection process for detecting the movement of the subject within a plurality of video regions including the recording region. The control section executes the second recording process a plurality of times, executes the detection process between the plurality of second recording processes. In the detection process, it is preferable that the control section acquires a detection video of an angle of view including the recording region and at least one video region, and moves at least one of the plurality of video regions or changes the second angle of view in accordance with the movement of the subject.

[0011] A first image sensor that is an image sensor for imaging an optical image of a recorded video in a first recording process and a second recording process, and a second image sensor that is different from the first image sensor and acquires videos of a plurality of corresponding regions corresponding to a plurality of video regions including a recording region. The control unit executes a detection process for detecting the movement of a subject within the plurality of corresponding regions. In the detection process, the control unit acquires a detection video of an angle of view including the plurality of corresponding regions from the second image sensor, and it is preferable to move at least one of the plurality of video regions or change the angle of view along with the movement of the subject. Preferably, the moving image data does not include data representing the detection video. Preferably, the pixel resolution of the pixel region of the image sensor is 70 million or more.

[0012] An embodiment of the present invention includes an image sensor that images an optical image of a reference video at a first angle of view and has a pixel region in which a plurality of pixels are arranged in a first direction and a second direction intersecting the first direction, and a control unit that records moving image data of a video based on the optical image imaged by the image sensor. The control unit executes a setting process of setting a plurality of video regions having a second angle of view smaller than the first angle of view within the reference video, a determination process of determining a recording region from among the plurality of video regions, a reading process of reading a video signal of a pixel region corresponding to the recording region, and a recording process of recording the moving image data of the recorded video of the recording region. In the reading process, the control unit thins out the pixels in the first direction and the second direction and reads out the video signal of a partial region of the pixel region corresponding to the recording region. In the recording process, the control unit records the video of the partial region as a recording target, and provides an imaging device. Preferably, in the reading process, the control unit changes the thinning rate of the thinning drive that thins out a part of the reading of a plurality of pixels in the recording region due to the change in the second angle of view. Preferably, the pixel resolution of the pixel region of the image sensor is 70 million or more.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Embodiments for Carrying Out the Invention

[0014] Hereinafter, based on the preferred embodiments shown in the accompanying drawings, an imaging device according to an embodiment of the present invention will be described in detail. The figures described below are exemplary for explaining one embodiment of the present invention, and one embodiment of the present invention is not limited to the figures shown below. Note that, in the following, "~" indicating a numerical range includes the numerical values described on both sides. For example, when ε is a numerical value from numerical value α to numerical value β, the range of ε is a range including numerical value α and numerical value β, and in mathematical notation, it is α ≤ ε ≤ β. Angles such as "vertical" and "orthogonal" include the error range generally acceptable in the corresponding technical field unless otherwise specified. Also, "entire surface" etc. include the error range generally acceptable in the corresponding technical field.

[0015] <First Example of Imaging Device> FIG. 1 is a perspective view showing a first example of an imaging device according to an embodiment of the present invention, FIG. 2 is a rear view showing the rear side of the first example of the imaging device according to an embodiment of the present invention, and FIG. 3 is a block diagram showing the configuration of the first example of the imaging device according to an embodiment of the present invention.

[0016] [Basic Configuration of Imaging Device] The imaging device 10 is a digital camera and is used for video shooting. The imaging device 10 has a function of shooting still images and moving images. Also, the imaging device 10 has a function of displaying the shot image (video) and recording the video. Further, the imaging device 10 has a function of executing various signal processes such as image processing on the video which is a moving image in real time. Note that, in the following description, unless otherwise specified, "video" means a live video (live view image), that is, a video shot in real time.

[0017] The imaging device 10 shown in FIGS. 1 and 2 is an interchangeable-lens digital camera and includes an imaging device body 12 and a photographing lens 14. The photographing lens 14 is detachably attached to the mount 13 of the imaging device body 12. However, it is not limited thereto, and the imaging device 10 may be an integrated-lens digital camera.

[0018] (Photographing Lens) As shown in FIG. 3, the photographing lens 14 includes an optical component unit 18, a diaphragm 20, an electronic dimming filter 21, a focus driving unit 22, a diaphragm driving unit 23, and a voltage application unit 24. The optical component unit 18 has a plurality of lenses, including a focusing optical component 19 (focusing lens). When the focusing optical component 19 moves in the direction of the optical axis L1 (see FIG. 1), the focus (sharpness) of the photographing lens 14 changes.

[0019] The optical component unit 18 includes a wide-angle lens, an ultra-wide-angle lens, a 360-degree lens, an anamorphic lens, or the like. Thereby, the imaging device 10 can photograph an image with a wide horizontal angle of view. Here, the maximum angle of view when the imaging device 10 photographs an image is the first angle of view. Therefore, the first angle of view is determined according to the use of the optical component unit 18 and the imaging element 40 described later imaged with the maximum pixels of the imaging device 10. Note that the optical image photographed at the first angle of view as described above is the reference video A1 (see FIG. 4). Note that the imaging device 10 may include a plurality of optical component units 18 having different angles of view from each other.

[0020] The diaphragm 20 is disposed in the optical path of the photographing lens 14 and is configured such that the aperture shape is variable, and is an optical component that adjusts the diaphragm amount (specifically, the diaphragm value or F value) of the incident light to the photographing lens 14. In the present embodiment, the diaphragm 20 is one in which the aperture shape is changed by mechanically driving a member for changing the size of the aperture. However, the present invention is not limited to this, and a diaphragm (physical property diaphragm) in which the aperture shape is changed by driving a liquid crystal or an electrochromic element may be used. The aperture shape of the diaphragm 20, that is, the diaphragm amount, is adjusted by the diaphragm driving unit 23.

[0021] (Imaging device main body) As shown in FIGS. 1 and 2, the imaging device main body 12 includes an operation unit operated by a user. For example, a release button 26 is disposed on the upper surface of the imaging device main body 12. For example, when the user fully presses the release button 26, recording of an image captured by the imaging device 10 or an image based on the captured image is started. Examples of the image based on the captured image include a recorded image in a recording area described later. Note that the release instruction from the user may be input to the control unit 46 via the touch detection function of the display 28 described later.

[0022] On the back surface of the imaging device main body 12, a display 28 constituting a display screen is disposed, and a plurality of buttons are disposed around the display 28.

[0023] The display 28 is constituted by, for example, an LCD (Liquid Crystal Display), an organic EL (Organic Electroluminescence) display, an LED (Light Emitting Diode) display, or electronic paper. On the display screen of the display 28, an image captured by the imaging device 10 or an image based on the captured image is displayed. In addition, on the display screen of the display 28, a selection menu related to shooting conditions, notification information to the user including warnings, and a playback image of an image acquired in the past are also displayed.

[0024] The display 28 has a function of detecting a touch of a user's finger. In the present embodiment, a transmissive touch panel 36 is overlaid on the display 28. The touch panel 36 detects the contact position of a user's finger or a stylus pen and its displacement, and outputs a signal based on the detection result to a predetermined output destination. Note that the touch panel 36 may be incorporated inside the display 28.

[0025] As shown in FIG. 3, an optical shutter 38, an imaging element 40, an analog signal processing circuit 44, a control unit 46, an internal memory 50, a card slot 52, and a buffer 56 are provided in the housing of the imaging device main body 12.

[0026] The optical shutter 38 is, for example, a focal plane shutter disposed immediately in front of the imaging device 40, which is maintained in an open state during preview and is once closed immediately before the exposure operation for taking a still image is performed. Thereafter, when the front curtain and the rear curtain in the focal plane shutter travel to perform the exposure (shutter opening / closing operation), the optical shutter 38 is maintained in the open state again. Note that, in addition to the focal plane shutter, a mechanical shutter such as a diaphragm shutter or a liquid crystal shutter can also be used as the optical shutter 38.

[0027] The imaging device 40 captures the optical image of the reference video of the first angle of view and has an image sensor. Examples of the image sensor include solid-state imaging devices such as CCD (Charged Coupled Device) and CMOS. As shown in FIG. 4, the imaging device 40 has a pixel region 43 in which a plurality of unit pixels 42 are arranged in a first direction D1 and a second direction D2 intersecting the first direction D1. In the imaging device 40, each of the plurality of unit pixels 42 may have an on-chip microlens and a color filter. The pixel resolution of the pixel region 43 of the imaging device 40 is preferably 70 million or more. If the pixel resolution of the pixel region 43 is 70 million or more, the reference video becomes a high-quality video, and even if a plurality of video regions are set for one reference video, each recording region has sufficient resolution. Therefore, the visibility of the recording region obtained from the reference video is ensured. The upper limit of the pixel resolution is not particularly limited, and it is preferably 1 billion or less, more preferably 500 million or less. The detailed configuration of the imaging device 40 will be described later.

[0028] FIG. 4 shows the correspondence relationship between the plurality of unit pixels 42 of the imaging device 40 and the imaging region A0. The reference video A1 is the video reflected in the imaging region A0 imaged by the imaging device 10 at the first angle of view. The reference video A1 reflected in the imaging region A0 is composed of a plurality of unit pixels 42 and is composed of a maximum number of pixels equal to the pixel region 43. The plurality of unit pixels 42 respectively correspond to one of the unit areas Ad that constitute the reference video A1, as shown in FIG. 4. The unit area Ad is the smallest unit area in the reference video A1 and is composed of a plurality of pixels.

[0029] The imaging device 40 receives light from the subject that has passed through the imaging lens 14, forms an image, and converts the optical image into an electrical signal (pixel signal) for output. Also, the imaging device 40 is configured such that an exposure operation by a shutter provided in the imaging device can be performed. That is, with the shutter function, it is possible to perform photoelectric conversion and charge accumulation for a controlled time while keeping the optical shutter 38 in an open state. Also, the exposure time by the shutter is adjustable. The imaging device 10 can be configured to include, for example, an imaging lens (not shown) having a zoom function. When the imaging device 10 includes an imaging lens having a zoom function, the imaging device receives light that has passed through the imaging lens having a zoom function and images the optical image of the subject. That is, it images the optical image of the reference video with the first angle of view. In the following description, unless otherwise specified, "exposure" means performing exposure by the shutter in the imaging device 40 while keeping the optical shutter 38 in an open state. Also, the "exposure time" refers to the exposure time by the shutter and is strictly the charge accumulation time.

[0030] In FIG. 3, the analog signal processing circuit 44 converts the analog electrical signal (pixel signal) output from the imaging device 40 into a digital signal, and the processed signal is sent to the video processing unit 48 of the control unit 46. The converted digital signal is the digitized pixel signal obtained by the imaging device 40 and corresponds to the digital image data of the frame image that constitutes the reference video, which is a moving image. Since the pixel signal is ultimately a signal for representing an image, it is also referred to as a video signal. Note that the analog signal processing circuit 44 may be incorporated into the column processing unit 113 of FIG. 5 described later.

[0031] The control unit 46 controls each part of the imaging device 10 and executes various processes related to the creation of video files. As shown in FIG. 3, the control unit 46 includes a controller 47 and a video processing unit 48. For example, a control program (not shown) for controlling each part of the imaging device 10 and causing various processes related to the subsequent thinning process and the creation of video files to be executed is stored in the internal memory 50. The control unit 46 reads out the control program from the internal memory 50 and executes the control program.

[0032] In the present embodiment, the control unit 46 has, for example, one or more processors for executing the control program. As the processor, a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or other IC (Integrated Circuit) can be used, or these can be combined to configure the processor. As represented by an SoC (System on Chip) or the like, the above-described processor may configure the functions of the entire control unit 46 including the controller 47 and the video processing unit 48 with one IC (Integrated Circuit) chip. Note that the hardware configuration of each of the above-described processors may be realized by an electric circuit (Circuitry) combining circuit elements such as semiconductor elements.

[0033] The controller 47 comprehensively controls the imaging device 10 according to the user's operations or a prescribed control pattern. For example, the aforementioned imaging element 40 and analog signal processing circuit 44 are controlled by the controller 47 to capture video (moving images) at a predetermined frame rate. Further, the control unit 46 includes a first mode and a second mode as video imaging modes. The selection between the first mode and the second mode provided as video imaging modes, and the switching between the first mode and the second mode are controlled by the controller 47. The thinning drive for thinning out a part of the reading of a plurality of pixels among the optical images obtained by the imaging element 40 is also controlled by the controller 47.

[0034] In addition, the controller 47 determines shooting conditions according to the shooting environment, and controls the imaging element 40, the analog signal processing circuit 44, and the video processing unit 48 so that the shooting conditions become the determined conditions. The shooting conditions include the exposure amount, white balance, and focus (sharpness) of the shooting lens 14 when shooting video. Furthermore, the controller 47 controls the video processing unit 48 so that the captured video or the video based on the captured video is recorded on the recording medium.

[0035] The video processing unit 48 performs various processes such as gamma correction, white balance correction, and defect correction on the digital image data output from the analog signal processing circuit 44. In addition, the video processing unit 48 compresses the processed digital image data in a compression format conforming to a predetermined standard.

[0036] Then, during video shooting, the video processing unit 48 generates compressed digital image data at a specific frame rate, and acquires a video (strictly speaking, a frame image) from the data. The video (frame image) acquired at this time corresponds to the video shot at the aforementioned first angle of view, that is, the reference video.

[0037] Also, under the control of the controller 47, the video processing unit 48 executes various processes (for example, the setting process described later) on the acquired video, and outputs the processed video to the display 28 frame by frame at a specific frame rate. Furthermore, the video processing unit 48 records the processed video on a recording medium and creates a video file. Since the video processing unit 48 has the function of creating a video file in this way, the imaging device 10 having the video processing unit 48 is used as a video creation device in this embodiment.

[0038] Hereinafter, unless otherwise specified, the operations and processes of the controller 47 and the video processing unit 48 will be described as the operations and processes of the control unit 46. In addition, the processes by the control unit 46 will be described in detail in a later section.

[0039] The internal memory 50 built in the imaging device main body 12 and the memory card 54 that can be attached to and detached from the imaging device main body 12 via the card slot 52 are recording media, and videos are recorded by the control unit 46. Note that the internal memory 50 and the memory card 54 may be outside the imaging device main body 12. In that case, the control unit 46 may record videos on an external recording medium by wire or wirelessly. The buffer 56 functions as a work memory of the control unit 46. Next, the imaging element 40 will be specifically described.

[0040] (Configuration of the imaging element) FIG. 5 shows a configuration example of the imaging element 40. The imaging element 40 shown in FIG. 5 is a diagram showing an outline of the configuration of a CMOS image sensor, which is a type of X-Y address type imaging device, for example. Here, a CMOS image sensor is an image sensor created by applying the CMOS process or using the CMOS process partially.

[0041] The imaging device 40 shown in FIG. 5 includes a pixel array unit 111 including a plurality of unit pixels 42 formed on a semiconductor substrate, and a peripheral circuit unit integrated on the same semiconductor substrate as the pixel array unit 111. The peripheral circuit unit is composed of, for example, a vertical drive unit 112, a column processing unit 113, a horizontal drive unit 114, and a system control unit (control unit 46 in FIG. 3).

[0042] The imaging device 40 further includes a signal processing unit 118 and a data storage unit 119. The signal processing unit 118 and the data storage unit 119 may be mounted on the same substrate as the imaging device 40, or may be arranged on a substrate different from the imaging device 40. Also, each process of the signal processing unit 118 and the data storage unit 119 may be performed by an external signal processing unit provided on a substrate different from the imaging device 40, for example, a DSP (Digital Signal Processor) circuit or software.

[0043] The pixel array unit 111 has a configuration in which unit pixels 42 (hereinafter, may be simply described as "pixels") having a photoelectric conversion unit that generates and accumulates photoelectric charges according to the amount of received light are two-dimensionally arranged in the row direction and the column direction, that is, in a matrix. Here, the row direction refers to the arrangement direction of the pixels in the pixel row (that is, the horizontal direction), and the column direction refers to the arrangement direction of the pixels in the pixel column (that is, the vertical direction).

[0044] In the pixel array unit 111, for the matrix pixel arrangement, pixel drive lines 116 are wired in the row direction for each pixel row, and vertical signal lines 117 are wired in the column direction for each pixel column. The pixel drive line 116 transmits a drive signal for driving when reading a signal from the pixel. In FIG. 5, the pixel drive line 116 is shown as a single wiring, but it is not limited to one. One end of the pixel drive line 116 is connected to the output end corresponding to each row of the vertical drive unit 112.

[0045] The vertical driving unit 112 is composed of a shift register, an address decoder, etc., and drives each pixel of the pixel array unit 111 all at once or by row units. That is, the vertical driving unit 112, together with the system control unit that controls the vertical driving unit 112, constitutes a driving unit that drives each pixel of the pixel array unit 111. Although the specific configuration of this vertical driving unit 112 is not shown, generally, it has a configuration with two scanning systems: a readout scanning system and a blanking scanning system.

[0046] The readout scanning system sequentially selects and scans the unit pixels 42 of the pixel array unit 111 by row units in order to read signals from the unit pixels 42. The signals read from the unit pixels 42 are analog signals. The blanking scanning system performs blanking scanning on the readout row where the readout scanning is performed by the readout scanning system, leading the readout scanning by a time equal to the shutter speed.

[0047] By the blanking scanning by this blanking scanning system, unnecessary charges are swept out from the photoelectric conversion units of the unit pixels 42 in the readout row, thereby resetting the photoelectric conversion units. And by sweeping out (resetting) the unnecessary charges by this blanking scanning system, the so-called electronic shutter operation is performed.

[0048] The signals output from each unit pixel 42 of the pixel row selected and scanned by the vertical driving unit 112 are input to the column processing unit 113 through each of the vertical signal lines 117 for each pixel column. The column processing unit 113 performs predetermined signal processing on the signals output from each pixel 42 of the selected row through the vertical signal lines 117 for each pixel column of the pixel array unit 111, and temporarily holds the pixel signals after the signal processing.

[0049] Specifically, the column processing unit 113 performs noise removal processing, for example, CDS (Correlated Double Sampling) processing, as signal processing. By the CDS processing performed by this column processing unit 113, reset noise and pixel-specific fixed pattern noise such as variations in the threshold values of amplification transistors within the pixel are removed. In addition to noise removal processing, the column processing unit 113 can also be provided with, for example, an AD (analog-digital) conversion function to convert an analog pixel signal into a digital signal and output it.

[0050] The horizontal drive unit 114 is composed of a shift register, an address decoder, etc., and sequentially selects unit circuits corresponding to the pixel columns of the column processing unit 113. By the selective scanning performed by this horizontal drive unit 114, the pixel signals signal-processed for each unit circuit in the column processing unit 113 are sequentially output.

[0051] The system control unit is composed of a timing generator that generates various timing signals, etc., and performs drive control of the vertical drive unit 112, the column processing unit 113, the horizontal drive unit 114, etc. based on the various timings generated by the timing generator.

[0052] The signal processing unit 118 has at least an arithmetic processing function and performs various signal processes such as arithmetic processing on the pixel signals output from the column processing unit 113. The data storage unit 119 temporarily stores the data necessary for the processing when performing signal processing in the signal processing unit 118.

[0053] (Decimation processing) The imaging device 40 shown in FIG. 5 can perform imaging by reading out the pixel signals of all the pixels 42 in the pixel region 43 in a predetermined order such as raster scan order, and can also perform decimation readout. The all-pixel readout utilizes all the pixels 42 in the pixel region 43 to enable high-resolution imaging.

[0054] Decimation readout means, for example, skipping the signals of pixels 42 at a period set for pixel region 43 and reading out signals from the remaining pixels. Also, decimation readout means, for example, reading out signals of a plurality of pixels 42 from pixel region 43 and outputting an image by decimating the signals of some pixels with an analog signal processing circuit 44 or the like. Further, decimation readout includes "decimation readout by addition" in which the number of imaging signals is reduced by reading out while adding imaging signals of a plurality of pixels. In decimation readout, since the number of pixels to be read out in pixel region 43 can be reduced, an image can be generated in a shorter time than full pixel readout. Therefore, it is effective when performing high-speed imaging with a reduced resolution and when performing imaging with low power consumption. Note that decimation readout is also called decimation processing. Decimation processing is performed in both the first mode and the second mode. Decimation processing is performed on the reference video of the first angle of view and the recorded video of the second angle of view. For the recorded video in the recording area of the second angle of view, so-called cropped video, the decimation rate is lowered compared to the reference video, that is, a larger number of read lines are read. The above decimation readout is decimation driving.

[0055] Decimation readout includes 1 / 2 decimation, 1 / 4 decimation, etc. depending on the decimation rate. Here, the decimation rate is the reduction ratio of the pixels in the pixel region of the image sensor corresponding to the video region to be recorded. For example, in image sensor 40, it represents the ratio of the number of read lines in the first direction D1 (see FIG. 6) of pixel region 43, that is, the vertical direction V (see FIG. 6). For example, 1 / 2 decimation is decimation readout in which the number of read lines in the vertical direction is decimated to 1 / 2 of the whole. Also, the ratio of the number of read lines in the second direction D2 (see FIG. 7) of pixel region 43, that is, the horizontal direction H (see FIG. 7) is also called the decimation rate.

[0056] In the following description, the decimation process will be described by taking as an example the decimation process of skipping specific pixels. For example, in the 1 / 2 decimation, as shown in FIG. 6, in the pixel region 43 of the imaging device 40, after the row 76a of the pixel 42 to be read first is read, the row 76a of the pixel 42 that has advanced by one pixel in the vertical direction V, that is, in the row direction from the pixel 42, is read. In this case, the resolution of the 1 / 2 decimation is 1 / 2 of the full pixel readout. Also, in the 1 / 2 decimation, as shown in FIG. 7, in the pixel region 43 of the imaging device 40, after the column 76b of the pixel 42 to be read first is read, the column 76b of the pixel 42 that has advanced by one pixel in the horizontal direction H, that is, in the column direction from the pixel 42, may be read. In this case, the resolution of the 1 / 2 decimation is 1 / 2 of the full pixel readout. When reading pixels in a specific column direction, for example, all pixels are read and converted into digital signals by the analog signal processing circuit 44. Then, the control unit 46 performs signal processing to read out the pixel data of the column 76b of the pixel 42 that has advanced by one pixel in the horizontal direction H from the pixel data corresponding to the column 76b of the pixel 42 first read in the column direction. By repeating this signal processing, decimated pixel data in the column direction is obtained. In this case, the resolution of the 1 / 2 decimation is 1 / 2 of the full pixel readout.

[0057] [Processing by the control unit] Next, the processing by the control unit 46 will be described. The processing by the control unit 46 includes display processing, first recording processing, second recording processing, setting processing, selection processing, switching processing, recording processing, exposure amount adjustment processing, focus adjustment processing, and white balance adjustment processing. Hereinafter, each of the above-described processes will be described individually.

[0058] (Display processing) The display processing is a process of displaying various videos on the display screen of the display 28. For example, when the control unit 46 starts the display processing after the imaging device 10 is activated, at that time, the reference video captured by the imaging device 10 at the first angle of view, that is, the video shown in the reference video A1 in FIG. 4, is displayed. Also, when the setting process is executed, the control unit 46 displays the recorded video in the recording area, which will be described later, on the display screen in the display process. Further, when the video area is switched (that is, when the switching process, which will be described later, is performed), the control unit 46 switches the video displayed on the display screen to the recorded video in the recording area after the switching process.

[0059] The execution operation of the display process by the control unit 46 as described above corresponds to the display step. In this embodiment, it is assumed that the video displayed on the display 28 by the display process is a live video (live view image) captured in real time at that point.

[0060] (First Recording Process) The imaging device 10 includes a first mode and a second mode as video imaging modes by the control unit 46. The first recording process is a process of recording the video data of the reference video with the first angle of view using the imaging element 40 in the first mode.

[0061] (Second Recording Process) The second recording process is a process of recording the video data of the recorded video in the recording area, which has a second angle of view smaller than the first angle of view, in the second mode. Note that the process of recording the video data in the first recording process and the second recording process will be described in the recording process, which will be described later.

[0062] (Setting Process) The setting process is a process of setting a recording area, which has a second angle of view smaller than the first angle of view, within the reference video in the second mode. In the setting process, the user performs an operation for setting the recording area within the reference video. This setting operation will be specifically described with reference to FIG. 8. FIG. 8 shows the procedure for setting the recording area within the reference video by the imaging device according to the embodiment of the present invention.

[0063] As shown in FIG. 8, the user performs a setting operation with the reference video A1 being displayed on the display screen of the display 28. Specifically, when the user performs a predetermined button operation in a state where the reference video A1 is displayed, as shown in FIG. 8, for example, a rectangular region setting frame FR is superimposed and displayed on the reference video A1. The video region A2 surrounded by this region setting frame FR becomes the region of the video extracted from the reference video A1. The viewing angle of the extracted video is the viewing angle of the recording region and is the second viewing angle. The second viewing angle is smaller than the first viewing angle of the reference video A1.

[0064] The region setting frame FR is displayed in a state where its position, size, and aspect ratio are variable. FIG. 9 shows a state in which three video regions A2, A3, and A4 are set within the reference video A1. When a plurality of video regions are set within the reference video A1, the second viewing angles, that is, the sizes, of the respective video regions A2, A3, A4 may be the same among the video regions A2, A3, A4, or may be different among the video regions A2, A3, A4. Also, the number of video regions to be set is not particularly limited, but may be plural, and in this embodiment, it is three.

[0065] When the setting operation by the user is repeated a plurality of times, the control unit 46 accepts each setting operation and sets a number of video regions A2 corresponding to the number of setting operations within the reference video A1 of the imaging region A0. Such an operation of the control unit 46 corresponds to the setting step. In the reference video, the process of extracting the set recording region from the reference video is called the extraction process. The extracted video is also called a crop video or a cut-out video and is the recording target in the second recording process. Also, extracting the recording region from the reference video is also referred to as cutting out.

[0066] (Selection Process and Switching Process) The selection process is a process of selecting, from among the plurality of video regions A2, A3, A4 set within the reference video A1 of the imaging region A0, the video region in which the video to be recorded is shown. The switching process is a process of reselecting a video area from among a plurality of video areas A2, A3, and A4 after the selection process and switching the video area.

[0067] Regarding the procedures of the selection process and the switching process, after the user performs the above-described setting operation to set a plurality of video areas A2, the user selects any other video area A3 or A4 as the video area. Then, the user inputs the video area A2 selected through the first operation button 30, the second operation button 32, the third operation button 34, the touch panel 36, etc. shown in FIG. 2, and the control unit 46 (see FIG. 3) receives the input operation. Based on the user's input operation, the control unit 46 determines the video area from among a plurality of video areas A3 and A4. The operation by such a control unit 46 corresponds to the selection step.

[0068] Also, after the selection step, if the user reselects the video area A2, for example, to another video area A3 and performs the input operation again, the control unit 46 receives the re-input operation and switches the previous video area to another video area A3 based on the re-input operation. The operation by such a control unit 46 corresponds to the switching step.

[0069] As described above, in the present embodiment, in the selection step, a recording area is selected from among a plurality of video areas A2 to A4, and the recorded video of the recording area is recorded as a video file. Also, when the selection step is performed, in conjunction therewith, for example, the subject of the video area A4 selected as the recording area F4 is displayed on the display screen of the display 28 of the imaging device 10 as the recorded video J4 as shown in FIG. 10. FIG. 10 shows the recorded video J4 of the recording area displayed on the display screen of the display 28 of the imaging device 10.

[0070] Also, by performing the switching step after the selection step, the recording area can be switched, and the recorded video of the recording area to be recorded can be changed from the video of one subject to the video of another subject. Thereby, for each of a plurality of subjects in the same scene (location), individual videos (specifically, videos with close-ups of each subject) can be captured by a single imaging device 10 without using a plurality of devices. In addition, the video area A2 on which the recorded video is projected can be switched in time series according to the user's operation, that is, it can be switched, and the video files on which the videos before and after the switching are projected can be easily obtained. Furthermore, in conjunction with the execution of the switching process, the video displayed on the display screen of the display 28 is switched from the recorded video in the recording area before the switching process to the recorded video in the recording area after the switching process. Thereby, the user can confirm the switching of the video area on the display screen.

[0071] (Recording process) The recording process is a process in which the control unit 46 records the optical image of the reference video of the first angle of view and the recorded video of the imaging recording area on a recording medium, and creates a video file (specifically, moving image data) of the video. The target of the recording process is the moving image data in the first recording process of the first mode and the second recording process of the second mode. The format of the moving image data is not particularly limited, and for example, known formats such as MP4, AVCHD (Advanced Video Codec High Definition), AVI (Audio Video Interleave), and MOV can be used. The operation in which the control unit 46 records the recorded video of the recording area in the recording process corresponds to the recording step. Also, the operation in which the control unit 46 creates a video file in the recording process corresponds to the creation step. In this embodiment, the recording of the video and the creation of the video file, that is, the creation of the moving image data, are synonymous, and it is assumed that the creation step corresponds to the recording step.

[0072] Further, when the control unit 46 performs the switching process, it may record the recorded video of the recording area before the switching process and the recorded video of the recording area after the switching process respectively to create a video file. In this case, in the creation process, the control unit 46 combines the recorded video (moving image) of the recording area before the switching process and the recorded video (moving image) of the recording area after the switching process to create moving image data as a video file. Thereby, moving image data of a video in which the subject changes due to the switching of the video area is acquired. The subject of the video area A2 selected as the recording area F2 is displayed as the recorded video J2 as shown in FIG. 11, and at the timing Mc of switching the recording area, for example, the subject of the video area A4 selected as the recording area F4 is displayed as the recorded video J4, and the display on the recorded video changes. Here, FIG. 11 is a schematic diagram showing an example of the recorded video of the recording area based on the moving image data by the imaging device of the embodiment of the present invention. Further, while the video area is being displayed, for example, the entire image may be sandwiched and displayed. That is, after the switching timing Mc and before the display of the recorded video of the video area A4, for example, the reference video A1 shown in FIG. 9 may be displayed.

[0073] The appropriate value (appropriate exposure amount) of the exposure amount of the video area changes according to which area is selected among the plurality of video areas A2, A3, A4 set in the reference video A1 of the imaging area A0. Therefore, in the reference video A1, it is preferable to specify in advance the appropriate exposure amounts of the video areas A2, A3, A4 before the switching process. Thereby, when the video areas A2, A3, A4 are switched, a cropped video of the recording area can be obtained in a short time.

[0074] Specifically, the exposure amount of the video area is such that the control unit 46 determines the values of each of the exposure time (shutter speed) and the aperture amount (F value) based on the appropriate exposure amount of a predetermined area. Such an operation corresponds to a condition determination process by the control unit 46. The value of each parameter determined in this condition determination process is a value indicating the exposure condition when shooting the reference video including the recorded video of the recording area. In the present embodiment, values of both the exposure time and the aperture amount are determined, but the present invention is not limited to this, and only one of the values of the exposure time or the aperture amount may be determined.

[0075] Further, after determining the values of the exposure time and the aperture amount respectively, the control unit 46 determines the value of the sensitivity (ISO sensitivity) of the pixel 42 corresponding to the video area of the recording area according to the combination of these values. Strictly speaking, the gain (amplification ratio) for the pixel signal of the pixel 42 corresponding to the video area of the recording area is determined. Note that the gain may be, for example, an analog gain for an analog signal in the analog signal processing circuit 44 or the like, or a digital gain for a digital signal after digital conversion in the video processing unit 48 or the like.

[0076] Then, after the exposure time, the aperture amount, and the sensitivity are determined, the control unit 46 shown in FIG. 3 controls the aperture drive unit 23, the imaging device 40, the video processing unit 48, etc. so that these parameters become the determined values.

[0077] During the recording of the video by the above procedure, the exposure amount adjustment process is repeatedly performed periodically (for example, for each frame). And each time the exposure amount adjustment process is performed, the values of the exposure time, the aperture amount, and the sensitivity are determined.

[0078] Also, during the period when the moving image is recorded, the exposure amount adjustment process may be repeatedly performed periodically. In this case, among the exposure conditions, the exposure time and the aperture amount change over time, and accordingly, the exposure amount when shooting the reference video is adjusted over time. Furthermore, in conjunction with the changes in the exposure time and the aperture amount, the sensitivity of the pixel 42 corresponding to the recording area changes over time. As a result, the exposure amount of the recording area is adjusted over time. Note that in the present embodiment, "adjusting over time" means determining the adjustment amount for each frame for the parameter to be adjusted and increasing or decreasing each parameter by the adjustment amount for each frame. For reducing the load on the control unit, the exposure amount adjustment process may be performed in conjunction with the switching process.

[0079] (Focus adjustment process) The focus adjustment process is a process in which the control unit 46 controls the focus driving unit 22 to automatically adjust the focus of the imaging lens 14 in the autofocus mode. For focus adjustment in the autofocus mode, known autofocus technologies can be used. Specific examples thereof include contrast autofocus, phase difference autofocus on the image plane, directional light autofocus, and autofocus of the Depth-from-Defocus method. Among these autofocus methods, any one of them may be adopted, or a plurality of methods may be adopted in combination.

[0080] In the imaging device 10, during video shooting, in the reference video A1 of the first angle of view, the recording area of the second angle of view can be moved over time. Here, "moving over time" means moving relative to the reference video A1 so that the position of the recording area gradually changes, and it may include cases where the movement stops (interrupts) halfway. Below, the case where the video area A2 is selected as the recording area F2 will be described.

[0081] The movement of the recording area F2 over time is realized by the video processing unit 48. More specifically, when the user selects one mode through the extraction mode selection screen shown in FIG. 12, the video processing unit 48 moves within the reference video A1 of the first angle of view along the movement path corresponding to the selected mode. FIG. 12 shows the extraction mode selection screen displayed on the display 28 of the imaging device 10.

[0082] More specifically, for example, two types of extraction modes are prepared. One mode is a mode in which the recording area F2 is moved so as to track the subject to be tracked (hereinafter referred to as the tracking mode). The other mode is a mode in which the recording area F2 is moved in a certain direction (hereinafter referred to as the panning mode).

[0083] When the tracking mode is selected, the video processing unit 48 moves the recording area F2 representing the video of the subject to track the subject to be tracked. The procedure will be described as follows. When the tracking mode is selected, if the user designates by touching on the screen the recording area F2 representing the video of the subject within the reference video A1 of the first angle of view, the video processing unit 48 sets the subject as the tracking target. Thereafter, as shown in FIG. 13, the video processing unit 48 moves the recording area F2 so that the subject to be tracked is contained within the recording area F2. FIG. 13 shows the movement path of the recording area F2 when the tracking mode of the imaging device 10 is selected.

[0084] As described above, when the tracking mode is selected, as long as the subject to be tracked exists within the reference video A1 of the first angle of view, the reference video A1 of the first angle of view in which the subject to be tracked is reflected can be constantly displayed on the display 28. In addition, it is not necessary for the user to move the imaging device 10 himself / herself to track the subject, and thus there is no manual change of the angle of view, and it is also possible to avoid video disturbances (such as video blurring) caused by manual change of the angle of view. Such an effect is particularly effective when shooting is performed at the angle of view when using an anamorphic lens.

[0085] Note that the algorithm for searching for the subject to be tracked within the reference video A1 of the first angle of view is not particularly limited. For example, the image of the subject set as the tracking target is stored in the buffer 56 (see FIG. 3) as a template image, and a known template matching technique is applied to compare the above template image with the reference video. Then, as a result of the comparison, the video of the portion that matches the template image may be specified.

[0086] When the panning mode is selected, the video processing unit 48 slides the recording area F2 in a certain direction (for example, the horizontal direction of the reference video A1 of the first angle of view), that is, pans it. To explain the procedure, when the panning mode is selected, the user sets the start position, moving direction, moving speed, etc. of the recording area F2 on the setting screen (not shown) of the display 28. After that, as shown in FIG. 14, the video processing unit 48 automatically pans the recording area F2 from the set start position in the set direction at the set speed. FIG. 14 shows the movement path of the recording area F2 when the panning mode of the imaging device 10 is selected.

[0087] As described above, when the panning mode is selected, it is possible to display on the display 28 a video in which the shooting angle is continuously changed in a certain direction, so to speak, a panoramic video. In addition, it is not necessary for the user to move the imaging device 10 himself for changing the angle, and therefore there is no manual change of the angle of view, and it is also possible to avoid image disturbance (image blur, etc.) caused by manual change of the angle of view. Such an effect is particularly effective when shooting at the angle of view when using an anamorphic lens.

[0088] By the way, regarding the setting items related to the recording area F2, for example, size, aspect ratio, area ratio, and moving speed during movement, etc., usually the initially set content is adopted, but the user can arbitrarily set it on the setting screen (not shown) of the display 28. Here, the aspect ratio of the recording area F2 is the aspect ratio of the reference video A1 of the first angle of view, and strictly speaking, it is the ratio of the number of pixels in the vertical and horizontal directions respectively. The area ratio of the recording area F2 is the area ratio of the recording area F2 with respect to the reference video A1 of the first angle of view. The moving speed of the recording area F2 is the number of pixels passed by the moving recording area F2 per unit time when the angle of view is divided into pixel units.

[0089] <Imaging according to the first example of the imaging device> Next, imaging using the imaging device 10 having the above-described functions will be described. FIG. 15 is a flowchart showing the first mode of the imaging device according to an embodiment of the present invention, and FIG. 16 is a flowchart showing the second mode of the imaging device according to an embodiment of the present invention. As described above, the control unit 46 of the imaging device 10 includes the first mode and the second mode as video imaging modes, and records video data of a video based on an optical image of a reference video of a first angle of view.

[0090] (First mode) In FIG. 15, in the first mode, the control unit 46 captures an optical image of a reference video of a first angle of view using the imaging element 40 (step S10), and executes a first recording process (step S12) for recording video data of the reference video. In the first recording process of the first mode, the control unit 46 performs thinning drive for thinning out a part of the reading of a plurality of pixels in the optical image (step S14). The first recording process and the thinning drive are as described above. (Second mode) Also, in FIG. 16, in the second mode, the control unit 46 executes a setting process (step S16) for setting a recording area, which is a second angle of view smaller than the first angle of view, in the reference video, and a second recording process (step S18) for recording video data of the recorded video in the recording area. In the second recording process of the second mode, the control unit 46 performs thinning drive for thinning out a part of the reading of a plurality of pixels in the optical image (step S20). The second recording process and the thinning drive are as described above.

[0091] When the control unit 46 performs thinning drive in step S14 and step S20, the control unit 46 sets the second thinning rate in the thinning drive (step S20) of the second recording process to be lower than the first thinning rate in the thinning drive (step S14) of the first recording process. That is, the second recording process reads out a larger number of pixels than the first recording process. Specifically, in FIG. 6, in the thinning drive of the first recording process, in the first direction D1 of the pixel region 43, taking three image columns as one unit, two image columns are read out of the three image columns. In the thinning drive of the second recording process, in the first direction D1 of the pixel region 43, taking four image columns as one unit, three image columns are read out of the four image columns.

[0092] (Change in the second subsampling rate) In FIG. 17, in the second mode, the control unit 46 preferably changes the second subsampling rate of the recording area according to the second angle of view. The second angle of view is a recording area smaller than the first angle of view, but its size changes according to user settings. If the size of the second angle of view changes, the total number of pixels corresponding to the second angle of view also changes. That is, as the recording area becomes larger, the corresponding number of images increases, and as the recording area becomes smaller, the corresponding number of pixels decreases. If the subsampling rate in the second angle of view is made the same regardless of the size of the recording area and the recording area is reduced, there is a possibility that sufficient resolution cannot be obtained after subsampling. For this reason, it is preferable to change the second subsampling rate of the recording area according to the second angle of view. In this case, as shown in FIG. 17, the size of the recording area is specified (step S30). The size of the recording area can be specified by the total number of pixels. A threshold value is set in advance for the size of the recording area, and compared with the threshold value (step S32). If it exceeds the threshold value, the second subsampling rate is increased (step S34), and if it is less than the threshold value, the second subsampling rate is decreased (step S36). Note that in step S32, if the size of the recording area is the same as the threshold value, the second subsampling rate is not changed (step S38).

[0093] (Input resolution and output resolution) Also, as described above, the second subsampling rate preferably changes according to the size of the recording area (second angle of view). Due to the change in the second subsampling rate of the control unit 46, the imaging device 40 changes the input resolution of the recorded video of the recording area input to the control unit 46. When the input resolution from the imaging device 40 is higher than the output resolution which is the recording format of the moving image data, it is preferable to reduce the input resolution to match the output resolution. When the input resolution of the recorded video of the recording area is lower than the output resolution, it is preferable to use a known complement technology to complement the input resolution to match the output resolution. Then, the control unit 46 outputs a moving image file having the output resolution to the memory card 54 (see FIG. 3) and records the moving image file on the memory card 54.

[0094] (Notification to User) When the input resolution of the reference video is lower than a predetermined set resolution, the total number of pixels of the reference video decreases. Therefore, a cropped video with sufficient resolution cannot be obtained, and the image quality of the cropped video deteriorates. For this reason, the control unit 46 preferably compares the input resolution with the set resolution, and when the input resolution is small, increases the imaging magnification of the reference video by the zoom function of the imaging lens to increase the total number of pixels of the reference video. Thereby, a cropped video with sufficient resolution can be obtained. Increasing the imaging magnification by the zoom function of the imaging lens to capture the reference video is called optical zoom processing.

[0095] The control unit 46 notifies the user about the zoom function. In this case, for example, a notification regarding the zoom function is displayed on the display 28. The display of the notification is not particularly limited, and for example, it may be indicated by characters such as "Please use the zoom function". In addition to this, as the display of the notification, the entire surface of the display 28 may be simply repeated in a single red color or a plurality of colors and blinked.

[0096] (Decimation Drive) When performing decimation, the control unit 46 divides the pixel region 43 into a drive region 43a composed of pixels to which a drive signal is supplied through the first wiring to the switch unit and a non-drive region 43b other than the drive region 43a in the first direction D1 as shown in FIGS. 18 and 19 with respect to the image sensor 40 (see FIG. 5). The control unit 46 preferably records the video region A2 using the drive region 43a in the second recording process, and associates the non-drive region 43b with the partial regions E3 and E4 corresponding to the video regions A3 and A4 other than the video region A2. That is, in the second recording process, the partial region E2 of the drive region 43a of the pixel region 43 corresponding to the video region A2 is recorded as the recorded video, the non-drive region 43b other than the drive region 43a of the pixel region 43 has no pixels read out, and the video regions A3 and A4 are not recorded. The drive region 43a is decimated at the second decimation rate. The decimation process improves the processing speed of the control unit 46 by reducing the number of pixels read from the image sensor 40. Here, FIG. 20 is a schematic diagram showing a reference video A1 to be thinned out. For example, three video regions A2 to A4 are set. The partial region E2 shown in FIG. 18 corresponds to the video region A2 shown in FIG. 20, the partial region E3 shown in FIG. 18 corresponds to the video region A3 shown in FIG. 20, and the partial region E4 shown in FIG. 18 corresponds to the video region A4 shown in FIG. 20.

[0097] In the second recording process, the control unit 46 moves the position of the recording region in the reference video or changes the second angle of view of the recording region in accordance with the movement of the subject in the recording region. It is preferable that the control unit 46 changes the position or width of the driving region in the pixel region in accordance with the movement of the position of the recording region or the change of the second angle of view. When one of the plurality of video regions is specified as the recording region, when the subject moves, at least one of the position of the recording region and the size of the recording region changes. That is, at least one of the second angle of view and the position of the second angle of view changes. For example, in FIGS. 18 and 19, the video region A2 moves and the sizes of the video regions A2 are different. The video region E2 in FIG. 19 is larger than the video region A2 in FIG. 18, and the width d2 of the driving region 43a in the first direction D1 in FIG. 19 is wider than the width d1 of the driving region 43a in the first direction D1 in FIG. 18. Note that the width of the driving region 43a in the first direction D1 may be made wider than the width d2 of the driving region 43a in the first direction D1 specified based on the movement of the subject when the subject moves. Also, based on the moving speed of the subject, the moving direction of the subject, etc., the set position of the driving region 43a in the first direction D1 and the width of the driving region 43a can be set. As a result, since the video is read out at an angle of view larger than that of the video region A2, even if there is a sudden movement of the subject in the video region A2, the control unit 46 can stably detect the subject.

[0098] In order to suppress the lack of resolution due to the thinning process, when changing the position or width of the driving region 43a, the control unit 46 preferably determines the width of the driving region 43a so that the output resolution becomes equal to or higher than the set resolution. More preferably, the output resolution is within a certain range. Thereby, in the reference video, regardless of the size of the subject shown, imaging can be performed while maintaining the output resolution. In this way, it is preferable to change the reading position, reading width, and number of readings of the pixel region 43 of the image sensor 40 in real time so that the output resolution becomes constant according to the size and position of the recording region.

[0099] When there is movement of the subject as described above, by implementing the above-described tracking mode, the control unit 46 automatically tracks the subject, and the movement position and size of the subject are specified. Thereby, even if the video region A2 moves, the position and width of the driving region 43a of the pixel region 43 corresponding to the video region A2 can be specified in real time. Furthermore, the thinning rate is also specified based on the width of the driving region 43a of the pixel region 43 and the output resolution. Note that the movement of the subject in the video region A2 has been taken as an example, but it is not limited thereto. For example, for each subject in the three video regions A2, A3, and A4, automatically track in the above-described tracking mode, specify the movement position and size of the subject, and the position and width of the driving region 43a of the pixel region 43 corresponding to the video regions A2, A3, and A4, and the thinning rate can also be specified in real time.

[0100] Also, even when there is no movement of the subject, the control unit 46 sets the position and size in advance for the video regions A2, A3, and A4 shown in FIG. 20, for example. Furthermore, the position and width of the driving region 43a of the pixel region 43 corresponding to the video regions A2, A3, and A4 are specified. Furthermore, the thinning rate is also specified based on the width of the driving region 43a of the pixel region 43 and the output resolution. Thereby, for example, when switching from the video region A2 to the video region A4 during video shooting, as shown in FIG. 21, the driving region 43a of the pixel region 43 specified in advance is applied to the partial region E4 corresponding to the video region A4, and the video region A4 can be obtained as the recorded video. Even when there is no movement of the subject, in order to suppress the lack of resolution due to the thinning process, when the position or width of the drive area 43a is changed, the control unit 46 preferably determines the width of the drive area 43a so that the output resolution becomes equal to or higher than the set resolution. It is more preferable that the output resolution is within a certain range.

[0101] (Determination of input resolution) When the drive area 43a corresponds to a plurality of video areas including the recording area, it is preferable to determine the second thinning rate based on the smallest second viewing angle among the second viewing angles of the plurality of video areas. This is because if sufficient resolution is obtained at the smallest second viewing angle, sufficient resolution can be obtained regardless of which video area is selected as the recording area. The second thinning rate at the smallest second viewing angle is preferably set such that the input resolution is larger than the output resolution. Thereby, it is not necessary to complement the input resolution to match the output resolution, which is the recording format of the video file, and the image quality of the recorded video file can be improved. Hereinafter, the detection of the movement of the subject will be described. The movement of the subject is detected by performing a subject movement detection process. A known detection processing method can be used for the detection of the movement of the subject.

[0102] (Creation of video data) In the second mode, the control unit 46 executes a setting process of setting a plurality of video areas with the second viewing angle in the reference video and a selection process of selecting a recording area from the plurality of video areas, and the second recording process preferably combines the recording video before selection and the recording video after selection in the selection process to record the video data. In the setting process, a plurality of video regions, for example, three video regions A2, A3, and A4 as shown in FIG. 20, are set in the reference video. Next, in the selection process, a recording region is selected from the plurality of video regions A2, A3, and A4. For example, the video region A2 is selected as the recording region. In the second recording process, the video region A3 before the video region A2 is selected and the video region A2 are combined to record moving image data. In this case, for example, moving image data in the form shown in FIG. 11 can be obtained. That is, when the video regions A2 to A4 are selected by the user, they become the recording regions.

[0103] Also, when the video region A4 in FIG. 20 is used as the recording region, the control unit 46 reads only the video region A4 in FIG. 20 corresponding to the selected recording region from the pixel region 43 (see FIG. 4), and does not read the other video regions (the video regions A2 and A3 in FIG. 20) from the pixel region 43 (see FIG. 4). Therefore, the movement of the subject in the other video regions cannot be detected. When the recording region is switched from the video region A4 to the video region A2, the control unit 46 may record the video region A2 where no subject is shown. Therefore, the control unit 46 executes a second recording process having a non-driven region 43b (see FIG. 18) and a detection process of reading a plurality of video regions and detecting the movement of the subject in each region. Then, by the detection process, the control unit 46 detects the position of the subject in each of the video regions A2, A3, and A4, and appropriately changes the positions and angles of view of each of the video regions A2, A3, and A4. For example, as shown in FIG. 21, during two second recording processes in which only the partial region E4 corresponding to the video region A4 is used as the driven region 43a, a detection process of a detection video K (see FIG. 23) having a wide region as the driven region 43a is performed.

[0104] In the example shown in FIG. 22, when the video S1 obtained in the first second recording process is compared with the video S2 obtained in the second second recording process, by using the detection process, the position of the video region A2 (human) that was not the recording region has moved. This is because the control unit 46 can detect the movement of the subject in the video region that is not recorded by the entire video of the detection video K (see FIG. 23). In the detection process, when no movement is detected, if the video S1 obtained in the first second recording process is compared with the video S2 obtained in the second second recording process, the position of the video area A2 (human) that was not in the recording area is the same although not shown in the figure. Even in this case, it is because the control unit 46 can detect the subject in the video area that has not been recorded based on the entire video of the detection video K (see FIG. 23).

[0105] The detection process is performed for each predetermined frame, and the detection video K has a higher decimation rate than the recorded video of the recording process. Since the detection video K only needs to be able to detect the position of each subject in the reference video, even if it is less than a predetermined resolution, it is acceptable. When displaying the detection video K, it is preferable to set the frame rate of the entire video higher than that of a general video. For example, since the frame rate of the detection video K is preferably 15 to 30 fps (frames per second), the frame rate of the entire video is preferably 30 to 60 fps. In this case, it is preferable that the video data does not include the data representing the detection video K. This is because the detection video K represents the entire imaging range and is for detecting the position of each subject, so there is no need to record it on the recording medium.

[0106] By using the detection video K to perform the above-described tracking mode, the control unit 46 automatically tracks the subjects in the video areas A2, A3, and A4, respectively, and identifies the moving positions and sizes of the respective subjects. Furthermore, based on the width of the drive area 43a of the pixel area 43 and the output resolution, the decimation rate of each of the video areas A2, A3, and A4 can also be determined. According to the switched recording area, the position and width of the drive area 43a of the pixel area 43 and the decimation rate can also be specified in real time. Note that the detection video K (see FIG. 23) is, for example, the reference video A1 or the reference video A1 with a high decimation rate, but is not limited thereto, and may be an image with a wider angle of view than the reference video A1. For example, it may be an image with an angle of view wider than the first angle of view including each of the video areas A2, A3, and A4.

[0107] <Second Example of Imaging Device> FIG. 24 and FIG. 25 are perspective views showing a second example of the imaging device according to an embodiment of the present invention. FIG. 26 is a schematic diagram showing a corresponding video corresponding to the reference video according to an embodiment of the present invention, and FIG. 27 is a schematic diagram showing another example of the reference video according to an embodiment of the present invention.

[0108] The imaging device 200 shown in FIGS. 24 and 25 is a smartphone including a plurality of imaging units. Note that, regarding functions related to the smartphone in the imaging device 200, although detailed description thereof is omitted, the imaging device 200 can perform data communication and voice calls. The imaging device 200 shown in FIGS. 24 and 25 includes a front camera 212 provided on the surface where the display 211 is provided, and two main cameras 213 and 214 provided on the back surface of the surface where the display 211 is provided. The main cameras 213 and 214 are imaging units for obtaining an image with a desired imaging magnification by, for example, optical zoom. The display 211 can have the same configuration as the display 28 shown in the imaging device 10 shown in FIG. 3.

[0109] The main camera 213 and the main camera 214 differ in functions and the like. For example, the imaging angles of view that can be imaged are different. The main camera 214 can image with a wider angle of view than the main camera 213. The main camera 213 executes a first mode and a second mode, and is an imaging unit for performing normal imaging to obtain a crop video. The main camera 214 is an imaging unit for imaging a wide-angle image.

[0110] In the imaging device 200, the main camera 213 captures images in the same manner as the imaging device 10, and obtains, for example, a reference image A1 (see FIG. 20) including a plurality of video regions A2 to A4. The control unit 46 sets the set position in the first direction D1 (see FIG. 21) of the drive region 43a (see FIG. 21) of the pixel region 43 corresponding to the set recording region, and the width of the drive region 43a, based on the sizes and positions of the plurality of video regions. Further, based on the width of the drive region 43a of the pixel region 43 and the output resolution, if the recording region is switched, the decimation ratio of the switched recording region is specified. On the other hand, the main camera 214 captures images with a wider angle of view than the main camera 213, and obtains, for example, the above-described detection video K (see FIG. 23). The control unit 46 obtains the positions, sizes of the video regions, focus information, exposure information, etc. of the unselected video regions among the plurality of video regions from the detection video K including the unselected video regions among the plurality of video regions obtained by the main camera 214. Thereby, when there is a switching of the recording region in the plurality of video regions, for the recording region after the switching, the position and width of the drive region 43a of the pixel region 43 and the decimation ratio can also be specified in real time according to the switched recording region. Thereby, a moving image of the subject in the recording region after the switching can be acquired in a short time.

[0111] For example, as shown in FIG. 20, when there are three video regions A2, A3, and A4 in the reference video A1, the video region A2 is selected as the recording region and recorded by the main camera 213. On the other hand, the main camera 214 captures a corresponding video B1 (see FIG. 26) corresponding to the reference video A1 in FIG. 20, and acquires videos of corresponding regions B2, B3, and B4 (see FIG. 26) corresponding to the plurality of video regions A2, A3, and A4 (see FIG. 20). For each of the corresponding regions B2, B3, and B4, the control unit 46 acquires the position, size, focus information, exposure information, etc. respectively. Based on the acquired information, the set position in the first direction D1 of the drive region 43a of the pixel region 43 and the width of the drive region 43a are set. Further, based on the width of the drive region 43a of the pixel region 43 and the output resolution, the decimation ratio of the recording region is specified. Thereby, after recording the video region A2 shown in FIG. 20, when switching to another video region A3 shown in FIG. 20, a recorded video of the video region A3 with appropriate position and size and in focus can be recorded in a short time.

[0112] In the imaging device 200, the main camera 213 (the first imaging element of the present invention) captures an optical image of the recorded video in the first recording process and the second recording process. The first imaging element is responsible for capturing the reference video and the recorded video. On the other hand, the main camera 214 (the second imaging element of the present invention) acquires videos of a plurality of corresponding regions corresponding to a plurality of video regions including the recording region. The second imaging element captures the reference video and is responsible for acquiring information such as the positions of the plurality of video regions. The control unit 46 executes a detection process for detecting the movement of the subject within the plurality of corresponding regions. In the detection process, the control unit 46 causes the main camera 214 to acquire a detection video of an angle of view including the plurality of corresponding regions, and moves at least one of the plurality of video regions or changes the angle of view as the subject moves. In this case, it is preferable that the moving image data does not include data representing the detection video. By detecting the position of each subject in the entire imaging range by the main camera 214, when the recording region is switched, recording can be performed in a short time after the switching of the recording region.

[0113] Regarding motion detection, it is as described above. For example, as time passes, the subject moves, the video area A2 (human) moves as shown in FIG. 27, the video area A2 becomes larger, and the video areas A3 and A4 shown in FIG. 20 do not exist. The main camera 214 records the corresponding video B1 (see FIG. 26) corresponding to the reference video A1 shown in FIG. 27, and the control unit 46 acquires the position, size, focus information, etc. for the corresponding area B2 (not shown) corresponding to the video area A2. Therefore, as shown in FIG. 27, even if the position, size, focus information, etc. of the video area A2 change, the recorded video of the video area A2 can be obtained in a short time. Note that the main camera 214 may capture an image with a wider angle of view than the reference video A1. By using this image with a wider angle of view, even if the movement range of the subject exceeds the range of the reference video A1 with the first angle of view, it can be tracked.

[0114] <The Third Example of the Imaging Device> The device configuration of the third example of the imaging device has the same configuration as the imaging device 10 shown in FIG. 3 described above, except that the processing content of the control unit 46 is different from that of the imaging device 10 shown in FIG. 3. Therefore, the same reference numerals as those of the above-described imaging device 10 are used for the description. FIG. 28 is a flowchart showing the processing of the control unit of the third example of the imaging device according to the embodiment of the present invention.

[0115] The imaging device 10 records moving image data of a video based on an optical image captured by an imaging element having a pixel region in which a plurality of pixels are arranged in a first direction and a second direction intersecting the first direction, and an optical image captured by the control unit 46 of a reference video with a first angle of view. The control unit 46 executes a setting process (step S40) of setting a plurality of video regions, which are video regions with a second angle of view smaller than the first angle of view, in the reference video, a determination process (step S42) of determining a recording region from among the plurality of video regions, a reading process (step S44) of reading a video signal of a pixel region corresponding to the recording region, and a recording process (step S46) of recording the moving image data of the recorded video of the recording region. Thereby, the moving image data of the recording region can be acquired (step S48). In the reading process (step S44), the control unit 46 thins out the reading of the pixels in the first direction and the second direction, and reads a video signal of a partial region of the pixel region corresponding to the recording region. Then, in the recording process (step S46), the control unit 46 records the video of the partial region as a recording target.

[0116] An example of thinning-out driving in the first direction and the second direction will be described. For example, as described in FIG. 18, the pixel drive line 116 (see FIG. 5) of the pixel row corresponding to the partial region is driven to drive only a specific pixel row. Then, in FIG. 5, the video signal of the driven specific pixel row flows through the vertical signal line 117 to the column processing unit 113. Then, the horizontal drive unit 114 selects only the video signal of the pixel column corresponding to the partial region and supplies the video signal to the signal processing unit 118. By using such a method, the video signal of the partial region can be read out.

[0117] FIG. 29 is a schematic diagram showing a reference video A1 to be thinned out according to an embodiment of the present invention. In FIG. 29, for example, three video regions A2 to A4 are set in the reference video A1 by a setting process performed by the control unit 46. When the reference video A1 shown in FIG. 29 is made to correspond to the pixel region 43 of the image sensor 40 shown in FIG. 5 described above, as shown in FIG. 30, a partial region E2 corresponding to the video region A2, a partial region E3 corresponding to the video region A3, and a partial region E4 corresponding to the video region A4 are formed in the pixel region 43. That is, when any one of the video regions A2 to A4 is selected as the recording region, any one of the partial regions E2 to E4 of the pixel region 43 corresponding to any one of the selected video regions A2 to A4 becomes the thinned-out region in the first direction D1 and the second direction D2. The positions and sizes of the partial regions E2 to E4 are specified by the control unit 46 as, for example, coordinate data representing the positions of the pixels 42.

[0118] The control unit 46 reads out the video signal of the pixel region corresponding to the video region determined as the recording region from among the video regions A2 to A4 set by the setting process from the image sensor 40. Then, when reading out from the image sensor 40, the control unit 46 thins out the pixels in the first direction D1 and the second direction D2 and reads out the video signal of the partial region of the pixel region 43 corresponding to the recording region. Then, in the recording process, the control unit 46 records the video of the partial region as the recording target. That is, when, for example, the video region A2 is determined as the recording region from among the plurality of video regions A2 to A4, the control unit 46 records the moving image data of the recorded video of the partial region E2 corresponding to the recording region F2. In this case, in the recording process, the control unit 46 sets the video of the partial region E2 as the recording target and excludes the videos of the other partial regions E3 and E4 from the recording target. Thereby, an image of only the recording region selected from the video regions A2 to A4 can be obtained. Note that the control unit 46 may change the thinning rate of the thinning drive for thinning out a part of the reading of a plurality of pixels in the recording region due to the change in the second viewing angle in the reading process. Thereby, sufficient resolution can be obtained for the recording region after thinning. Also, the sizes of the partial region and the recording region do not have to exactly match, and the size of the partial region may be slightly larger than the recording region.

[0119] One embodiment of the present invention is basically configured as described above. Although the imaging device according to one embodiment of the present invention has been described in detail above, one embodiment of the present invention is not limited to the above-described embodiment, and various improvements or modifications may be made without departing from the gist of the present invention, of course.

Explanation of Reference Numerals

[0120] 10 Imaging device 12 Imaging device main body 13 Mount 14 Imaging lens 18 Optical component unit 19 Focus optical component 20 Diaphragm 21 Electronic dimming filter 22 Focus drive unit 23 Diaphragm drive unit 24 Voltage application unit 26 Release button 28 Display 30 First operation button 32 Second operation button 34 Third operation button 36 Touch panel 38 Optical shutter 40 Image sensor 42 Unit pixel (pixel) 43 Pixel region 43a Drive region 43b Non-drive region 44 Analog signal processing circuit 46 Control unit 47 Controller 48 Video processing unit 50 Internal memory 52 Card slot 54 Memory card 56 Buffer 111 Pixel array section 112 Vertical drive section 113 Column processing section 114 Horizontal drive section 116 pixel drive lines 117 vertical signal lines 118 signal processing unit 119 data storage unit 200 imaging device 211 display 212 front camera 213, 214 main cameras A0 imaging area A1 reference image A2, A3, A4 image areas B1 corresponding image B2, B3, B4 corresponding areas Ad unit area D1 first direction D2 second direction E2, E3, E4 partial areas F2, F4 recording areas FR area setting frame H horizontal direction J2, J4 recorded images K detected image L1 optical axis Mc timing S1, S2 images V vertical direction

Claims

1. An imaging device including: an image sensor having a pixel region in which a plurality of pixels are arranged in a first direction and a second direction intersecting the first direction, the image sensor capturing an optical image of a reference image of a first angle of view; and a control unit configured to have a first mode and a second mode as video imaging modes, and record video data of a video based on the optical image, wherein the control unit performs thinning drive to thin out a part of the reading of the plurality of pixels in the first mode and the second mode, the control unit sets a second thinning rate in the thinning drive in the second mode to be lower than a first thinning rate in the thinning drive in the first mode, and further, in the second mode, the control unit changes a second angle of view of the recording area in accordance with movement of a subject within a recording area that is a second angle of view smaller than the first angle of view, and changes the second thinning rate of the recording area by the change in the second angle of view, due to the change in the second thinning rate by the control unit, the image sensor changes an input resolution of a recorded video of the recording area to be input to the control unit, and when the input resolution is higher than an output resolution of the video data, the control unit reduces the input resolution to match the output resolution, and when the input resolution is lower than the output resolution, the control unit complements the input resolution to match the output resolution, in the second mode, the control unit performs a setting process of setting a plurality of video areas of the second angle of view within the reference image, and a selection process of selecting the recording area from the plurality of video areas, and the second mode combines a recorded video before selection and a recorded video after selection in the selection process to record video data, the control unit performs a detection process of detecting movement of the subject within the plurality of video areas including the recording area, the control unit executes the second mode a plurality of times, executes the detection process between the plurality of executions of the second mode, and in the detection process, the control unit acquires a detection video of an angle of view including the recording area and at least one of the plurality of video areas, and moves at least one of the plurality of video areas or changes the second angle of view in accordance with the movement of the subject.

2. An imaging device further comprising an imaging lens having a zoom function, wherein the image sensor receives light transmitted through the imaging lens and captures the optical image of the reference image of the first angle of view. The imaging device according to claim 1, wherein when the input resolution is lower than the set resolution, the control unit notifies the user of the zoom function.

3. The pixel includes a photoelectric conversion element and a switch unit, and a pixel circuit including a plurality of first wirings extending in the first direction and a plurality of second wirings extending in the second direction is provided, the first wiring and the second wiring are electrically connected to the switch unit, and when the control unit supplies a drive signal to the switch unit through the first wiring, a video signal based on the photoelectric conversion element is output to the second wiring according to the drive signal. The control unit executes the thinning drive for dividing the pixel region in the first direction into a drive region composed of the pixels to which the drive signal is supplied to the switch unit through the first wiring and a non-drive region other than the drive region. The imaging device according to claim 1, wherein in the second mode, the control unit records the recorded video using the drive region and associates the non-drive region with the video region other than the recording region.

4. The imaging device according to claim 3, wherein the control unit changes the position or width of the drive region in the pixel region in accordance with the movement of the position of the recording region or the change of the second angle of view.

5. The imaging device according to claim 4, wherein the control unit determines the second thinning rate based on the smallest second angle of view among the second angles of view of the plurality of video regions.

6. The imaging device according to claim 1, wherein the moving image data does not include data representing the detected video.

7. The imaging device according to any one of claims 1 to 6, wherein the pixel resolution of the pixel region of the imaging element is 70 million or more.

8. The imaging device according to any one of claims 1 to 7, wherein the thinning drive includes thinning readout by addition for reducing the number of signals of the imaging signal by reading out while adding the imaging signals of the plurality of pixels.

Images