Video compression device, electronic apparatus, and video compression program

Abstract

To achieve compression according to a frame rate for a frame for which a different frame rate (FR) is set.SOLUTION: A video compression device includes: an acquisition unit that acquires video data output from a video element that has a first imaging area and a second imaging area for imaging a subject and can set a first FR in the first imaging area and a second FR faster than the first FR in the second imaging area; a motion detection unit that sets a specific motion vector indicating that there is no motion in an object in imaging data on the first imaging area about the imaging data on the first imaging area in the first FR of the video data, and detects a motion vector about the video data on the second image area in the 2FR frame; and a motion compensation unit that motion-compensates the image data on the first imaging area based on the specific motion vector and motion-compensates the image data on the second imaging area based on the motion vector.SELECTED DRAWING: Figure 9

Description

Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2017-192102, filed on September 29, 2017, the contents of which are incorporated herein by reference. [Technical Field]

[0002] The present invention relates to a video compression device, an electronic device, and a video compression program. [Background technology]

[0003] An electronic device has been proposed that includes an imaging element in which a back-illuminated imaging chip and a signal processing chip are stacked (hereinafter referred to as a stacked imaging element) (see Patent Document 1). In the stacked imaging element, the back-illuminated imaging chip and the signal processing chip are stacked so that they are connected via microbumps in each predetermined area. However, when a stacked imaging element allows multiple imaging conditions to be set within the imaging area, frames captured under the multiple imaging conditions are output, and video compression of such frames has not been considered in the past. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-49361 Summary of the Invention

[0005] A video compression device according to one aspect of the technology disclosed in the present application has a first imaging area for imaging a subject and a second imaging area for imaging the subject, and comprises an acquisition unit that acquires video data including a plurality of frames output from an image sensor capable of setting a first frame rate for the first imaging area and a second frame rate for the second imaging area that is faster than the first frame rate; a motion detection unit that sets a specific motion vector indicating that an object in the image data of the first image area is not moving for image data of the first image area in the frames captured at the first frame rate among the video data acquired by the acquisition unit, and detects the motion vector for image data of the second image area in the frames captured at the second frame rate; and a motion compensation unit that performs motion compensation for the image data of the first image area based on the specific motion vector, and performs motion compensation for the image data of the second image area based on the motion vector detected by the motion detection unit.

[0006] Another aspect of the technology disclosed in the present application is a video compression device that compresses video data including multiple frames output from an image sensor that has a first imaging area for imaging a subject and a second imaging area for imaging the subject, is capable of setting a first frame rate in the first imaging area, and is capable of setting a second frame rate faster than the first frame rate in the second imaging area, and has: a generation unit that generates, for each of the multiple frames, a reference frame to be referenced by a frame that is input temporally later than the frame based on difference data between the frame and a predicted frame that predicts the frame, and the predicted frame; and a motion compensation unit that performs motion compensation on a frame of the multiple frames that was imaged at the second frame rate, using a specific reference frame from the reference frames generated by the generation unit.

[0007] An electronic device that is one aspect of the technology disclosed in the present application has a first imaging area that images a subject and a second imaging area that images the subject, an imaging element that is capable of setting a first frame rate for the first imaging area and a second frame rate that is faster than the first frame rate for the second imaging area, images the subject at the frame rate set for each imaging area and outputs multiple frames that are video data, and a compression unit that compresses each of the multiple frames imaged by the imaging element based on the first frame rate and the second frame rate.

[0008] Another aspect of the technology disclosed in the present application is an electronic device having a first imaging area for imaging a subject and a second imaging area for imaging the subject, an imaging element capable of setting a first frame rate for the first imaging area and a second frame rate faster than the first frame rate for the second imaging area, imaging the subject at the frame rates set for each imaging area and outputting multiple frames of video data, a generation unit that generates, for each of the multiple frames, a reference frame to be referenced by a frame input temporally later than the frame, based on difference data between the frame and a predicted frame that predicts the frame, and the predicted frame, and a motion compensation unit that performs motion compensation on a frame of the multiple frames that was imaged at the second frame rate, using a specific reference frame from the reference frames generated by the generation unit.

[0009] A video compression program that is one aspect of the technology disclosed in the present application is a video compression program that causes a processor to compress video data including multiple frames output from an image sensor that has a first imaging area for imaging a subject and a second imaging area for imaging the subject, and that is capable of setting a first frame rate for the first imaging area and a second frame rate that is faster than the first frame rate for the second imaging area, and causes the processor to perform an acquisition process that acquires the video data, and a compression process that compresses the video data acquired by the acquisition process based on the first frame rate and the second frame rate.

[0010] Another aspect of the technology disclosed in the present application is a video compression program that causes a processor to compress video data including multiple frames output from an image sensor that has a first imaging area for imaging a subject and a second imaging area for imaging the subject, that is capable of setting a first frame rate for the first imaging area, and that is capable of setting a second frame rate that is faster than the first frame rate for the second imaging area.The program causes the processor to perform a generation process for each of the multiple frames, which generates a reference frame to be referenced by a frame that is input temporally later than the frame based on difference data between the frame and a predicted frame that predicts the frame, and the predicted frame, and a motion compensation process that performs motion compensation on a frame among the multiple frames that was imaged at the second frame rate, using a specific reference frame from the reference frames generated by the generation process. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a cross-sectional view of a stacked imaging device. [Figure 2] FIG. 2 is a diagram illustrating the pixel arrangement of the imaging chip. [Figure 3] FIG. 3 is a circuit diagram of the imaging chip. [Figure 4] FIG. 4 is a block diagram showing an example of the functional configuration of the imaging element. [Figure 5]FIG. 5 is an explanatory diagram illustrating an example of a block configuration of an electronic device. [Figure 6] FIG. 6 is an explanatory diagram showing an example of the structure of a moving image file. [Figure 7] FIG. 7 is an explanatory diagram showing the relationship between the imaging surface and the subject image. [Figure 8] FIG. 8 is an explanatory diagram showing a specific example of the structure of a moving image file. [Figure 9] FIG. 9 is an explanatory diagram of an example of video compression according to the first embodiment. [Figure 10] FIG. 10 is an explanatory diagram showing an image processing example 1 in the moving image compression shown in FIG. [Figure 11] FIG. 11 is an explanatory diagram showing a second example of image processing in the moving image compression shown in FIG. [Figure 12] FIG. 12 is a block diagram illustrating an example of the configuration of the control unit illustrated in FIG. [Figure 13] FIG. 13 is a block diagram illustrating an example of the configuration of the compression unit. [Figure 14] FIG. 14 is a sequence diagram illustrating an example of an operation processing procedure of the control unit. [Figure 15] FIG. 15 is a flowchart showing a detailed example of the procedure of the setting process (steps S1404 and S1410) shown in FIG. [Figure 16] FIG. 16 is a flowchart illustrating a detailed example of the processing procedure of the additional information setting process (step S1505) shown in FIG. [Figure 17] FIG. 17 is a flowchart illustrating an example of a detailed processing procedure of the moving image file generation processing. [Figure 18] FIG. 18 is a flowchart showing a detailed example of the processing procedure of the image processing (steps S1413 and S1415) shown in FIG. [Figure 19] FIG. 19 is a flowchart illustrating an example of a compression control process procedure of the first compression control method performed by the compression control unit. [Figure 20] FIG. 20 is a flowchart illustrating an example of a motion detection process procedure according to the first compression control method performed by the motion detection unit. [Figure 21]FIG. 21 is a flowchart illustrating an example of a motion compensation process procedure according to the first compression control method performed by the motion compensation unit. [Figure 22] FIG. 22 is a flowchart illustrating an example of a compression control process procedure of the second compression control method performed by the compression control unit. [Figure 23] FIG. 23 is a flowchart illustrating an example of a motion detection process procedure according to the second compression control method performed by the motion detection unit. [Figure 24] FIG. 24 is a flowchart illustrating an example of a motion compensation process procedure according to the second compression control method performed by the motion compensation unit. [Figure 25] FIG. 25 is an explanatory diagram showing a specific processing flow of the moving image processing example 1 shown in FIG. [Figure 26] FIG. 26 is an explanatory diagram illustrating a first example of compositing frames at 60 [fps] according to the second embodiment. [Figure 27] FIG. 27 is an explanatory diagram illustrating a second example of compositing frames at 60 [fps] according to the second embodiment. [Figure 28] FIG. 28 is an explanatory diagram illustrating a fourth example of compositing frames at 60 [fps] according to the second embodiment. [Figure 29] FIG. 29 is a flowchart illustrating a first example of a synthesis process procedure according to the first example of frame synthesis performed by the image processing unit. [Figure 30] FIG. 30 is a flowchart illustrating a second example of a synthesis process procedure according to the second example of frame synthesis performed by the image processing unit. [Figure 31] FIG. 31 is a flowchart illustrating a third example of a synthesis process procedure according to the third example of frame synthesis performed by the image processing unit. [Figure 32] FIG. 32 is a flowchart illustrating a fourth example of a synthesis process procedure according to the fourth example of frame synthesis performed by the image processing unit. [Figure 33] FIG. 33 is an explanatory diagram illustrating an example of compositing frames at 60 [fps] according to the third embodiment. [Figure 34] FIG. 34 is an explanatory diagram showing the correspondence between the setting of the imaging area and the image area of ​​the frame. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Example of imaging element configuration> First, we will explain a stacked imaging element to be mounted on an electronic device. This stacked imaging element is described in Japanese Patent Application No. 2012-139026, previously filed by the applicant of the present application. The electronic device is, for example, an imaging device such as a digital camera or a digital video camera.

[0013] 1 is a cross-sectional view of a stacked imaging element 100. The stacked imaging element (hereinafter simply referred to as "imaging element") 100 includes a back-illuminated imaging chip (hereinafter simply referred to as "imaging chip") 113 that outputs pixel signals corresponding to incident light, a signal processing chip 111 that processes the pixel signals, and a memory chip 112 that stores the pixel signals. The imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by conductive bumps 109 such as Cu.

[0014] As shown in FIG. 1, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. In this embodiment, the surface of the imaging chip 113 on which the incident light is incident is referred to as the back surface. As shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the front direction on the paper, perpendicular to the Z axis and the X axis, is the positive Y axis. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood, based on the coordinate axes in FIG. 1.

[0015] An example of the imaging chip 113 is a back-illuminated MOS (Metal Oxide Semiconductor) image sensor. A PD (Photodiode) layer 106 is arranged on the back side of a wiring layer 108. The PD layer 106 has a plurality of PDs 104 arranged two-dimensionally and accumulating electric charges according to incident light, and transistors 105 provided corresponding to the PDs 104.

[0016] A color filter 102 is provided on the incident side of the PD layer 106, on which incident light is incident, via a passivation film 103. There are multiple types of color filters 102 that transmit different wavelength ranges, and each has a specific arrangement corresponding to each PD 104. The arrangement of the color filters 102 will be described later. A set of a color filter 102, a PD 104, and a transistor 105 forms one pixel.

[0017] A microlens 101 is provided corresponding to each pixel on the incident light side of the color filter 102. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0018] The wiring layer 108 has wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multi-layered, and may be provided with passive elements and active elements.

[0019] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0020] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0021] The bonding between the bumps 109 is not limited to Cu bump bonding by solid-phase diffusion, but may also employ micro-bump bonding by solder melting. For example, it is sufficient to provide approximately one bump 109 per block, as will be described later. Therefore, the size of the bumps 109 may be larger than the pitch of the PDs 104. Furthermore, in a peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bumps 109 corresponding to the pixel region may also be provided.

[0022] The signal processing chip 111 has through-silicon vias (TSVs) 110 that connect circuits provided on the front and back surfaces of the chip to each other. The TSVs 110 are preferably provided in the peripheral region. The TSVs 110 may also be provided in the peripheral region of the imaging chip 113 and the memory chip 112.

[0023] 2 is a diagram illustrating the pixel arrangement of the imaging chip 113. In particular, it shows the imaging chip 113 observed from the back side. (a) is a plan view schematically showing an imaging surface 200, which is the back side of the imaging chip 113, and (b) is an enlarged plan view of a partial region 200a of the imaging surface 200. As shown in (b), a large number of pixels 201 are arranged two-dimensionally on the imaging surface 200.

[0024] Each pixel 201 has a color filter (not shown). The color filters are of three types: red (R), green (G), and blue (B), and the notations "R," "G," and "B" in (b) indicate the type of color filter that the pixel 201 has. As shown in (b), the pixels 201 equipped with such color filters are arranged in a so-called Bayer array on the imaging surface 200 of the image sensor 100.

[0025] The pixel 201 having a red filter photoelectrically converts light in the red wavelength band of the incident light and outputs a light reception signal (photoelectric conversion signal). Similarly, the pixel 201 having a green filter photoelectrically converts light in the green wavelength band of the incident light and outputs a light reception signal. Furthermore, the pixel 201 having a blue filter photoelectrically converts light in the blue wavelength band of the incident light and outputs a light reception signal.

[0026] The image sensor 100 is configured so that each unit group 202, each consisting of four pixels 201 (2 pixels by 2 pixels) adjacent to each other, can be individually controlled. For example, when charge accumulation starts simultaneously in two different unit groups 202, charge readout, i.e., light reception signal readout, occurs 1 / 30 seconds after the start of charge accumulation in one unit group 202, and charge readout occurs 1 / 15 seconds after the start of charge accumulation in the other unit group 202. In other words, the image sensor 100 can set a different exposure time (charge accumulation time, so-called shutter speed) for each unit group 202 in one image capture.

[0027] In addition to the exposure time described above, the image sensor 100 can also vary the amplification factor of the image signal (so-called ISO sensitivity) for each unit group 202. The image sensor 100 can change the timing for starting charge accumulation and the timing for reading out the light reception signal for each unit group 202. In other words, the image sensor 100 can change the frame rate for capturing a moving image for each unit group 202.

[0028] To summarize the above, the image sensor 100 is configured to be able to vary imaging conditions such as exposure time, amplification factor, and frame rate for each unit group 202. For example, if a readout line (not shown) for reading out imaging signals from a photoelectric conversion unit (not shown) included in the pixel 201 is provided for each unit group 202 and imaging signals can be read out independently for each unit group 202, it is possible to vary the exposure time (shutter speed) for each unit group 202.

[0029] Furthermore, if an amplifier circuit (not shown) that amplifies the image signal generated by the photoelectrically converted charge is provided independently for each unit group 202 and the amplification factor of the amplifier circuit is configured to be controllable independently for each amplifier circuit, the signal amplification factor (ISO sensitivity) can be made different for each unit group 202.

[0030] Furthermore, imaging conditions that can be varied for each unit group 202 include, in addition to the imaging conditions described above, the frame rate, gain, resolution (thinning rate), the number of rows or columns for adding pixel signals, the charge accumulation time or number of accumulations, the number of digitization bits, etc. Furthermore, the control parameters may be parameters for image processing after image signals are acquired from the pixels.

[0031] In addition, the imaging conditions can be controlled by, for example, providing the image sensor 100 with a liquid crystal panel having sections (each section corresponding to one unit group 202) that can be controlled independently for each unit group 202, and using this as a neutral density filter that can be turned on and off, thereby making it possible to control the brightness (aperture value) for each unit group 202.

[0032] The number of pixels 201 constituting the unit group 202 does not have to be the above-mentioned 2×2=4 pixels. The unit group 202 only needs to have at least one pixel 201, and conversely, the unit group 202 may have more than four pixels 201.

[0033] 3 is a circuit diagram of the imaging chip 113. In FIG. 3, a rectangle surrounded by a dotted line representatively represents a circuit corresponding to one pixel 201. Furthermore, a rectangle surrounded by a dashed line corresponds to one unit group 202 (202-1 to 202-4). Note that at least a part of the transistors described below corresponds to the transistor 105 in FIG. 1.

[0034] As described above, the reset transistors 303 of the pixels 201 are turned on / off for each unit group 202. The transfer transistors 302 of the pixels 201 are also turned on / off for each unit group 202. In the example shown in Fig. 3, a reset wiring 300-1 is provided for turning on / off the four reset transistors 303 corresponding to the upper left unit group 202-1, and a TX wiring 307-1 is also provided for supplying transfer pulses to the four transfer transistors 302 corresponding to the same unit group 202-1.

[0035] Similarly, a reset wiring 300-3 for turning on / off the four reset transistors 303 corresponding to the lower left unit group 202-3 is provided separately from the reset wiring 300-1. Also, a TX wiring 307-3 for supplying transfer pulses to the four transfer transistors 302 corresponding to the same unit group 202-3 is provided separately from the TX wiring 307-1.

[0036] Similarly, for the upper right unit group 202-2 and the lower right unit group 202-4, a reset line 300-2 and a TX line 307-2, and a reset line 300-4 and a TX line 307-4 are provided in each unit group 202, respectively.

[0037] The 16 PDs 104 corresponding to each pixel 201 are connected to the corresponding transfer transistors 302. A transfer pulse is supplied to the gate of each transfer transistor 302 via the TX wiring for each unit group 202. The drain of each transfer transistor 302 is connected to the source of the corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of the corresponding amplification transistor 304.

[0038] The drains of the reset transistors 303 are commonly connected to a Vdd wiring 310 to which a power supply voltage is supplied. A reset pulse is supplied to the gate of each reset transistor 303 via the reset wiring for each unit group 202.

[0039] The drains of the amplifier transistors 304 are commonly connected to a Vdd line 310 to which a power supply voltage is supplied. The source of each amplifier transistor 304 is connected to the drain of the corresponding selection transistor 305. The gate of each selection transistor 305 is connected to a decoder line 308 to which a selection pulse is supplied. The decoder line 308 is provided independently for each of the 16 selection transistors 305.

[0040] The sources of the selection transistors 305 are connected to a common output wiring 309. A load current source 311 supplies a current to the output wiring 309. That is, the output wiring 309 for the selection transistors 305 is formed by a source follower. The load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0041] Here, the flow from the start of charge accumulation to pixel output after accumulation is completed will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring for each unit group 202, and at the same time a transfer pulse is applied to the transfer transistor 302 through the TX wiring for each unit group 202 (202-1 to 202-4), the potentials of the PD 104 and the floating diffusion FD are reset for each unit group 202.

[0042] When the transfer pulse is released, each PD 104 converts the incident light it receives into electric charges and stores them. After that, when the transfer pulse is applied again without the reset pulse being applied, the stored electric charges are transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after the electric charges are stored.

[0043] When a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, a fluctuation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 via the amplification transistor 304 and the selection transistor 305. As a result, a pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0044] As described above, the reset wiring and TX wiring are common to the four pixels forming the unit group 202. That is, the reset pulse and transfer pulse are each applied simultaneously to the four pixels in the unit group 202. Therefore, all the pixels 201 forming a certain unit group 202 start and end charge accumulation at the same timing. However, pixel signals corresponding to the accumulated charges are selectively output from the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305.

[0045] In this way, the charge accumulation start timing can be controlled for each unit group 202. In other words, different unit groups 202 can capture images at different timings.

[0046] 4 is a block diagram showing an example of the functional configuration of the image sensor 100. An analog multiplexer 411 sequentially selects the 16 PDs 104 that form a unit group 202 and outputs the pixel signals from each of the 16 PDs 104 to the output wiring 309 provided corresponding to the unit group 202. The multiplexer 411 is formed on the image sensor chip 113 together with the PDs 104.

[0047] The pixel signals output via the multiplexer 411 undergo correlated double sampling (CDS) and analog-to-digital (A / D) conversion by a signal processing circuit 412 that performs CDS and A / D conversion and is formed in the signal processing chip 111. The A / D converted pixel signals are passed to a demultiplexer 413 and stored in pixel memories 414 corresponding to the respective pixels. The demultiplexer 413 and pixel memories 414 are formed in the memory chip 112.

[0048] The arithmetic circuit 415 processes the pixel signals stored in the pixel memory 414 and passes them to a downstream image processing unit. The arithmetic circuit 415 may be provided in the signal processing chip 111 or in the memory chip 112. Note that although Fig. 4 shows connections for four unit groups 202, in reality, these exist for every four unit groups 202 and operate in parallel.

[0049] However, it is not necessary for there to be an arithmetic circuit 415 for each of the four unit groups 202; for example, one arithmetic circuit 415 may process the values ​​of the pixel memories 414 corresponding to each of the four unit groups 202 sequentially by referring to them in order.

[0050] As described above, output wiring 309 is provided corresponding to each unit group 202. Since the imaging element 100 has the imaging chip 113, the signal processing chip 111, and the memory chip 112 stacked on top of each other, by using the bumps 109 for electrical connection between the chips for these output wiring 309, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0051] <Example of electronic device block configuration> 5 is an explanatory diagram showing an example block configuration of an electronic device. The electronic device 500 is, for example, a lens-integrated camera. The electronic device 500 includes an imaging optical system 501, an imaging element 100, a control unit 502, an LCD monitor 503, a memory card 504, an operation unit 505, a DRAM 506, a flash memory 507, and an audio recording unit 508. The control unit 502 includes a compression unit that compresses video data, as described below. Therefore, the electronic device 500 including at least the control unit 502 serves as a video compression device.

[0052] The imaging optical system 501 is made up of a plurality of lenses, and forms a subject image on the imaging surface 200 of the image sensor 100. For convenience, the imaging optical system 501 is illustrated as a single lens in FIG.

[0053] The imaging element 100 is, for example, an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and captures an image of a subject formed by an imaging optical system 501 and outputs an imaging signal. The control unit 502 is an electronic circuit that controls each unit of the electronic device 500, and is composed of a processor and its peripheral circuits.

[0054] A predetermined control program is written in advance in flash memory 507, which is a non-volatile storage medium. Control unit 502 controls each unit by reading and executing the control program from flash memory 507. This control program uses DRAM 506, which is a volatile storage medium, as a working area.

[0055] The liquid crystal monitor 503 is a display device that uses a liquid crystal panel. The control unit 502 causes the image sensor 100 to repeatedly capture an image of the subject at predetermined intervals (for example, 1 / 60th of a second). Then, various types of image processing are performed on the image signal output from the image sensor 100 to create a so-called through image, which is displayed on the liquid crystal monitor 503. In addition to the through image, the liquid crystal monitor 503 also displays, for example, a setting screen for setting image capturing conditions.

[0056] The control unit 502 creates an image file (described later) based on the imaging signal output from the imaging element 100, and records the image file on a portable recording medium, such as a memory card 504. The operation unit 505 has various operation members such as push buttons, and outputs operation signals to the control unit 502 in response to the operation of these operation members.

[0057] Recording unit 508 is configured by, for example, a microphone, and converts environmental sounds into audio signals and inputs them to control unit 502. Note that control unit 502 may record the video file on a recording medium (not shown) built into electronic device 500, such as a hard disk, instead of recording the video file on memory card 504, which is a portable recording medium.

[0058] <Video file configuration example> 6 is an explanatory diagram showing an example of the structure of a moving image file. A moving image file 600 is generated during compression processing by a compression unit 902 (described later) in the control unit 502, and is stored in the memory card 504, DRAM 506, or flash memory 507. The moving image file 600 is composed of two blocks: a header section 601 and a data section 602. The header section 601 is the block located at the beginning of the moving image file 600. The header section 601 stores a file basic information area 611, a mask area 612, and an imaging information area 613 in the order described above.

[0059] The file basic information area 611 records, for example, the size and offset of each section (header section 601, data section 602, mask area 612, imaging information area 613, etc.) in the video file 600. The mask area 612 records imaging condition information and mask information, which will be described later. The imaging information area 613 records information related to imaging, such as the model name of the electronic device 500 and information about the imaging optical system 501 (for example, information about optical characteristics such as aberration). The data section 602 is a block located after the header section 601, and records image information, audio information, etc.

[0060] <Relationship between the imaging surface and the subject image> 7 is an explanatory diagram showing the relationship between the imaging surface and a subject image. (a) schematically shows the imaging surface 200 (imaging range) of the image sensor 100 and a subject image 701. In (a), the control unit 502 captures the subject image 701. The imaging in (a) may also serve as imaging performed to create, for example, a live view image (a so-called through image).

[0061] The control unit 502 executes a predetermined image analysis process on the subject image 701 obtained by capturing (a). The image analysis process is a process of detecting a main subject region and a background region, for example, by using a well-known subject detection technology (a technology that calculates feature amounts to detect an area where a predetermined subject exists). The image analysis process divides the imaging plane 200 into a main subject region 702 where the main subject exists and a background region 703 where the background exists.

[0062] In (a), the main subject region 702 is shown as a region that roughly includes the subject image 701, but the main subject region 702 may have a shape that follows the outline of the subject image 701. In other words, the main subject region 702 may be set so as to include as little as possible of anything other than the subject image 701.

[0063] The control unit 502 sets different imaging conditions for each unit group 202 in the main subject region 702 and each unit group 202 in the background region 703. For example, a faster shutter speed is set for each of the former unit groups 202 than for each of the latter unit groups 202. In this way, image blurring is less likely to occur in the main subject region 702 when imaging (c) is performed after imaging (a).

[0064] Furthermore, when the main subject region 702 is backlit due to the influence of a light source such as the sun present in the background region 703, the control unit 502 sets a relatively high ISO sensitivity and a slow shutter speed for each of the former unit groups 202. Furthermore, the control unit 502 sets a relatively low ISO sensitivity and a fast shutter speed for each of the latter unit groups 202. In this way, in the image capture of (c), it is possible to prevent crushed shadows in the main subject region 702 that is backlit and blown out highlights in the background region 703 that is bright.

[0065] The image analysis process may be a process different from the process of detecting the main subject region 702 and the background region 703 described above. For example, it may be a process of detecting parts of the entire imaging surface 200 that are brighter than a certain level (parts that are too bright) and parts that are less than a certain level (parts that are too dark). When the image analysis process is such a process, the control unit 502 sets the shutter speed and ISO sensitivity for the unit groups 202 included in the former region so that the exposure value (Ev value) is lower than that of the unit groups 202 included in the other regions.

[0066] Furthermore, the control unit 502 sets the shutter speed and ISO sensitivity for the unit groups 202 included in the latter region so that the exposure value (Ev value) is higher than that of the unit groups 202 included in the other regions. In this way, the dynamic range of the image obtained by capturing (c) can be wider than the original dynamic range of the image sensor 100.

[0067] 7(b) shows an example of mask information 704 corresponding to the imaging plane 200 shown in (a). A "1" is stored at the position of the unit group 202 belonging to the main subject region 702, and a "2" is stored at the position of the unit group 202 belonging to the background region 703.

[0068] The control unit 502 performs image analysis processing on the image data of the first frame to detect a main subject region 702 and a background region 703. As a result, the frame captured in (a) is divided into a main subject region 702 and a background region 703 as shown in (c). The control unit 502 sets different imaging conditions for each unit group 202 in the main subject region 702 and each unit group 202 in the background region 703, performs the imaging in (c), and creates image data. An example of mask information 704 at this time is shown in (d).

[0069] The mask information 704 (b) corresponding to the imaging result of (a) and the mask information 704 (d) corresponding to the imaging result of (c) are captured at different times (there is a time difference), and therefore, for example, if the subject is moving or if the user moves the electronic device 500, these two pieces of mask information 704 will have different contents. In other words, the mask information 704 is dynamic information that changes over time. Therefore, different imaging conditions are set for each frame in a certain unit group 202.

[0070] <Example of video file> 8 is an explanatory diagram showing a specific example of the configuration of the video file 600. In the mask area 612, identification information 801, image capture condition information 802, and mask information 704 are recorded in the order described above.

[0071] The identification information 801 indicates that this moving image file 600 was created using a multi-frame rate moving image capturing function, which is a function for capturing moving images using the image sensor 100 with multiple frame rates set.

[0072] The imaging condition information 802 is information that indicates what kind of use (purpose, role) exists in the unit group 202. For example, as described above, when the imaging plane 200 (FIG. 7(a)) is divided into a main subject region 702 and a background region 703, each unit group 202 belongs to either the main subject region 702 or the background region 703.

[0073] In other words, the imaging condition information 802 is information that indicates that when this moving image file 600 was created, the unit group 202 had two uses, for example, "shooting a moving image of the main subject area at 60 fps" and "shooting a moving image of the background area at 30 fps," and that a unique number was assigned to each of these uses. For example, the number 1 is assigned to the use of "shooting a moving image of the main subject area at 60 fps," and the number 2 is assigned to the use of "shooting a moving image of the background area at 30 fps."

[0074] The mask information 704 is information that indicates the use (purpose, role) of each unit group 202. The mask information 704 is defined as "information that expresses the numbers assigned to the imaging condition information 802 in the form of a two-dimensional map in accordance with the positions of the unit groups 202." In other words, when the unit groups 202 arranged two-dimensionally are specified by two-dimensional coordinates (x, y) using two integers x and y, the use of the unit group 202 present at the position (x, y) is expressed by the number present at the position (x, y) of the mask information 704.

[0075] For example, if the number "1" is entered at the coordinate (3, 5) position in the mask information 704, it can be seen that the unit group 202 located at the coordinate (3, 5) has been assigned the purpose of "capturing the main subject area at 60 [fps]." In other words, it can be seen that the unit group 202 located at the coordinate (3, 5) belongs to the main subject area 702.

[0076] It should be noted that the mask information 704 is dynamic information that changes for each frame, and is therefore recorded for each frame, that is, for each data block Bi (described later), during the compression process (not shown).

[0077] Data blocks B1 to Bn are stored as moving image data in the order of capture for each frame F (F1 to Fn) in the data section 602. Each data block Bi (i is an integer satisfying the condition 1≦i≦n) includes mask information 704, image information 811, a Tv value map 812, an Sv value map 813, a Bv value map 814, Av value information 815, audio information 816, and additional information 817.

[0078] Image information 811 is information in which the imaging signal output from the imaging element 100 by the imaging of FIG. 7C is recorded in a form before various image processing is performed, and is so-called RAW image data.

[0079] The Tv value map 812 is information in which the Tv values ​​indicating the shutter speeds set for each unit group 202 are expressed in the form of a two-dimensional map according to the positions of the unit groups 202. For example, the shutter speed set for the unit group 202 located at the coordinates (x, y) can be determined by checking the Tv value stored at the coordinates (x, y) of the Tv value map 812.

[0080] The Sv value map 813 is information in which the Sv value indicating the ISO sensitivity set for each unit group 202 is expressed in the form of a two-dimensional map, similar to the Tv value map 812 .

[0081] The Bv value map 814 is information that represents the subject brightness measured for each unit group 202 during the imaging of Figure 7 (c), i.e., the Bv value representing the brightness of the subject light incident on each unit group 202, in the form of a two-dimensional map, similar to the Tv value map 812.

[0082] The Av value information 815 is information that represents the aperture value at the time of capturing the image in (c) of Fig. 7. Unlike the Tv value, Sv value, and Bv value, the Av value is not a value that exists for each unit group 202. Therefore, unlike the Tv value, Sv value, and Bv value, the Av value stores only a single value, and is not information in which multiple values ​​are mapped two-dimensionally.

[0083] To facilitate video playback, the audio information 816 is divided into information for each frame, multiplexed with the data block Bi, and stored in the data section 602. Note that the audio information 816 may be multiplexed not for each frame, but for each predetermined number of frames. Note that the audio information 816 does not necessarily have to be included.

[0084] The additional information 817 is information that expresses, in the form of a two-dimensional map, the frame rate set for each unit group 202 when capturing the image of (c) in Fig. 7. The setting of the additional information 817 will be described later with reference to Figs. 14 and 15. The additional information 817 may be stored in the frame F, or may be stored in a cache memory of the processor 1201, which will be described later. In particular, when performing compression processing in real time, it is preferable to use a cache memory from the viewpoint of high-speed processing.

[0085] As described above, the control unit 502 performs imaging using the multi-frame rate video imaging function, and records on the memory card 504 a video file 600 in which image information 811 generated by the image sensor 100, in which imaging conditions can be set for each unit group 202, and data related to the imaging conditions for each unit group 202 (imaging condition information 802, mask information 704, Tv value map 812, Sv value map 813, Bv value map 814, etc.) are associated.

[0086] An embodiment of video compression using the image sensor 100 described above will now be described. [Example]

[0087] <Video compression example> 9 is an explanatory diagram illustrating an example of video compression according to the first embodiment. The electronic device 500 includes the image sensor 100 described above and a control unit 502. The control unit 502 includes an image processing unit 901 and a compression unit 902. As described above, the image sensor 100 has a plurality of imaging regions for capturing images of a subject. The imaging region is a set of at least one pixel, for example, one or more unit groups 202 described above. A frame rate can be set for each unit group 202 in the imaging region.

[0088] Here, it is assumed that a first frame rate (e.g., 30 [fps]) is set for a first imaging area among the imaging areas, and a second frame rate (e.g., 60 [fps]) faster than the first frame rate is set for a second imaging area other than the first imaging area. Note that the values ​​of the first frame rate and the second frame rate are merely examples, and other values ​​may be used as long as the second frame rate is faster than the first frame rate.

[0089] The image sensor 100 captures an image of a subject and outputs an image signal (for convenience, this is referred to as first moving image data 910 including multiple frames in FIG. 9) to the image processing unit 901. Within a frame, an area of ​​image data captured in a certain imaging area of ​​the image sensor 100 is referred to as an image area. Furthermore, of the image data of a first image area a1 (shaded) captured at a first frame rate (30 fps) set in the first imaging area and the image data of a second image area a2 (blacked out) captured at a second frame rate (60 fps) set in the second imaging area, a frame including at least the image data of the first image area a1 is referred to as a first frame.

[0090] Specifically, for example, if a specific subject (train) is not detected in the first frame, the entire frame (landscape) becomes a first image area a1 captured at a first frame rate (30 fps). Also, if a specific subject is detected in the first frame, the portion of the entire frame capturing the specific subject (train) becomes a second image area a2 captured at a second frame rate (60 fps), and the remaining portion (landscape) becomes the first image area a1 captured at the first frame rate (30 fps).

[0091] Moreover, a frame of only the second image area a2 captured at the second frame rate (60 [fps]) set for the second imaging area is referred to as a second frame.

[0092] For example, when an imaging area is made up of one unit group 202 (2×2 pixels), the size of the corresponding image area is also the size of the unit group 202. Similarly, when an imaging area is made up of 2×2 unit groups 202 (4×4 pixels), the size of the corresponding image area is also the size of the 2×2 unit group 202.

[0093] 9, of the first moving image data 910 output from the image sensor 100, a frame including the first image area a1 is referred to as the first frame, and a frame including only the image of the specific subject in the second image area a2 is referred to as the second frame. Note that there may be three or more image areas. In this case, a frame rate different from the first frame rate and the second frame rate can be set for the third and subsequent image areas.

[0094] The image processing unit 901 performs image processing on moving image data (hereinafter referred to as first moving image data) 910 input from the image sensor 100. Specifically, for example, the image processing unit 901 references a first frame, which is one frame temporally before the second frame, and copies, that is, synthesizes, the first frame onto the second frame that is the reference source. The synthesized frame is referred to as a third frame. The third frame is a frame in which the image of a specific object in the second frame is superimposed on the image of the object in the first frame. The image processing unit 901 outputs moving image data (hereinafter referred to as second moving image data) 920, which includes the first frame captured at 30 [fps] and the third frame, which is the synthesized frame, to the compression unit 902.

[0095] The above-described first video data 910 cannot be compressed as is by the compression unit 902. Therefore, when a frame sequence contains a mixture of first and second frames, the image processing unit 901 generates second video data 920 that the compression unit 902 can operate on. This allows the general-purpose compression unit 902 to compress the second video data 920 in the same way as normal video data.

[0096] The compression unit 902 compresses the second moving image data 920 input from the image processing unit 901. The compression unit 902 performs compression by, for example, hybrid coding that combines motion compensation inter-frame prediction (MC), discrete cosine transform (DCT), and entropy coding.

[0097] Compression unit 902 performs compression processing that does not require motion detection or motion compensation on the shaded first image area a1 of the first and third frames that make up second video data 920, and compresses the blackened second image area a2 of the specific subject image using the hybrid coding described above. In this way, motion detection and motion compensation are not performed on first image area a1 other than the specific subject image, thereby reducing the processing load of video compression.

[0098] FIG. 10 is an explanatory diagram showing image processing example 1 in the video compression shown in FIG. 9. In image processing example 1, electronic device 500 captures a moving train as a specific subject while capturing fixed-point images of a landscape including rice fields, mountains, and the sky. The specific subject, the train, is identified using the well-known subject detection technology described above. The captured frames are frames F1, F2-60, F3, F4-60, and F5 in chronological order. Here, the train is assumed to travel from right to left within frames F1, F2-60, F3, F4-60, and F5.

[0099] Frames F1, F3, and F5 are first frames containing image data of a first image area a1 captured at a first frame rate of 30 fps and image data of a second image area a2 captured at a second frame rate of 60 fps. Frames F2-60 and F4-60 are second frames containing image data of a second image area a2 captured at a second frame rate of 60 fps.

[0100] Specifically, for example, frames F1, F3, and F5 are first frames in which a landscape including rice fields, mountains, and the sky is captured in the first image area a1, and a running train is captured as a specific subject in the second image area a2. Frames F2-60 and F4-60 are frames in which a train is captured in the second image area a2. That is, in frames F1, F2-60, F3, F4-60, and F5, the image data of the second image area a2 in which the train is captured is image data captured in the second image area (60 [fps]), and in frames F1, F3, and F5, the image data of the first image area a1 in which the landscape is captured is image data captured in the first image area (30 [fps]). Because the first image area is captured at the first frame rate, nothing is captured in the first image area a1 of frames F2-60 and F4-60, which are captured at the second frame rate.

[0101] Frames F1, F2-60, F3, F4-60, ... correspond to the first moving image data 910 described above, and cannot be compressed as is by the compression unit 902. If the frame sequence contains a mixture of first and second frames, it is necessary to generate second moving image data 920 that the compression unit 902 can use.

[0102] The image processing unit 901 copies the image data (the train) of the second image area a2 of frame F2-60 to the image data (the scenery excluding the train) of the first image area a1 of frame F1, which is temporally immediately preceding frame F2-60, thereby generating frame F2, which is the third frame.

[0103] Similarly, for frame F4-60, the image processing unit 901 copies the image data (the train) of the second image region a2 of frame F4-60 to the image data (the scenery excluding the train) of the first image region a1 of frame F3, which is temporally immediately preceding frame F4-60. In this way, the image processing unit 901 generates frame F4, which is the third frame. The image processing unit 901 then outputs second moving image data 920 including frames F1 to F5.

[0104] In this way, by interpolating the image data of the first image region a1 of frames F2-60 and F4-60 with the frames F1 and F3 that are temporally immediately preceding at the first frame rate, it is possible to make the difference between frames F1 and F2 and the difference between frames F3 and F4 in the first image region a1 almost zero. This makes it possible to compress a frame sequence in which the first and second frames are mixed using the conventional compression unit 902. Furthermore, it is possible to reduce the processing load of the compression process.

[0105] In frame F2, the image data of the first image area a1 of frame F1 (the scenery excluding the train) is copied. Therefore, the part that was originally the second image area a2 of frame F1 (the end of the train) is not copied to frame F2. As a result, frame F2 has a range Da1 where nothing is output.

[0106] Similarly, in frame F4, the image data of the first image area a1 of frame F3 (the scenery excluding the train) is copied. Therefore, the part that was originally the second image area a2 of frame F3 (the end of the train) is not copied to frame F4. As a result, frame F4 has a range Da3 where nothing is output.

[0107] In the first embodiment, the image processing unit 901 may fill the ranges Da1 and Da3 with a specific color (for example, white, black, or gray) or may perform demosaicing using surrounding pixels, thereby enabling video compression and reproducing frames F2, F4, ... with less unnaturalness.

[0108] 11 is an explanatory diagram showing an image processing example 2 in the video compression shown in FIG. 9. In image processing example 2, electronic device 500 is, for example, a drive recorder, and captures a vehicle (preceding vehicle) traveling ahead and the scenery. In this case, the preceding vehicle is a specific subject to be tracked, and the scenery changes as the vehicle travels. The captured frames are frames F6, F7-60, F8, F9-60, and F10 in chronological order.

[0109] Frames F6, F8, and F10 are first frames containing image data of a first image area a1 captured at a first frame rate of 30 fps and image data of a second image area a2 captured at a second frame rate of 60 fps. Frames F7-60 and F9-60 are second frames containing image data of a second image area a2 captured at a second frame rate of 60 fps.

[0110] Specifically, for example, frames F6, F8, and F10 are first frames in which a preceding vehicle is captured in the first image area a1 and a changing landscape is captured in the second image area a2. Frames F7-60 and F9-60 are frames in which a landscape is captured in the second image area a2. That is, in frames F6, F7-60, F8, F9-60, and F10, the image data of the second image area a2 in which the landscape is captured is image data captured in the second imaging area (60 [fps]), and in frames F6, F8, and F10, the image data of the first image area a1 in which the preceding vehicle is captured is image data captured in the first imaging area (30 [fps]). Because the first imaging area is captured at the first frame rate, nothing is captured in the first image area a1 of frames F7-60 and F9-60, which are captured at the second frame rate.

[0111] The image processing unit 901 copies the image data (scenery) of the second image area a2 of frame F7-60 to the image data (the preceding vehicle excluding the scenery) of the first image area a1 of frame F6, which is temporally immediately preceding frame F7-60, thereby generating frame F7, which is the third frame.

[0112] Similarly, for frame F9, the image processing unit 901 copies the image data (scenery) of the second image area a2 of frame F9-60 to the image data (the preceding vehicle excluding the scenery) of the first image area a1 of frame F8, which is temporally immediately preceding frame F9-60. In this way, the image processing unit 901 generates frame F9, which is the third frame. The image processing unit 901 then outputs second moving image data 920 including frames F6 to F10.

[0113] In this way, by interpolating the image data of the first image region a1 in frames F7-60 and F9-60 with the frames F6 and F8 that are temporally immediately preceding at the first frame rate, it is possible to make the difference between frames F6 and F7 and the difference between frames F8 and F9 for the first image region a1 zero. This makes it possible to compress a frame sequence in which the first and second frames are mixed using the conventional compression unit 902. Furthermore, it is possible to reduce the processing load of the compression process.

[0114] The control unit 502 may perform the compression process of the second moving image data 920 in real time or batch processing. For example, the control unit 502 may temporarily store the first moving image data 910 and the second moving image data 920 from the image sensor 100, the preprocessing unit 900, or the image processing unit 901 in the memory card 504, the DRAM 506, or the flash memory 507, and then read out the first moving image data 910 and the second moving image data 920 (after the first moving image data 910 is converted into the second moving image data 920 by the image processing unit 901) automatically or when triggered by a user operation, and cause the compression unit 902 to perform the compression process.

[0115] <Configuration example of control unit 502> Fig. 12 is a block diagram showing an example of the configuration of the control unit 502 shown in Fig. 5. The control unit 502 has a preprocessing unit 1210, an image processing unit 901, an acquisition unit 1220, and a compression unit 902, and is composed of a processor 1201, a memory 1202, an integrated circuit 1203, and a bus 1204 connecting these.

[0116] The preprocessing unit 1210, image processing unit 901, acquisition unit 1220, and compression unit 902 may be realized by having the processor 1201 execute a program stored in the memory 1202, or may be realized by an integrated circuit 1203 such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). The processor 1201 may use the memory 1202 as a work area. The integrated circuit 1203 may use the memory 1202 as a buffer for temporarily storing various data including image data.

[0117] The pre-processing unit 1210 performs pre-processing of the image processing by the image processing unit 901 on the first moving image data 910 from the image sensor 100. Specifically, for example, the pre-processing unit 1210 has a detection unit 1211 and a setting unit 1212. The detection unit 1211 detects a specific subject using the well-known subject detection technique described above.

[0118] The setting unit 1212 assigns additional information 817 to each frame constituting the first moving image data 910 from the image sensor 100. The setting unit 1212 also changes the frame rate of the imaging area of ​​the image sensor 100, in which the specific subject is detected, from the first frame rate (for example, 30 [fps]) to the second frame rate (60 [fps]).

[0119] Specifically, for example, the setting unit 1212 detects a motion vector of the specific subject from the difference between the imaging area in the input frame where the specific subject is detected and the imaging area in the previously input frame where the specific subject is detected, and predicts the imaging area of ​​the specific subject in the next input frame. The setting unit 1212 outputs an instruction to the image sensor 100 to change the frame rate of the predicted imaging area to the second frame rate.

[0120] The image processing unit 901 performs image processing on each frame of the first moving image data 910 output from the pre-processing unit 1210. Specifically, the image processing unit 901 includes a specifying unit 1213 and a combining unit 1214, for example.

[0121] The identification unit 1213 identifies a difference region between the second image region a2 corresponding to the second imaging region in the first frame and the second image region a2 corresponding to the second imaging region in the second frame, based on the first frame (for example, frame F1 in FIG. 10) and the second frame (for example, frame F2-60 in FIG. 10) among the multiple frames of the first video data 910. The difference region between frames F1 and F2-60 is the first image region a1 behind the train in frame F2-60.

[0122] As shown in FIGS. 9 to 11, the synthesis unit 1214 copies and synthesizes a first frame (e.g., frame F1 in FIG. 10) containing image data of the temporally preceding first image region a1 onto a second frame (e.g., frame F2-60 in FIG. 10) containing only image data of the second image region a2, thereby generating a third frame (e.g., frame F2 in FIG. 10). The synthesis unit 1214 also copies image data of the second image region a2 (the tail end of the train) at the same position as the differential region of the first frame into the differential region (range Da1) identified by the identification unit 1213 (see the dotted circle in frame F2-60 in FIG. 10). This makes it possible to reduce the difference between the temporally consecutive first and third frames to approximately zero.

[0123] The acquisition unit 1220 stores the second moving image data 920 output from the image processing unit 901 in the memory 1202, and outputs the multiple frames included in the second moving image data 920 one frame at a time in chronological order to the compression unit 902 at a predetermined timing.

[0124] 9, the compression unit 902 compresses the input second moving image data 920. Specifically, for example, of the first and third frames constituting the second moving image data 920, the compression unit 902 performs compression processing that does not require motion detection or motion compensation on the image data of the first image region a1, and compresses the image data of the second image region a2 in which the specific subject is captured using the hybrid coding described above. In this way, motion detection and motion compensation are not performed on the regions other than the image of the specific subject, thereby reducing the processing load of moving image compression.

[0125] <Configuration example of compression unit 902> 13 is a block diagram showing an example configuration of the compression unit 902. As described above, the compression unit 902 compresses each frame of the second video data 920 by hybrid coding, which combines motion-compensated inter-frame prediction (MC), discrete cosine transform (DCT), and entropy coding.

[0126] The compression unit 902 includes a subtraction unit 1301, a DCT unit 1302, a quantization unit 1303, an entropy coding unit 1304, a code amount control unit 1305, an inverse quantization unit 1306, an inverse DCT unit 1307, a generation unit 1308, a frame memory 1309, a motion detection unit 1310, a motion compensation unit 1311, and a compression control unit 1312. The subtraction unit 1301 to the motion compensation unit 1311 have the same configuration as existing compressors.

[0127] Specifically, for example, the subtraction unit 1301 subtracts a predicted frame from an input frame, the predicted frame being output from the motion compensation unit 1311, and outputs the difference data. The DCT unit 1302 performs a discrete cosine transform on the difference data from the subtraction unit 1301.

[0128] The quantization unit 1303 quantizes the differential data that has been subjected to the discrete cosine transform. The entropy coding unit 1304 entropy codes the quantized differential data and also entropy codes the motion vectors from the motion estimation unit 1310.

[0129] The code amount control unit 1305 controls the quantization by the quantization unit 1303. The inverse quantization unit 1306 inverse quantizes the difference data quantized by the quantization unit 1303 to generate differential data that has been subjected to discrete cosine transform. The inverse DCT unit 1307 performs inverse discrete cosine transform on the inverse quantized differential data.

[0130] The generation unit 1308 adds the inverse discrete cosine transformed difference data to a predicted frame from the motion compensation unit 1311 to generate a reference frame to be referenced by a frame input temporally after the input frame. The frame memory 1309 holds the reference frame obtained from the generation unit 1308. The motion detection unit 1310 detects a motion vector using the input frame and the reference frame. The motion compensation unit 1311 generates a predicted frame using the reference frame and the motion vector.

[0131] Specifically, the motion compensation unit 1311 performs motion compensation on frames captured at the second frame rate, for example, by using a specific reference frame and a motion vector from among multiple reference frames stored in the frame memory 1309. By using the specific reference frame as the reference frame, it is possible to suppress high-load motion compensation that also uses reference frames other than the specific reference frame. Furthermore, by using a single reference frame obtained from the frame immediately preceding the input frame as the specific reference frame, it is possible to avoid high-load motion compensation and reduce the processing load of the motion compensation.

[0132] The compression control unit 1312 controls the motion detection unit 1310 and the motion compensation unit 1311. Specifically, for example, the compression control unit 1312 executes a first compression control method in which the motion detection unit 1310 sets a specific motion vector indicating that there is no motion, and a second compression control method in which motion detection itself is skipped.

[0133] In the first compression control method, the compression control unit 1312 controls the motion detection unit 1310 to set a specific motion vector indicating no motion for a first image region a1 captured at a first frame rate (for example, 30 fps) rather than detecting a motion vector, and output this to the motion compensation unit 1311, and to detect a motion vector for a second image region a2 captured at a second frame rate (for example, 60 fps), and output this to the motion compensation unit 1311. The specific motion vector is a motion vector whose direction is not specified and whose amount of motion is zero.

[0134] In this case, the compression control unit 1312 controls the motion compensation unit 1311 to perform motion compensation for the image data of the first image area a1 based on a specific motion vector and a reference frame, and to perform motion compensation for the image data of the second image area a2 based on the motion vector detected by the motion detection unit 1310.

[0135] For the second compression control method, the compression control unit 1312 controls the motion detection unit 1310 to not detect motion vectors for the first image area a1 captured at a first frame rate (e.g., 30 fps), but to detect motion vectors for the second image area a2 captured at a second frame rate (e.g., 60 fps).

[0136] In this case, the compression control unit 1312 controls the motion compensation unit 1311 to perform motion compensation on the image data of the first image region a1 based on the reference frame. That is, because there is no motion vector, the compression control unit 1312 controls the motion compensation unit 1311 to determine the reference frame for the image data of the first image region a1 as a predicted frame that predicts the frame temporally subsequent to the input frame. The compression control unit 1312 also controls the motion compensation unit 1311 to perform motion compensation on the image data of the second image region a2 based on the reference frame and the motion vector detected by the motion detection unit 1310.

[0137] According to the first compression control method, since the motion vector is a specific motion vector, motion detection in the first image region a1 is simplified, thereby reducing the processing load of video compression. Furthermore, according to the second compression control method, since motion detection itself is not performed for the first image region a1, the processing load of video compression is reduced more than with the first compression control method.

[0138] <Example of Operational Processing Procedure of Control Unit 502> Fig. 14 is a sequence diagram showing an example of the operation processing procedure of the control unit 502. For convenience of explanation, the acquisition unit 1220 is omitted from Fig. 14. The preprocessing unit 1210 sets the imaging condition for the entire imaging surface 200 of the image sensor 100 to a first frame rate (e.g., 30 [fps]) (step S1401), for example, when the user operates the operation unit 505, or automatically if a specific subject is not detected in step S1412 (step S1412: Yes).

[0139] Then, the pre-processing unit 1210 transmits a first frame rate setting instruction including the setting content of step S1401 to the image sensor 100 (step S1402). As a result, the image sensor 100 sets the imaging conditions for the entire imaging surface 200 to the first frame rate, and the image sensor 100 images the subject at the first frame rate and outputs the first video data 910 to the pre-processing unit 1210 (step S1403).

[0140] When the first moving image data 910 is input (step S1403), the pre-processing unit 1210 executes a setting process (step S1404). The setting process (step S1404) sets additional information 817 for each frame of the first moving image data 910. As described above, the additional information 817 is the frame rate set for the image area within the frame captured in each imaging area. For example, an image area to which the first frame rate (e.g., 30 [fps]) is added as the additional information 817 is recognized as a first image area a1, and an image area to which the second frame rate (e.g., 60 [fps]) is added as the additional information 817 is recognized as a second image area a2.

[0141] Furthermore, the pre-processing unit 1210 outputs the first moving image data 910, in which the additional information 817 has been added to each frame, to the image processing unit 901 (step S1405).

[0142] Furthermore, if an image area of ​​the next input frame having the second frame rate is not detected in the setting process (step S1404) (step S1406: No), the preprocessing unit 1210 waits for the input of the first moving image data 910 in step S1403. On the other hand, if an image area of ​​the next input frame having the second frame rate is detected in the setting process (step S1404) (step S1406: Yes), the preprocessing unit 1210 changes the setting of the second image area a2 including the specific subject to the second frame rate (for example, 60 [fps]) (step S1407).

[0143] Then, the pre-processing unit 1210 transmits a second frame rate setting instruction including the setting change content of step S1407 to the image sensor 100 (step S1408). As a result, the image sensor 100 sets the imaging condition of the second imaging region of the entire imaging surface 200 to the second frame rate, and the image sensor 100 images the subject at the first frame rate in the first imaging region and at the second frame rate in the second imaging region, and outputs the first moving image data 910 to the pre-processing unit 1210 (step S1409).

[0144] When the first moving image data 910 is input (step S1409), the pre-processing unit 1210 executes additional information setting processing (step S1410). The additional information setting processing (step S1410) is the same processing as the additional information setting processing (step S1404). Details of the additional information setting processing (step S1410) will be described later with reference to FIG. 15. The pre-processing unit 1210 outputs the first moving image data 910, in which the additional information 817 has been added to each frame, to the image processing unit 901 (step S1411).

[0145] If the specific subject is no longer detected (step S1412: Yes), the pre-processing unit 1210 returns to step S1401 and changes the setting of the entire imaging surface 200 to the first frame rate (step S1401). On the other hand, if the specific subject continues to be detected (step S1412: No), the pre-processing unit 1210 returns to step S1407 and changes the setting of the second image region a2 corresponding to the detection position of the specific subject to the second frame rate (step S1407). In this case, the pre-processing unit 1210 changes the setting of the image region where the specific subject is no longer detected to the first frame rate.

[0146] Furthermore, when the first moving image data 910 is input (step S1405), the image processing unit 901 executes image processing (step S1413) with reference to the additional information 817. In the image processing (step S1413), the image processing unit 901 refers to the additional information 817 of each frame and identifies each frame of the first moving image data 910 as being only the first frame.

[0147] Therefore, since the specific subject is not captured, the image processing unit 901 does not generate a third frame. Details of the image processing (step S1413) will be described later with reference to Fig. 18. The image processing unit 901 outputs the first moving image data 910 to the compression unit 902 (step S1414).

[0148] Furthermore, when the first moving image data 910 is input (step S1411), the image processing unit 901 performs image processing (step S1415) with reference to the additional information 817. In the image processing (step S1415), the image processing unit 901 refers to the additional information 817 of each frame and identifies each frame of the first moving image data 910 as including a first frame and a second frame.

[0149] Therefore, since the specific subject is captured in the first and second frames, the image processing unit 901 generates the third frame. Details of the image processing (step S1415) will be described later with reference to Fig. 18. The image processing unit 901 outputs the second moving image data 920 including the first and third frames to the compression unit 902 (step S1416).

[0150] When the first moving image data 910 is input (step S1414), the compression unit 902 performs a compression process on the first moving image data 910 (step S1417). Because the first moving image data 910 is made up of only the first frame, the compression unit 902 performs compression coding in the compression process (step S1417) that does not require motion detection or motion compensation. Details of the compression process (step S1417) will be described later with reference to FIGS. 19 to 24.

[0151] Furthermore, when second moving image data 920 is input (step S1416), compression unit 902 performs compression processing on second moving image data 920 (step S1418). Because second moving image data 920 is made up of first and third frames, compression unit 902 performs compression coding that does not require motion detection or motion compensation on first image region a1 in the compression processing (step S1418), and performs compression using normal hybrid coding on second image region a2. Details of the compression processing (step S1418) will be described later with reference to FIGS. 19 to 24.

[0152] <Setting Process (Steps S1404, S1410)> Fig. 15 is a flowchart showing a detailed example of the processing procedure of the setting process (steps S1404 and S1410) shown in Fig. 14. In Fig. 15, a first frame rate (e.g., 30 [fps]) is set in advance in the image sensor 100, and an image area of ​​a second frame rate (e.g., 60 [fps]) is tracked using the subject detection technology of the detection unit 1211 and fed back to the image sensor 100. Note that the image areas of the first frame rate and the second frame rate may be fixed at all times.

[0153] The preprocessing unit 1210 waits for input of frames constituting the first video data 910 (step S1501: No), and if a frame is input (step S1501: Yes), it determines whether or not a specific subject such as a main subject has been detected by the detection unit 1211 (step S1502). If a specific subject has not been detected (step S1502: No), the process proceeds to step S1504.

[0154] On the other hand, if a specific subject is detected (step S1502: Yes), the preprocessing unit 1210 causes the detection unit 1211 to compare the input frame with the previous frame in time (for example, a reference frame) to detect a motion vector, predict an image area of ​​the next input frame at the second frame rate, output this to the image sensor 100, and proceed to step S1504 (step S1503). As a result, the image sensor 100 sets the imaging conditions of the unit groups 202 that constitute an imaging area corresponding to the predicted image area to the second frame rate, sets the imaging conditions of the remaining unit groups 202 to the first frame rate, and images the subject.

[0155] 8 for the input frame (step S1504), executes additional information setting processing (step S1505), and returns to step S1501. The additional information setting processing (step S1505) is processing for setting the above-mentioned additional information, and will be described in detail with reference to FIG.

[0156] If no frames have been input (step S1501: No), the input of the first moving image data 910 has ended, and the pre-processing unit 1210 therefore ends the setting process (steps S1404, S1410).

[0157] <Additional Information Setting Process (Step S1505)> 16 is a flowchart showing a detailed example of the processing procedure of the additional information setting process (step S1505) shown in FIG. 15. When a frame is input (step S1601), the preprocessing unit 1210 determines whether or not there is an unselected image area in the input frame (step S1602). If there is an unselected image area (step S1602: Yes), the preprocessing unit 1210 selects one unselected image area (step S1603) and determines whether or not the detection flag for a specific subject is ON (step S1604). The detection flag is information indicating whether or not a specific subject has been detected, and its default is OFF (non-detection).

[0158] 14 (step S1406: Yes), the pre-processing unit 1210 changes the detection flag from OFF to ON (detecting). If the specific subject is not detected in step S1412 (step S1412: Yes), the pre-processing unit 1210 changes the detection flag from ON to OFF.

[0159] 16, if the detection flag is OFF (step S1604: No), the preprocessing unit 1210 sets information indicating the first frame rate for the selected image area in the additional information 817 (step S1605), and returns to step S1602. On the other hand, if the detection flag is ON (step S1604: Yes), the preprocessing unit 1210 determines whether the selected image area is an image area in which an image of a specific subject exists (step S1606).

[0160] If a specific subject image does not exist (step S1606: No), the process returns to step S1602. On the other hand, if a specific subject image exists (step S1606: Yes), the pre-processing unit 1210 sets information indicating the second frame rate for the selected image region in the additional information 817 (step S1607), and the process returns to step S1602.

[0161] In step S1602, if there is no unselected image area (step S1602: No), the pre-processing unit 1210 ends the additional information setting process. After that, the pre-processing unit 1210 transmits a frame rate setting instruction to the image sensor 100 (steps S1402, S1408).

[0162] By setting the additional information 817 for each image region of each frame, the preprocessing unit 1210 can identify which frame rate should be set for the imaging region of the image sensor 100 corresponding to which image region. Alternatively, the image processing unit 901 and the compression unit 902 can identify the frame rate for each image region of the input frame from the additional information 817.

[0163] <Video file generation process> 17 is a flowchart showing an example of the procedure for a video file generation process. The video file generation process is executed, for example, during the compression process of the compression unit 902, but may be executed after the image processing unit 901 when the video file 600 is generated without compression.

[0164] 8, the control unit 502 generates identification information 801, imaging condition information 802, and mask information 704, and stores them in that order in the mask area 612 (step S1701). Next, the pre-processing unit 1210 stores the imaging information in the imaging information area 613 (step S1702).

[0165] Then, the control unit 502 generates a Tv value map 812, an Sv value map 813, a Bv value map 814, and Av value information 815 (step S1703). Next, the preprocessing unit 1210 stores the mask information 704, image information 811, Tv value map 812, Sv value map 813, Bv value map 814, and Av value information 815 in the data area in this order (step S1704).

[0166] Finally, the control unit 502 generates the file basic information and stores it in the file basic information area 611 at the beginning of the header unit 601 (step S1705).

[0167] <Image processing (steps S1413, S1415)> Fig. 18 is a flowchart showing a detailed example of the processing procedure of the image processing (steps S1413 and S1415) shown in Fig. 14. When the image processing unit 901 inputs a frame (step S1801), it refers to the additional information 817 (step S1802). The image processing unit 901 determines whether the frame rate in the additional information 817 is the second frame rate only (step S1803).

[0168] If the additional information 817 does not include only the second frame rate (step S1803: No), the additional information 817 includes only the first frame rate or the first and second frame rates. Therefore, the image processing unit 901 holds the image information of the input frame as a storage target and overwrites it in the buffer (step S1804), and proceeds to step S1806.

[0169] On the other hand, if the additional information 817 is only the second frame rate (step S1803: Yes), the image information of the second frame is generated as the storage target using the image information in the buffer overwritten in step S1804 and the image information of the input frame (step S1805), and the process proceeds to step S1806.

[0170] In image processing (step S1413), the additional information 817 contains only the first frame rate (step S1803: No). On the other hand, in image processing (step S1415), the additional information 817 contains either the first frame rate and the second frame rate (step S1803: No), or only the second frame rate (step S1803: Yes).

[0171] 9 to 11, the image processing unit 901 can interpolate a frame consisting of only the second image region a2 at the second frame rate with the first image region a1 at the immediately preceding first frame rate, and synthesize the frame into a frame including the first image region a1 and the second image region a2. This makes it possible to absorb differences in frame rates within one frame.

[0172] <Compression processing example: First compression control method> Next, the compression process by the compression unit 902 will be explained separately for the first compression control method and the second compression control method.

[0173] 19 is a flowchart showing an example of a compression control process procedure of the first compression control method by the compression control unit 1312. The compression control unit 1312 acquires an input frame (step S1901) and selects an unselected image area from the acquired input frame (step S1902). Then, the compression control unit 1312 refers to the additional information 817 to find the frame rate of the selected image area (step S1903).

[0174] If the frame rate of the selected image area is the second frame rate (step S1903: second FR), the compression control unit 1312 outputs the image data of the selected image area to the motion detection unit 1310 (step S1904). As a result, the motion detection unit 1310 detects a motion vector for the selected image area of ​​the second frame rate using a reference frame as usual.

[0175] On the other hand, if the frame rate of the selected image area is the first frame rate (step S1903: 1st FR), the compression control unit 1312 sets a skip flag for the selected image area of ​​the first frame rate and outputs it to the motion detection unit 1310 (step S1905). As a result, the motion detection unit 1310 sets a specific motion vector indicating no motion for the selected image area of ​​the first frame rate.

[0176] After step S1904 or S1905, the compression control unit 1312 determines whether or not there is an unselected image area in the acquired input frame (step S1906). If there is an unselected image area (step S1906: Yes), the process returns to step S1902. On the other hand, if there is no unselected image area (step S1906: No), the compression control unit 1312 ends the series of processes.

[0177] 20 is a flowchart showing an example of the motion detection processing procedure of the first compression control method by the motion detection unit 1310. The motion detection unit 1310 obtains a reference frame temporally preceding the input frame from the frame memory 1309 (step S2001), and waits for input of the selected image area output in step S1904 or S1905 of FIG. 19 (step S2002: No).

[0178] If a selected image area is input (step S2002: Yes), the motion detection unit 1310 obtains image data of an image area in the same location as the selected image area from the reference frame (step S2003). Then, the motion detection unit 1310 determines whether or not a skip flag is set for the selected image area (step S2004). If there is no skip flag (step S2004: No), the frame rate of the selected image area is the second frame rate. Therefore, the motion detection unit 1310 detects a motion vector using the image data of the selected image area and the image data of the image area of ​​the reference frame obtained in step S2003 (step S2005).

[0179] On the other hand, if a skip flag is present (step S2004: Yes), motion detection unit 1310 sets a specific motion vector indicating no motion (step S2006). As a result, the motion detection process in motion detection unit 1310 always uses the specific motion vector indicating no motion, thereby reducing the processing load of motion detection for the selected image region of the first frame rate. Then, motion detection unit 1310 outputs the motion vector obtained in step S2005 or S2006 to motion compensation unit 1311 (step S2007), and the series of processes ends.

[0180] 21 is a flowchart showing an example of the motion compensation processing procedure of the first compression control method by the motion compensation unit 1311. The motion compensation unit 1311 obtains a reference frame from the frame memory 1309 (step S2101). The motion compensation unit 1311 obtains an image area in the same location as the selected image area from the reference frame (step S2102).

[0181] Then, the motion compensation unit 1311 performs motion compensation using the motion vector for the selected image area from the motion detection unit 1310 and the image area of ​​the reference frame acquired in step S2102 (step S2103). This allows the motion compensation unit 1311 to generate predicted image data for the selected image area.

[0182] Then, the motion compensation unit 1311 determines whether motion compensation for all selected image areas has been completed (step S2104). Specifically, for example, if the compression control unit 1312 determines in step S1906 that there is an unselected image area (step S1907: Yes), the motion compensation unit 1311 determines that motion compensation for all selected image areas has not been completed (step S2104: No), and returns to step S2102.

[0183] On the other hand, if the compression control unit 1312 determines in step S1906 that there are no unselected image areas (step S1906: No), the motion compensation unit 1311 determines that motion compensation for all selected image areas has been completed (step S2104: Yes).The motion compensation unit 1311 then outputs a predicted frame obtained by combining the predicted image data for all selected image areas to the subtraction unit 1301 and the generation unit 1308 (step S2105), thereby completing the series of processes.

[0184] <Compression processing example: Second compression control method> 22 is a flowchart showing an example of a compression control process procedure of the second compression control method by the compression control unit 1312. The compression control unit 1312 acquires an input frame (step S2201) and selects an unselected image area from the acquired input frame (step S2202). Then, the compression control unit 1312 refers to the additional information 817 to find the frame rate of the selected image area (step S2203).

[0185] If the frame rate of the selected image area is the second frame rate (step S2203: second FR), the compression control unit 1312 outputs the selected image area to the motion detection unit 1310 (step S2204). As a result, the motion detection unit 1310 detects a motion vector for the selected image area of ​​the second frame rate using a reference frame as usual.

[0186] On the other hand, if the frame rate of the selected image region is the first frame rate (step S2203: 1st FR), the compression control unit 1312 sets a skip flag for the selected image region of the first frame rate and outputs it to the motion detection unit 1310 (step S2205). As a result, the motion detection unit 1310 does not perform motion detection for the selected image region of the first frame rate. Then, the compression control unit 1312 issues a motion compensation stop instruction for the selected image region and outputs it to the motion compensation unit 1311 (step S2206). As a result, it is possible to stop the execution of motion compensation for the selected image region.

[0187] After step S2204 or S2206, the compression control unit 1312 determines whether or not there is an unselected image area in the acquired input frame (step S2207). If there is an unselected image area (step S2207: Yes), the process returns to step S2202. On the other hand, if there is no unselected image area (step S2207: No), the compression control unit 1312 ends the series of processes.

[0188] 23 is a flowchart showing an example of the motion detection processing procedure of the second compression control method by the motion detection unit 1310. The motion detection unit 1310 obtains a reference frame temporally preceding the input frame from the frame memory 1309 (step S2301), and waits for input of the selected image area output in step S2204 or S2205 of FIG. 22 (step S2302: No).

[0189] If a selected image area is input (step S2302: Yes), the motion detection unit 1310 obtains image data of an image area in the same location as the selected image area from the reference frame (step S2303). Then, the motion detection unit 1310 determines whether a skip flag is set for the selected image area (step S2304). If there is no skip flag (step S2304: No), the frame rate of the selected image area is the second frame rate. Therefore, the motion detection unit 1310 detects a motion vector using the image data of the selected image area and the image data of the image area of ​​the reference frame obtained in step S2003 (step S2305).

[0190] Then, the motion detection unit 1310 outputs the motion vector obtained in step S2305 to the motion compensation unit 1311 (step S2306), and ends the series of processes. On the other hand, if there is a skip flag (step S2304: Yes), the motion detection unit 1310 ends the series of processes without performing motion detection.

[0191] 24 is a flowchart showing an example of a motion compensation processing procedure of the second compression control method by the motion compensation unit 1311. The motion compensation unit 1311 obtains a reference frame from the frame memory 1309 (step S2401). The motion compensation unit 1311 obtains an image area in the same location as the selected image area from the reference frame (step S2402).

[0192] Then, the motion compensation unit 1311 determines whether the trigger input for motion compensation for the selected image area is a motion vector or a motion compensation stop instruction (step S2403). If the trigger input is a motion vector (step S2403: motion vector), the motion compensation unit 1311 performs motion compensation using the motion vector for the selected image area from the motion detection unit 1310 and the image area of ​​the reference frame acquired in step S2402 (step S2404). This allows the motion compensation unit 1311 to generate predicted image data for the selected image area.

[0193] On the other hand, if the trigger input is a motion compensation stop instruction (step S2403: motion compensation stop instruction), the motion compensation unit 1311 determines the image data of the obtained image area as image data of the prediction image area (prediction image data) (step S2405).

[0194] After step S2404 or S2405, the motion compensation unit 1311 determines whether motion compensation for all selected image areas has been completed (step S2406). Specifically, for example, if the compression control unit 1312 determines in step S2207 that there is an unselected image area (step S2007: Yes), the motion compensation unit 1311 determines that motion compensation for all selected image areas has not been completed (step S2406: No), and returns to step S2402.

[0195] On the other hand, if the compression control unit 1312 determines in step S2207 that there are no unselected image areas (step S2207: No), the motion compensation unit 1311 determines that motion compensation for all selected image areas has been completed (step S2406: Yes).The motion compensation unit 1311 then outputs a predicted frame obtained by combining the predicted image data for all selected image areas to the subtraction unit 1301 and the generation unit 1308 (step S2407), thereby completing the series of processes.

[0196] (1-1) As described above, the above-described video compression device includes an acquisition unit 1220 and a compression unit 902. The acquisition unit 1220 acquires video data including multiple frames output from an image sensor 100 that has a first imaging area for capturing an image of a subject and a second imaging area for capturing an image of the subject, and that is capable of setting a first frame rate (e.g., 30 [fps]) for the first imaging area and a second frame rate (e.g., 60 [fps]) that is faster than the first frame rate for the second imaging area. The compression unit 902 compresses the video data (second video data 920) acquired by the acquisition unit 1220 based on the first frame rate and the second frame rate.

[0197] This allows compression according to the frame rate for a single frame for which different frame rates are set to be realized by the general-purpose compression unit 902. Also, the processing load can be reduced compared to compression when a single frame rate is set.

[0198] (1-2) Furthermore, in the video compression device of (1-1) above, image data of a first image area a1 in a frame captured at a first frame rate is compressed based on the first frame rate, and image data of a second image area a2 in a frame captured at a second frame rate is compressed based on the second frame rate.

[0199] This makes it possible to compress the image area according to the frame rate for each frame rate for a single frame, thereby reducing the processing load compared to compression when a single frame rate is set.

[0200] (1-3) The video compression device of (1-2) above also has a motion detection unit 1310 and a motion compensation unit 1311. The motion detection unit 1310 sets a specific motion vector indicating that there is no motion in the object in the image data of the first image region a1 for the image data of the first image region a1, and performs motion vector detection for the image data of the second image region a2. The motion compensation unit 1311 performs motion compensation for the image data of the first image region a1 based on the specific motion vector, and performs motion compensation for the image data of the second image region a2 based on the motion vector detected by the motion detection unit 1310.

[0201] As a result, by setting a specific motion vector instead of detecting a motion vector for the image data of the first image area a1, motion detection is not performed, and the load of the compression process can be reduced.

[0202] (1-4) The video compression device of (1-3) above also includes a generation unit 1308. For each of a plurality of frames, the generation unit 1308 generates a reference frame to be referenced by a frame input temporally after the frame, based on the predicted frame and differential data between the frame and a predicted frame that predicts the frame. In this case, the motion compensation unit 1311 performs motion compensation on the image data of the first image region a1 based on a specific motion vector and the reference frame, and performs motion compensation on the image data of the second image region a2 based on the motion vector and the reference frame.

[0203] This makes it possible to perform motion compensation even when a specific motion vector is used, thereby realizing low-load compression processing.

[0204] (1-5) The video compression device of (1-2) above also includes a generation unit 1308, a motion detection unit 1310, and a motion compensation unit 1311. For each of a plurality of frames, the generation unit 1308 generates a reference frame to be referenced by a frame input temporally after the frame, based on the predicted frame and differential data between the frame and a predicted frame that predicts the frame. The motion detection unit 1310 does not detect a motion vector for image data of the first image region a1, but does detect a motion vector for image data of the second image region a2. The motion compensation unit 1311 performs motion compensation for image data of the first image region a1 based on the reference frame, and performs motion compensation for image data of the second image region a2 based on the reference frame and the motion vector detected by the motion detection unit 1310.

[0205] This makes it possible to reduce the load of the compression process by not performing motion detection on the image data of the first image area a1.

[0206] (1-6) In the video compression device described in (1-5) above, the motion compensation unit 1311 determines, as the reference frame for the image data of the first image area a1, a predicted frame that predicts the frame immediately following the frame in terms of time.

[0207] This makes it possible to perform motion compensation even when motion vectors are not detected, thereby realizing low-load compression processing.

[0208] (1-7) The video compression device of (1-1) above also includes an image processing unit 901. The image processing unit 901 updates a second frame to a third frame based on a first frame and a second frame among a plurality of frames. Here, the first frame is a frame captured by at least the first imaging area out of a first imaging area to which a first frame rate is set and a second imaging area to which a second frame rate is set.

[0209] The second frame is a frame captured by the second imaging region later than the first frame. The third frame is a frame obtained by combining the image data of the first image region a1 in the first frame with the image data of the second image region a2 in the second frame. The compression unit 902 compresses the image data of the first image region a1 of the third frame updated by the image processing unit 901 based on the first frame rate, and compresses the image data of the second image region a2 based on the second frame rate.

[0210] This prevents loss of image data in the second frame due to differences in frame rate. Therefore, even if there is a difference in frame rate for a single frame, compression processing is possible using the interpolated third frame.

[0211] (1-8) Furthermore, in the video compression device of (1-7) above, the image processing unit 901 updates the second frame to the third frame by applying the image data of the second image area a2 in the second frame to the area in the image data of the second image area a2 in the second frame that overlaps with the image data of the first image area a1 in the first frame.

[0212] As a result, for example, in an area where the front of the train in frame F2-60, the second frame, overlaps with the background area of ​​frame F1, the first frame, the image processing unit 901 preferentially applies the front of the train in frame F2, the second frame, and therefore an image (frame F2, the third frame) that looks less unnatural can be obtained.

[0213] (1-9) Furthermore, in the video compression device of (1-7) above, the image processing unit 901 updates the second frame to the third frame by applying image data of the second image area a2 in the first frame to an area in the second frame that does not belong to either the second image area a2 in the second frame or the first image area a1 in the first frame.

[0214] As a result, for example, for the image area between the tail of the train in frame F2-60 (the second frame) and the background area of ​​frame F1 (the first frame), image data of the second image area a2 (the tail of the train) in frame F1 (the first frame) is applied preferentially, thereby making it possible to obtain an image (frame F2 (the third frame)) that looks less unnatural.

[0215] (1-10) Another video compression device compresses video data including multiple frames output from an image sensor 100 that has a first imaging area for imaging a subject and a second imaging area for imaging the subject, and that is capable of setting a first frame rate for the first imaging area and a second frame rate for the second imaging area that is faster than the first frame rate. This video compression device has a generation unit 1308 and a motion compensation unit 1311.

[0216] The generation unit 1308 generates, for each of the multiple frames, a reference frame to be referenced by a frame input temporally after the frame, based on the predicted frame and differential data between the frame and the predicted frame that predicts the frame. The motion compensation unit 1311 performs motion compensation on a frame captured at the second frame rate among the multiple frames, using a specific reference frame among the multiple reference frames generated by the generation unit 1308.

[0217] This allows the reference frame to be fixed to a specific reference frame, thereby improving the efficiency of motion compensation.

[0218] (1-11) In the video compression device of (1-10), each of the plurality of frames includes image data of a first image area a1 corresponding to a first imaging area to which at least a first frame rate is set out of a first frame rate and a second frame rate faster than the first frame rate. The motion compensation unit 1311 performs motion compensation on the image data of the first image area a1 using a reference frame generated in the frame immediately preceding the frame as a specific reference frame.

[0219] This allows the closest frame to be referenced by setting the previous reference frame as the specific reference frame, thereby improving the efficiency of motion compensation while also increasing the accuracy of motion compensation.

[0220] (1-12) Furthermore, the electronic device 500 described above includes an image sensor 100 and a compression unit 902. The image sensor 100 has a first imaging area for capturing an image of a subject and a second imaging area for capturing an image of the subject, and is capable of setting a first frame rate for the first imaging area and a second frame rate faster than the first frame rate for the second imaging area. The image sensor 100 captures an image of the subject at the frame rates set for each imaging area and outputs multiple frames as video data. The compression unit 902 compresses each of the multiple frames captured by the image sensor 100 based on the first frame rate and the second frame rate.

[0221] This makes it possible to realize an electronic device 500 that can compress a single frame with different frame rates set according to the frame rate, thereby reducing the processing load compared to compression when a single frame rate is set.

[0222] (1-13) Another electronic device 500 has a first imaging area for imaging a subject and a second imaging area for imaging the subject, and compresses video data including multiple frames output from an image sensor 100 that can set a first frame rate for the first imaging area and a second frame rate that is faster than the first frame rate for the second imaging area. This electronic device 500 has a generation unit 1308 and a motion compensation unit 1311.

[0223] The generation unit 1308 generates, for each of the multiple frames, a reference frame to be referenced by a frame input temporally after the frame, based on the predicted frame and differential data between the frame and the predicted frame that predicts the frame. The motion compensation unit 1311 performs motion compensation on a frame captured at the second frame rate among the multiple frames, using a specific reference frame among the multiple reference frames generated by the generation unit 1308.

[0224] This makes it possible to fix the reference frame to a specific reference frame, thereby realizing electronic device 500 that improves the efficiency of motion compensation.

[0225] Examples of the electronic device 500 in (1-12) and (1-13) above include a digital camera, a digital video camera, a smartphone, a tablet, a surveillance camera, a drive recorder, and a drone.

[0226] (1-14) The above-described video compression program causes processor 1201 to compress video data including multiple frames output from image sensor 100, which has a first imaging area for imaging a subject and a second imaging area for imaging the subject, and is capable of setting a first frame rate for the first imaging area and a second frame rate faster than the first frame rate for the second imaging area. This video compression program causes processor 1201 to execute an acquisition process for acquiring video data and a compression process for compressing the video data acquired by the acquisition process based on the first frame rate and the second frame rate.

[0227] This allows software to compress a single frame with different frame rates set according to the frame rate, reducing the processing load compared to compression when a single frame rate is set.

[0228] (1-15) Furthermore, the other video compression program described above causes the processor 1201 to compress video data including multiple frames output from an image sensor 100 that has a first imaging area for imaging a subject and a second imaging area for imaging a subject, is capable of setting a first frame rate for the first imaging area, and is capable of setting a second frame rate for the second imaging area that is faster than the first frame rate.

[0229] This video compression program causes the processor 1201 to perform a generation process for generating, for each of a plurality of frames, a reference frame to be referenced by a frame input temporally later than the frame, based on difference data between the frame and a predicted frame that predicts the frame, and the predicted frame; and a motion compensation process for performing motion compensation on a frame among the plurality of frames that was captured at a second frame rate, using a specific reference frame from among the reference frames generated by the generation process.

[0230] This allows the reference frame to be fixed to a specific reference frame, and makes it possible to achieve efficient motion compensation using software.

[0231] The video compression programs (1-14) and (1-15) described above may be recorded on a portable recording medium such as a CD-ROM, a DVD-ROM, a flash memory, or the memory card 504. The video compression programs (1-14) and (1-15) described above may be recorded on a server that can be downloaded to the video compression device or the electronic device 500. [Example]

[0232] A second embodiment will be described. In the first embodiment, since the ranges Da1, Da3, etc. exist in the frames F2, F4, etc. shown in Fig. 10, the image processing unit 901 fills in a specific color or performs demosaicing. In the second embodiment, the image processing unit 901 generates frames F2, F4, etc. that have a less unnatural appearance without performing such image processing.

[0233] In the second embodiment, a configuration including the image processing unit 901 but not including the image sensor 100 or the compression unit 902 is referred to as an image processing device. In the first embodiment, the compression unit 902 compresses the frames that have been image-processed by the image processing device (image processing unit 901), but this does not necessarily have to be compressed, and the frames may be output to the liquid crystal monitor 503 in an uncompressed state. In the second embodiment, the same reference numerals are used for parts common to the first embodiment, and descriptions thereof will be omitted.

[0234] <Example of improved video compression> Here, an improved example of video compression in Example 2 will be described. In Fig. 10, image processing example 1 has been described in which the electronic device 500 captures an image of a running train as a specific subject while capturing fixed-point images of a landscape including rice fields, mountains, and the sky. The processing flow of video processing example 1 will be specifically described below.

[0235] Fig. 25 is an explanatory diagram showing a specific processing flow of moving image processing example 1 shown in Fig. 10. As also explained in Fig. 10, the image sensor 100 outputs frames F1, F2-60, F3, ... in chronological order. It is assumed that the train runs from right to left within frames F1, F2-60, F3.

[0236] 25, the sub-numbers of frames F1 to F3 indicate the frame rates of those frames F1 to F3. For example, odd-numbered frame F1-30 indicates image data of a first image region r1-30 in frame F1 that was captured at a frame rate of 30 fps, and frame F1-60 indicates image data of a second image region r1-60 in frame F1 that was captured at a frame rate of 60 fps.

[0237] The second image region r1-60 of frame F1-60, captured at a frame rate of 60 fps, contains image data of a train, but the second image region r1-60 does not exist in frame F1-30. Such a region in frame F1-30 is referred to as a non-image region n1-60. Similarly, the first image region r1-30 of frame F1-30, captured at a frame rate of 30 fps, contains image data of a landscape, but in frame F1-60, this image data does not exist in the second image region r1-60. Such a region in frame F1-60 is referred to as a non-image region n1-30.

[0238] Similarly, in frame F3, frame F3-30 is composed of a first image area r3-30 in which landscape image data is output and a non-image area n3-60 in which nothing is output, and frame F3-60 is composed of a second image area r3-60 in which train image data is output and a non-image area n3-60 in which nothing is output. The same is true for odd-numbered frames (not shown) after frames F3-30 and F3-60.

[0239] The even-numbered frame F2-60 is a frame that is composed of image data (a train) of a second image area r2-60 captured at a frame rate of 60 fps and a non-image area n2-30 where nothing is output. The same is true for the subsequent even-numbered frames (not shown).

[0240] The image processing unit 901 generates frame F2, which is composite image data, by combining image data (train) of the second image region r2-60 of frame F2-60 with image data (landscape) of the first image region r1-30 of frame F1-30. In this case, as described in FIG. 10, frame F2 has an overlapping range Da1 between the non-image region n1-60 of frame F1-30 and the non-image region n2-30 of frame F2-60.

[0241] In the first embodiment, the image processing unit 901 painted the area Da1 with a specific color or performed demosaic processing, but in the second embodiment, the image processing unit 901 copies the image data of the area Da1 in another image area without performing such image processing. As a result, the image processing unit 901 generates a frame F2 that is less unnatural. The same applies to the area Da3, but in the second embodiment, the description will focus on the area Da1.

[0242] <Example of compositing frame F2> Next, an example of compositing the frame F2 by the image processing unit 901 (compositing unit 1214) will be described.

[0243] [Synthesis Example 1] 26 is an explanatory diagram showing a first compositing example of a 60-fps frame F2 according to a second embodiment. In this compositing example, a range Db1 is used, which is located at the same position as the range Da1 in a first image area r3-30 of a frame F3 that is temporally subsequent to the frame F2-60, as another image area to be copied to the range Da1. The image data of the range Db1 is part of a landscape.

[0244] 26, the image processing unit 901 identifies an area Da1 where a non-image area n1-60 in frame F1-30 overlaps with a non-image area n2-30 in frame F2-60, and identifies an area Db1 in frame F3 that is in the same position as the identified area Da1. The image processing unit 901 then copies the image data of area Db1 to area Da1 in frame F2. This allows the image processing unit 901 to generate a frame F2 that appears less unnatural.

[0245] [Synthesis Example 2] 27 is an explanatory diagram showing Combining Example 2 of frame F2 at 60 [fps] according to Example 2. In Combining Example 1, image data in the first image region r1-30 of frame F1-30 is used as the source to be copied to the first image region of frame F2, and image data in the range Db1 of frame F3 is used as the source to be copied to the range Da1, but in Combining Example 2, on the other hand, image data in the first image region r3-30 of frame F3-30 is used as the source to be copied to the first image region of frame F2, and image data in the range Db2 of frame F1 is used as the source to be copied to the range Da2.

[0246] Here, range Da2 is the range where non-image region n3-60 of frame F3-30 and non-image region n2-30 of frame F2-60 overlap. Range Db2 of frame F1 is a range at the same position as range Da2.

[0247] 27, the image processing unit 901 identifies an area Da2 where non-image area n3-60 in frame F3-30 overlaps with non-image area n2-30 in frame F2-60, and identifies an area Db2 in frame F1 that is in the same position as the identified area Da2. The image processing unit 901 then copies the image data of area Db2 to area Da2 in frame F2. This allows the image processing unit 901 to generate a frame F2 that appears less unnatural.

[0248] [Synthesis Example 3] Combination example 3 is an example in which either combination example 1 or combination example 2 is selected and combined. In combination example 3, the image processing unit 901 specifies range Da1 in combination example 1 and range Da2 in combination example 2. The image processing unit 901 selects either range Da1 or Da2 and applies the combination example in which the selected range is specified. The image processing unit 901 applies combination example 1 when range Da1 is selected, and applies combination example 2 when range Da2 is selected.

[0249] The image processing unit 901 uses, for example, the narrowness of the range as a selection criterion for selecting either the range Da1 or Da2. In the examples of Figures 26 and 27, range Da1 is narrower than range Da2, so synthesis example 1 is applied. By selecting the narrower range, it is possible to minimize the sense of incongruity caused by copying.

[0250] [Synthesis Example 4] FIG. 28 is an explanatory diagram showing a fourth synthesis example of a frame F2 at 60 fps according to the second embodiment. In the fourth synthesis example, the source of the range Da1 in the first synthesis example is not the image data of the range Db1 (part of the scenery) in the first image area r3-30 of the frame F3, but the image data of the range Db3 (the end of the train) in the second image area r1-60 of the frame F1. As a result, the image data of the range Db3 is added to the image data of the second image area r2-60 of the frame F2 (the train). However, since the image data is added on the opposite side of the direction of travel of the image data of the second image area r2-60 (the train), when a user views the video, the image data of the second image area r2-60 (the train) is perceived as an afterimage of a running train. Therefore, in this case as well, frames F2, F4, ... can be generated with less awkwardness.

[0251] <Example of synthesis processing procedure for frame F2> Below, we will explain an example of the synthesis process procedure for frame F2 according to Synthesis Examples 1 to 4 described above. In the following flowcharts, the second frame is a frame that is captured only at the second frame rate (e.g., 60 [fps]) to be synthesized. For example, this is frame F2 in FIGS. 25 to 28.

[0252] The first frame is a frame that is temporally immediately preceding the second frame and includes an image area captured at at least the first frame rate (e.g., 30 fps) of the first and second frame rates, such as frame F1 in FIGS.

[0253] The third frame is a frame that is temporally subsequent to the second frame and includes an image area captured at at least the first frame rate out of the first and second frame rates, such as frame F3 in FIGS.

[0254] [Synthesis Example 1] 29 is a flowchart showing a first synthesis processing procedure example of the synthesis example 1 of frame F2 by the image processing unit 901. It is assumed that input frames are sequentially stored in the buffer. The image processing unit 901 determines whether the second frame is present in the buffer (step S2901). If the second frame is present in the buffer (step S2901: Yes), the image processing unit 901 identifies a range that is both a non-image area of ​​the first frame and a non-image area of ​​the second frame (step S2902). Specifically, for example, the image processing unit 901 identifies a range Da1 where the non-image area n1-60 of frame F1-30 and the non-image area n2-30 of frame F2-60 overlap.

[0255] Next, the image processing unit 901 copies the image data of the first image region a1 of the first frame (step S2903). Specifically, for example, the image processing unit 901 copies the image data (landscape) of the first image region r1-30 of the frame F1.

[0256] Then, the image processing unit 901 copies the image data of the range identified in step S2902 from the third frame (step S2904). Specifically, for example, the image processing unit 901 copies the image data of the range Db1 identical to the range Da1 identified in step S2902 from the frame F3.

[0257] Next, the image processing unit 901 updates the second frame (step S2905). Specifically, for example, the image processing unit 901 updates frame F2-60 to frame F2 by combining the second image region r2-60 of frame F2-60, the image data (landscape) of the duplicated first image region r1-30, and the image data of the duplicated range Db1.

[0258] After this, the process returns to step S2901. If the second frame is not in the buffer (step S2901: No), the image processing unit 901 ends the image processing (steps S1413 and S1415). This allows the image processing unit 901 to generate a frame F2 that is less unnatural.

[0259] [Synthesis Example 2] 30 is a flowchart showing a second synthesis processing procedure example for the second synthesis example of frame F2 by the image processing unit 901. It is assumed that input frames are sequentially stored in a buffer. The image processing unit 901 determines whether the second frame is in the buffer (step S3001). If the second frame is in the buffer (step S3001: Yes), the image processing unit 901 identifies a range that is both a non-image area of ​​the third frame and a non-image area of ​​the second frame (step S3002). Specifically, for example, the image processing unit 901 identifies a range Da2 where the non-image area n3-60 of frame F3-30 and the non-image area n2-30 of frame F2-60 overlap.

[0260] Next, the image processing unit 901 copies the image data of the first image region a1 of the third frame (step S3003). Specifically, for example, the image processing unit 901 copies the image data (landscape) of the first image region r3-30 of the frame F3.

[0261] Then, the image processing unit 901 copies the image data of the range identified in step S3002 from the first frame (step S3004). Specifically, for example, the image processing unit 901 copies the image data of the range Db2, which is the same as the range Da2 identified in step S3002, from the frame F1.

[0262] Next, the image processing unit 901 updates the second frame (step S3005). Specifically, for example, the image processing unit 901 updates frame F2-60 to frame F2 by combining the second image region r2-60 of frame F2-60, the image data (landscape) of the duplicated first image region r3-30, and the image data of the duplicated range Db2.

[0263] After this, the process returns to step S3001. If the second frame is not in the buffer (step S3001: No), the image processing unit 901 ends the image processing (steps S1413 and S1415). This allows the image processing unit 901 to generate a frame F2 that is less unnatural.

[0264] [Synthesis Example 3] 31 is a flowchart showing a third synthesis processing procedure example for the synthesis example 3 of frame F2 by the image processing unit 901. It is assumed that input frames are sequentially stored in a buffer. The image processing unit 901 determines whether a second frame is present in the buffer (step S3101). If the second frame is present in the buffer (step S3101: Yes), the image processing unit 901 identifies a first range that is a non-image area of ​​the first frame and a non-image area of ​​the second frame (step S3102). Specifically, for example, the image processing unit 901 identifies a range Da1 where the non-image area n1-60 of frame F1-30 and the non-image area n2-30 of frame F2-60 overlap.

[0265] The image processing unit 901 identifies a second range that is a non-image area of ​​the third frame and a non-image area of ​​the second frame (step S3103). Specifically, for example, the image processing unit 901 identifies a range Da2 where the non-image area n3-60 of the frame F3-30 and the non-image area n2-30 of the frame F2-60 overlap.

[0266] Next, the image processing unit 901 selects either the identified first range or second range (step S3104). Specifically, for example, the image processing unit 901 selects the narrower range (the one with the smaller area) of the first range or the second range. The selected range is referred to as the selected range. In the case of ranges Da1 and Da2, the image processing unit 901 selects range Da1. This makes it possible to minimize the range used for synthesis, further reducing the sense of incongruity.

[0267] Then, image processing unit 901 copies the image data of the first image region a1 of the selected frame (step S3105). The selected frame is the frame from which the selected range is specified. For example, if the first range (range Da1) is selected, the selected frame is the first frame (frame F1), and if the second range (range Da2) is selected, the selected frame is the third frame (frame F3). Therefore, the image data of the first image region a1 of the selected frame is the image data (landscape) of the first image region r1-30 of frame F1 if the selected frame is frame F1, and is the image data (landscape) of the first image region r3-30 of frame F3 if the selected frame is frame F3.

[0268] Then, the image processing unit 901 copies the image data of the selected range in step S3104 from a non-selected frame (step S3106). A non-selected frame is a frame that is used to identify the non-selected range. For example, if the first range (range Da1) is not selected, the non-selected frame is the first frame (frame F1), and if the second range (range Da2) is not selected, the non-selected frame is the third frame (frame F3). Therefore, if the selected range is range Da1, the image processing unit 901 copies the image data of range Db1, which is in the same position as range Da1, from frame F3, and if the selected range is range Da2, the image processing unit 901 copies the image data of range Db2, which is in the same position as range Da2, from frame F1.

[0269] Next, the image processing unit 901 updates the second frame (step S3107). Specifically, for example, if the selected range is the first range (range Da1), the image processing unit 901 updates frame F2-60 to frame F2 by combining the second image region r2-60 of frame F2-60, the image data (landscape) of the duplicated first image region r1-30, and the image data of the duplicated range Db1. Also, if the selected range is the second range (range Da2), the image processing unit 901 updates frame F2-60 to frame F2 by combining the second image region r2-60 of frame F2-60, the image data (landscape) of the duplicated first image region r3-30, and the image data of the duplicated range Db2.

[0270] After this, the process returns to step S3101. If the second frame is not in the buffer (step S3101: No), the image processing unit 901 ends the image processing (steps S1413 and S1415). By selecting the narrower range, the image processing unit 901 can minimize the sense of incongruity caused by copying.

[0271] [Synthesis Example 4] 32 is a flowchart showing a fourth synthesis process procedure for the image processing unit 901 for the frame F2. It is assumed that input frames are sequentially stored in a buffer. The image processing unit 901 determines whether the second frame is present in the buffer (step S3201). If the second frame is present in the buffer (step S3201: Yes), the image processing unit 901 identifies a range that is both a non-image area of ​​the first frame and a non-image area of ​​the second frame (step S3202). Specifically, for example, the image processing unit 901 identifies a range Da1 where the non-image area n1-60 of the frame F1-30 and the non-image area n2-30 of the frame F2-60 overlap.

[0272] Next, the image processing unit 901 copies the image data of the first image region a1 of the first frame (step S3203). Specifically, for example, the image processing unit 901 copies the image data (landscape) of the first image region r1-30 of the frame F1.

[0273] Then, the image processing unit 901 copies the image data of the range identified in step S3202 from the first frame (step S3204). Specifically, for example, the image processing unit 901 copies the image data of the range Da1 identified in step S3202 and the same range Db3 from the frame F1.

[0274] Next, the image processing unit 901 updates the second frame (step S3205). Specifically, for example, the image processing unit 901 updates frame F2-60 to frame F2 by combining the second image region r2-60 of frame F2-60, the image data (landscape) of the duplicated first image region r1-30, and the image data of the duplicated range Db1.

[0275] After this, the process returns to step S3201. If the second frame is not in the buffer (step S3201: No), the image processing unit 901 ends the image processing (steps S1413 and S1415). This allows the image processing unit 901 to generate a frame F2 that is less unnatural.

[0276] (2-1) In this way, the image processing device of Example 2 has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and performs image processing on a plurality of frames generated by output from an image sensor 100 that can set a first frame rate (e.g., 30 [fps]) in the first imaging area and a second frame rate (e.g., 60 [fps]) that is faster than the first frame rate in the second imaging area.

[0277] The image processing device has an identification unit 1213 and a synthesis unit 1214. Based on a first frame generated from outputs from the first imaging region and the second imaging region and a second frame (for example, frame F2-60) generated from output from the second imaging region, among the multiple frames, the identification unit 1213 identifies a range Da1 that is a non-image region n1-60 in the first frame that corresponds to the second imaging region and a non-image region n2-30 in the second frame that corresponds to the first imaging region.

[0278] The synthesis unit 1214 synthesizes the second frame with image data of the first image area r1-30 corresponding to the first imaging area in the first frame, and the image data of the first image area r1-30 in the first frame and specific image data of the range Da1 identified by the identification unit 1213 in other image areas other than the second frame.

[0279] This allows the non-image area n2-30 that was not captured in the second frame to be interpolated using a frame that is temporally close to the second frame, thereby obtaining a composite frame that is less unnatural than the second frame.

[0280] (2-2) Furthermore, in the image processing device of (2-1) above, the first frame may be a frame (e.g., frame F1) generated temporally earlier than the second frame, and the specific image data may be image data of a range (Da1) in the first image area a1 (r3-30) of a frame (e.g., frame F3) generated temporally later than the second frame by output from the first imaging area and the second imaging area (i.e., image data of a range Db1).

[0281] This allows the non-image area n2-30 that was not captured in the second frame to be interpolated using the first frame, which is one frame before the second frame, and the third frame, which is one frame after the second frame, resulting in a composite frame that is less unnatural than the second frame.

[0282] (2-3) Furthermore, in the image processing device of (2-1) above, the first frame may be a frame (e.g., frame F3) generated later in time than the second frame, and the specific image data may be image data of a range (Da2) in the first image area a1 (r1-30) of a frame (e.g., frame F1) generated by output from the first imaging area and the second imaging area earlier in time than the second frame (i.e., image data of a range Db2).

[0283] This allows the non-image area n2-30 that was not captured in the second frame to be interpolated using the first frame, which is one frame before the second frame, and the third frame, which is one frame after the second frame, resulting in a composite frame that is less unnatural than the second frame.

[0284] (2-4) In the image processing device of (2-1) above, the identification unit 1213 identifies a range to be used by the composition unit 1214 based on the first range (Da1) and the second range (Da2). The composition unit 1214 combines image data of the first image region a1 (r1-30 / r3-30) in the second frame, one frame (F1 / F3) that is the source of identification of one range (Da1 / Da2) among the first and third frames by the identification unit 1213, with image data (Db1 / Db2) of one range (Da1 / Da2) in the first image region a1 (r3-30 / r1-30) in the other frame (F3 / F1) that is the source of identification of the other range (Da2 / Da1) among the first and third frames that was not identified by the identification unit 1213.

[0285] As a result, the image processing unit 901 can minimize the sense of incongruity caused by copying by selecting the narrower range.

[0286] (2-5) Furthermore, in the image processing device of (2-1) above, the first frame is a frame generated temporally earlier than the second frame, and the specific image data may be image data of a range (Da1) in the second image area a2 of the first frame (i.e., image data of a range Db3).

[0287] This allows the non-image area n2-30 that was not captured in the second frame to be interpolated using the first frame, which is the frame immediately preceding the second frame, thereby obtaining a composite frame that is less unnatural than the second frame.

[0288] (2-6) Furthermore, the video compression device of Example 2 compresses video data including multiple frames generated by output from an image sensor 100 that has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging area and a second frame rate (e.g., 60 [fps]) faster than the first frame rate in the second imaging area.

[0289] The video compression device has an identification unit 1213 and a synthesis unit 1214. Based on a first frame generated from outputs from the first and second imaging regions and a second frame (e.g., frame F2-60) generated from output from the second imaging region, among the multiple frames, the identification unit 1213 identifies a range Da1 that is a non-image region n1-60 in the first frame that corresponds to the second imaging region and a non-image region n2-30 in the second frame that corresponds to the first imaging region.

[0290] The synthesis unit 1214 synthesizes the second frame with image data of the first image area r1-30 corresponding to the first imaging area in the first frame, and the image data of the first image area r1-30 in the first frame and specific image data of the range Da1 identified by the identification unit 1213 in other image areas other than the second frame.

[0291] The compression unit 902 compresses the first frame and the frame synthesized by the synthesis unit 1214 .

[0292] This makes it possible to obtain a composite frame that can be compressed in the same way as the second frame, but that is less unnatural than the second frame.

[0293] (2-7) Furthermore, the image processing program according to the second embodiment causes the processor 1201 to perform image processing of a plurality of frames generated by output from the image sensor 100, which has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 fps) in the first imaging area, and is capable of setting a second frame rate (e.g., 60 fps) faster than the first frame rate in the second imaging area.

[0294] The image processing program causes the processor 1201 to execute identification processing and synthesis processing. In the identification processing, the image processing program causes the processor 1201 to identify, among multiple frames, a range Da1 that is a non-image area n1-60 corresponding to the second imaging area in the first frame and a non-image area n2-30 corresponding to the first imaging area in the second frame, based on a first frame generated from outputs from the first imaging area and the second imaging area, and a second frame (for example, frame F2-60) generated from output from the second imaging area.

[0295] In the synthesis process, the image processing program causes the processor 1201 to synthesize the second frame, image data of the first image area r1-30 corresponding to the first imaging area in the first frame, and specific image data of the range Da1 identified by the identification process in the image data of the first image area r1-30 in the first frame and other image areas other than the second frame.

[0296] This allows software to interpolate the non-image area n2-30 that was not captured in the second frame with a frame that is temporally close to the second frame, thereby creating a composite frame that is less unnatural than the second frame.

[0297] (2-8) The video compression program according to the second embodiment causes the processor 1201 to execute a specifying process, a combining process, and a compressing process.

[0298] The video compression program causes the processor 1201 to execute a specification process and a composition process. In the specification process, the image processing program causes the processor 1201 to specify, among multiple frames, a range Da1 that is a non-image area n1-60 in the first frame that corresponds to the second imaging area and a non-image area n2-30 in the second frame that corresponds to the first imaging area, based on a first frame that is generated from outputs from the first imaging area and the second imaging area, and a second frame (for example, frame F2-60) that is generated from output from the second imaging area.

[0299] In the synthesis process, the video compression program causes the processor 1201 to synthesize the second frame, image data of the first image area r1-30 corresponding to the first imaging area in the first frame, and specific image data of the range Da1 identified by the identification process in the image data of the first image area r1-30 in the first frame and in other image areas other than the second frame.

[0300] In the compression process, the video compression program causes the processor 1201 to compress the first frame and the frame synthesized by the synthesis process.

[0301] This allows the software to obtain a composite frame that is as compressible as the second frame but that is less unnatural than the second frame.

[0302] The image processing program (2-7) and the video compression program (2-8) described above may be recorded on a portable recording medium such as a CD-ROM, a DVD-ROM, a flash memory, or the memory card 504. The image processing program (2-7) and the video compression program (2-8) described above may be recorded on a server that can be downloaded to the video compression device or the electronic device 500. [Example]

[0303] A third embodiment will be described. In the first embodiment, since the ranges Da1, Da3, etc. exist in the frames F2, F4, etc. shown in Fig. 10, the image processing unit 901 fills in a specific color or performs demosaicing. In the third embodiment, similar to the second embodiment, the image processing unit 901 generates frames F2, F4, etc. that have a less unnatural appearance without performing such image processing.

[0304] In the third embodiment, a configuration including the image processing unit 901 but not including the image sensor 100 or the compression unit 902 is referred to as an image processing device. Also, a configuration including the image sensor 100 and the pre-processing unit 1210 is referred to as an image capturing device. In the first embodiment, the compression unit 902 compresses the frames that have been image-processed by the image processing device (image processing unit 901), but this does not necessarily have to be compressed, and the frames may be output to the liquid crystal monitor 503 in an uncompressed state. In the third embodiment, the same reference numerals are used for parts common to the first and second embodiments, and their description will be omitted.

[0305] 33 is an explanatory diagram showing an example of compositing a 60 [fps] frame F2 according to Example 3. Before capturing frame F2-60, the pre-processing unit 1210 detects a specific subject, such as a train, from frames F1 and earlier, and detects the motion vector of the specific subject in the previous frame F1. The pre-processing unit 1210 can obtain a 60 [fps] image region R12-60 in the next frame F2-60 using the image region and motion vector of the specific subject in frame F1.

[0306] Furthermore, in synthesizing frame F2, which is a composite frame, as in Example 1, the image processing unit 901 duplicates the image data (landscape) of the first image area r1-30 of the previous frame F1, and synthesizes the image data (landscape) of the first image area r1-30 with the image data (train and part of the landscape) of the image area R12-60 to obtain frame F2.

[0307] 34 is an explanatory diagram showing the correspondence between the setting of the imaging area and the image area of ​​frame F2-60, where (A) shows an example of motion vector detection, and (B) shows the correspondence between the setting of the imaging area and the image area of ​​frame F2-60.

[0308] The imaging area p1-60 is an imaging area of ​​a specific subject that has already been detected after the generation of the frame F0-60 that is temporally immediately preceding the frame F1 and before the generation of the frame F1. Therefore, in the frame F1, the image data o1 of the specific subject (the train) exists in the second image area r1-60 that corresponds to the imaging area p1-60.

[0309] The pre-processing unit 1210 detects a motion vector mv of the specific subject from image data o1 of the specific subject in frame F0 and image data o1 of the specific subject in frame F1 using the detection unit 1211. Then, the pre-processing unit 1210 detects a second image area r2-60 in which the specific subject is displayed in the next frame F2-60 using a second image area r1-60 of the specific subject in frame F1 and the motion vector mv, and detects a detected imaging area p2-60 on the imaging surface 200 of the image sensor 100 that corresponds to the detected second image area r2-60.

[0310] The pre-processing unit 1210 sets the frame rate of a specific imaging area P12-60, which includes the imaging area p1-60 and the detection imaging area p2-60 identified when generating the frame F1, to the second frame rate by the setting unit 1212, and outputs the setting instruction to the image sensor 100. As a result, the image sensor 100 sets the frame rate of the specific imaging area P12-60 to the second frame rate, captures the image, and generates the frame F2-60.

[0311] The image processing unit 901 causes the synthesis unit 1214 to synthesize image data of the first image region r1-30 included in the frame F1 with image data of a specific imaging region P12-60 included in the second frame F2-60 generated by imaging at the second frame rate set by the setting unit 1212. As a result, the frame F2-60 is updated to the frame F2.

[0312] After generating frame F2-60 and before generating the next frame F3, the pre-processing unit 1210 sets the frame rate of the detection imaging area p2-60 to the second frame rate, and sets the frame rate of the other imaging areas of the imaging surface 200 other than the detection imaging area p2-60 to the first frame rate. As a result, in generating frame F3 obtained by imaging including an imaging area at the first frame rate, the detection imaging area p2-60 is the only second imaging area to which the second frame rate is set, just like frame F1. In this way, specific detection imaging areas are set for frames F2-60, F4-60, ... to be synthesized, thereby reducing unnecessary processing in frames F1, F3, ...

[0313] Frame F2-60 includes image data o1 of a specific subject (a train) and image data o2 of a portion of a landscape in an image area R12-60. In this way, image area R12-60 is expanded on the opposite side of the direction of movement of the specific subject compared to the second image area r2-60. Therefore, it is not necessary to specify ranges Da1 and Da2 and duplicate and combine image data of ranges Db1 and Db2 of other frames as in the second embodiment. Note that the combining process of the third embodiment is executed, for example, in step S1805 of FIG. 18. Furthermore, this combining process is applied when combining frames F2-60, F4-60, ... of only the second frame rate, and is not executed for frames F1, F3, ... that include an image area of ​​the first frame rate.

[0314] As described above, in the third embodiment, the image data to be synthesized are two, the image region R12-60 and the first image region r1-30 of the frame F1, so that the frame F2 can be generated with less unnaturalness. That is, the image data o1 and o2 are image data captured at the same time, so the boundary between the image data o1 and o2 is not unnatural and does not feel unnatural. Furthermore, since the processes of identifying the ranges Da1 and Da2 and selecting the optimal range from the ranges Da1 and Da2, as in the second embodiment, are not required, the load of the synthesis process for the frame F2 can be reduced.

[0315] (3-1) As described above, the imaging device according to the third embodiment includes the imaging element 100, the detection unit 1211, and the setting unit 1212. The imaging element 100 has a first imaging region for imaging a subject and a second imaging region for imaging the subject, and is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging region and a second frame rate (e.g., 60 [fps]) that is faster than the first frame rate in the second imaging region.

[0316] The detection unit 1211 detects a detected imaging area p2-60 of a specific subject in the imaging element 100 based on a second image area r1-60 of the specific subject included in the frame F1 generated by the output from the imaging element 100. The setting unit 1212 sets the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate the frame F1 and the imaging area detected by the detection unit 1211 (hereinafter referred to as the detected imaging area) p2-60 to the second frame rate.

[0317] This allows the imaging area at the second frame rate to be expanded, so that a specific subject can be imaged at the second frame rate without creating a range Da1 where non-image areas overlap in frames F1 and F2, thereby reducing image loss in frame F2-60 imaged at the second frame rate.

[0318] (3-2) Furthermore, in the imaging device of (3-1) above, the detection unit 1211 detects a detected imaging area p2-60 of the specific subject based on a second image area r1-60 of the specific subject included in frame F1 and a motion vector mv of the specific subject between frame F1 and frame F0-60 that is temporally earlier than frame F1.

[0319] This makes it possible to easily predict the detected image capture area p2-60 of the specific subject.

[0320] (3-3) Furthermore, in the imaging device of (3-1) above, when the frame is the first frame F1 generated by output from the first imaging area, the setting unit 1212 sets the frame rate of the specific imaging area to the second frame rate, and when the frame is the second frame F2-60 generated by output from the specific imaging area after the first frame F1, the setting unit 1212 sets the frame rate of the detection imaging area p2-60 to the second frame rate, and sets the frame rate of other imaging areas other than the detection imaging area p2-60 (parts of the imaging surface 200 excluding the detection imaging area p2-60) to the first frame rate.

[0321] As a result, a specific detection imaging area is set only for the frames F2-60, F4-60, . . . that are to be synthesized, and therefore unnecessary processing in the frames F1, F3, .

[0322] (3-4) Furthermore, the image processing device of Example 3 has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and performs image processing on frames generated by output from an image sensor 100 that is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging area and a second frame rate (e.g., 60 [fps]) that is faster than the first frame rate in the second imaging area.

[0323] This image processing device has a detection unit 1211, a setting unit 1212, and a synthesis unit 1214. The detection unit 1211 detects an imaging area p2-60 of a specific subject in the imaging element 100 based on a second image area r1-60 of the specific subject included in frame F1 generated by output from the imaging element 100. The setting unit 1212 sets the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate frame F1 and the detected imaging area p2-60 detected by the detection unit 1211 to the second frame rate.

[0324] The synthesis unit 1214 synthesizes image data of the first image area r1-30 included in the first frame F1 with image data of a specific imaging area P12-60 included in the second frame F2-60 generated by imaging at the second frame rate set by the setting unit 1212.

[0325] This allows the imaging area at the second frame rate to be expanded, and the specific subject to be imaged at the second frame rate without creating an overlapping area Da1 between the non-image areas of frames F1 and F2, thereby preventing image loss in frame F2-60 captured at the second frame rate. Furthermore, since there is no need to interpolate the overlapping area Da1 during composition, a less unnatural-looking image can be obtained, and the load of composition processing can be reduced.

[0326] (3-5) Furthermore, the video compression device of Example 3 compresses video data including multiple frames generated by output from an image sensor 100 that has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging area and a second frame rate (e.g., 60 [fps]) faster than the first frame rate in the second imaging area.

[0327] This video compression device has a detection unit 1211, a setting unit 1212, a synthesis unit 1214, and a compression unit 902. The detection unit 1211 detects an imaging area p2-60 of a specific subject in the imaging element 100 based on a second image area r1-60 of the specific subject included in frame F1 generated by output from the imaging element 100. The setting unit 1212 sets the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate frame F1 and the imaging area p2-60 detected by the detection unit 1211 to the second frame rate.

[0328] The combining unit 1214 combines image data of a first image region r1-30 included in the first frame F1 with image data of a specific imaging region P12-60 included in a second frame F2-60 generated by imaging at the second frame rate set by the setting unit 1212. The compression unit 902 compresses the first frame F1 and the combined second frame F2 obtained by the combining unit 1214.

[0329] This allows the imaging area at the second frame rate to be expanded, allowing the specific subject to be imaged at the second frame rate without creating an overlapping area Da1 between frames F1 and F2, thereby reducing image loss in frame F2-60, which is imaged at the second frame rate. Furthermore, since there is no need to interpolate the overlapping area Da1 during composition, a less unnatural image can be obtained and the load of the composition process can be reduced. Furthermore, since frame F2-60 is updated to frame F2 and then compressed, the difference between frames F1 and F2 can be minimized, reducing the load of the compression process.

[0330] (3-6) Furthermore, the setting program according to the third embodiment causes the processor 1201 to control the image sensor 100, which has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 fps) in the first imaging area, and is capable of setting a second frame rate (e.g., 60 fps) that is faster than the first frame rate in the second imaging area.

[0331] The setting program causes the processor 1201 to execute a detection process and a setting process. In the detection process, the setting program causes the processor 1201 to detect an imaging area p2-60 of a specific subject in the image sensor 100 based on a second image area r1-60 of the specific subject included in the frame F1 generated by output from the image sensor 100. In the setting process, the setting program causes the processor 1201 to set the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate the frame F1 and the detected imaging area p2-60 detected by the detection process to the second frame rate.

[0332] This allows the imaging area at the second frame rate to be expanded and a specific subject to be imaged at the second frame rate so that no overlapping range Da1 of non-image areas occurs between frames F1 and F2, and image loss in frame F2-60 imaged at the second frame rate can be suppressed using software.

[0333] (3-7) Furthermore, the image processing program of Example 3 causes the processor 1201 to perform image processing of frames generated by output from the image sensor 100, which has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging area, and is capable of setting a second frame rate (e.g., 60 [fps]) faster than the first frame rate in the second imaging area.

[0334] The image processing program causes the processor 1201 to execute a detection process, a setting process, and a composition process. In the detection process, the image processing program causes the processor 1201 to detect an imaging area p2-60 of a specific subject in the image sensor 100 based on a second image area r1-60 of the specific subject included in the frame F1 generated by output from the image sensor 100. In the setting process, the image processing program causes the processor 1201 to set the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate the frame F1 and the detected imaging area p2-60 detected by the detection process to the second frame rate.

[0335] In the synthesis process, the image processing program causes the processor 1201 to synthesize image data of the first image area r1-30 contained in the first frame F1 with image data of a specific imaging area P12-60 contained in the second frame F2-60 generated by imaging at the second frame rate set by the setting process.

[0336] This allows the imaging area at the second frame rate to be expanded, allowing the specific subject to be imaged at the second frame rate without creating an overlapping area Da1 between frames F1 and F2, and software can be used to reduce image loss in frame F2-60, which is imaged at the second frame rate. Furthermore, since there is no need to interpolate the overlapping area Da1 during composition, a less unnatural image can be obtained, and software can be used to reduce the load of the composition process. Furthermore, since frame F2-60 is updated to frame F2 and then compressed, the difference between frames F1 and F2 can be minimized, and software can be used to reduce the load of the compression process.

[0337] (3-8) Furthermore, the video compression program of Example 3 causes processor 1201 to compress video data including multiple frames generated by output from image sensor 100, which has a first imaging area for imaging a subject and a second imaging area for imaging a subject, and is capable of setting a first frame rate (e.g., 30 [fps]) in the first imaging area and a second frame rate (e.g., 60 [fps]) faster than the first frame rate in the second imaging area.

[0338] This video compression program causes the processor 1201 to execute a detection process, a setting process, a composition process, and a compression process. In the detection process, the video compression program causes the processor 1201 to detect an imaging area p2-60 of a specific subject in the image sensor 100 based on a second image area r1-60 of the specific subject included in the frame F1 generated by output from the image sensor 100. In the setting process, the video compression program causes the processor 1201 to set the frame rate of a specific imaging area P12-60 that includes the imaging area p1-60 of the specific subject used to generate the frame F1 and the detected imaging area p2-60 detected by the detection process to the second frame rate.

[0339] In the compositing process, the video compression program causes the processor 1201 to combine image data of a first image region r1-30 included in the first frame F1 with image data of a specific imaging region P12-60 included in the second frame F2-60 generated by imaging at the second frame rate set in the setting process. In the compression process, the video compression program causes the processor 1201 to compress the first frame F1 and the combined second frame F2 obtained by the compositing process.

[0340] This allows the imaging area at the second frame rate to be expanded, allowing the specific subject to be imaged at the second frame rate without creating an overlapping area Da1 between frames F1 and F2, and software can be used to reduce image loss in frame F2-60, which is imaged at the second frame rate. Furthermore, since there is no need to interpolate the overlapping area Da1 during composition, a less unnatural image can be obtained, and software can be used to reduce the load of the composition process. Furthermore, since frame F2-60 is updated to frame F2 and then compressed, the difference between frames F1 and F2 can be minimized, and software can be used to reduce the load of the compression process. [Explanation of symbols]

[0341] 100 imaging element, 200 imaging surface, 500 electronic device, 501 imaging optical system, 502 control unit, 503 liquid crystal monitor, 504 memory card, 505 operation unit, 507 flash memory, 508 sound recording unit, 600 video file, B1 to Bn data blocks, 817 additional information, 901 image processing unit, 902 compression unit, 1201 processor, 1202 memory, 1203 integrated circuit, 1204 bus, 1210 preprocessing unit, 1211 detection unit, 1212 setting unit, 1213 identification unit, 1214 synthesis unit, 1220 acquisition unit, 1301 subtraction unit, 1302 DCT unit, 1303 quantization unit, 1304 entropy coding unit, 1305 code amount control unit, 1306 inverse quantization unit, 1307 Inverse DCT unit, 1308 generation unit, 1309 frame memory, 1310 motion detection unit, 1311 motion compensation unit, 1312 compression control unit

Claims

[Claim 1] an acquisition unit that acquires video data including a plurality of frames output from an image sensor having a first imaging area for imaging a subject and a second imaging area for imaging the subject, the image sensor being capable of setting a first frame rate for the first imaging area and a second frame rate for the second imaging area that is faster than the first frame rate; a motion detection unit that sets a specific motion vector indicating that an object in the image data of a first image region in the frame captured at the first frame rate among the moving image data acquired by the acquisition unit, and that detects a motion vector in the image data of a second image region in the frame captured at the second frame rate; a motion compensation unit that performs motion compensation on the image data of the first image region based on the specific motion vector, and performs motion compensation on the image data of the second image region based on the motion vector detected by the motion detection unit; A video compression device having the above configuration.

Images