Moving image compression device

Abstract

To provide a moving image compression device, an electronic apparatus and a moving image compression program that classify and compress moving image data in which different imaging conditions are mixed.SOLUTION: In a moving image compression device that compresses a plurality of frames having a plurality of imaging areas for capturing a subject and output from an imaging device in which an imaging condition can be set for each imaging area, a control section 502 comprises: an acquisition unit that acquires a plurality of first frames 711 which are output from a first imaging area in which a first frame rate is set and a second imaging area in which a second frame rate is set, and acquires a plurality of second frames 713 which are output from the second imaging area; a complementing unit 702 that complements a defective area in which image data on the subject is not output from the first imaging area in each second frame, with a specific color to set it in a complementing area; and a compression unit that compresses the plurality of first frames and compresses the plurality of complemented second frames.SELECTED DRAWING: Figure 7

Description

Incorporation by Reference

[0001] This application claims the priority of Japanese Patent Application No. 2018-70203, filed on March 30, 2018, and incorporates the content thereof by reference into this application.

Technical Field

[0002] The present invention relates to a video compression device, an electronic device, and a video compression program.

Background Art

[0003] An imaging device equipped with an imaging element capable of setting different imaging conditions for each area is known (see Patent Document 1). However, video compression of frames captured under different imaging conditions has not been considered conventionally.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] The video compression device of the disclosed technology is a video compression device that has a plurality of imaging areas for imaging a subject and compresses a plurality of frames output from an imaging element capable of setting imaging conditions for each of the imaging areas, and includes an acquisition unit that acquires data output from a first imaging area with a first frame rate set and data output from a second imaging area with a second frame rate set, a generation unit that generates a plurality of first frames based on the data output from the first imaging area acquired by the acquisition unit and generates a plurality of second frames based on the data output from the second imaging area, and a compression unit that compresses the plurality of first frames generated by the generation unit and compresses the plurality of second frames.

[0006] The electronic device of the present disclosure technology has a plurality of imaging regions for imaging a subject, and an imaging element capable of setting imaging conditions for each of the imaging regions, data output from a first imaging region with a first frame rate set, and data output from a second imaging region with a second frame rate set, and an acquisition unit that acquires the data, a generation unit that generates a plurality of first frames based on the data output from the first imaging region acquired by the acquisition unit and generates a plurality of second frames based on the data output from the second imaging region, and a compression unit that compresses the plurality of first frames generated by the generation unit and compresses the plurality of second frames.

[0007] The video compression program of the present disclosure technology is a video compression program that causes a processor to execute compression of a plurality of frames output from an imaging element having a plurality of imaging regions for imaging a subject and capable of setting imaging conditions for each of the imaging regions. The processor is caused to execute an acquisition process of acquiring data output from a first imaging region with a first frame rate set and data output from a second imaging region with a second frame rate set, a generation process of generating a plurality of first frames based on the data output from the first imaging region acquired by the acquisition process and generating a plurality of second frames based on the data output from the second imaging region, and a compression process of compressing the plurality of first frames generated by the generation process and compressing the plurality of second frames.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

DETAILED DESCRIPTION OF THE INVENTION

[0009] <Configuration Example of Image Sensor> First, a stacked image sensor mounted on an electronic device will be described. The electronic device is, for example, an imaging device such as a digital camera or a digital video camera.

[0010] FIG. 1 is a cross-sectional view of a stacked image sensor 100. The stacked image sensor (hereinafter simply referred to as "image sensor") 100 includes a back-illuminated imaging chip (hereinafter simply referred to as "imaging chip") 113 that outputs a pixel signal corresponding to incident light, a signal processing chip 111 that processes the pixel signal, and a memory chip 112 that stores the pixel signal. These imaging chip 113, signal processing chip 111, and memory chip 112 are stacked and electrically connected to each other by bumps 109 having conductivity such as Cu.

[0011] As shown in FIG. 1, incident light mainly enters in the +Z-axis direction indicated by the white arrow. In the present embodiment, in the imaging chip 113, the surface on the side where the incident light enters is referred to as the back surface. Also, as shown in the coordinate axis 120, the left direction of the paper surface perpendicular to the Z-axis is defined as the +X-axis direction, and the front direction of the paper surface perpendicular to the Z-axis and the X-axis is defined as the +Y-axis direction. In some of the following figures, the coordinate axes are displayed based on the coordinate axes of FIG. 1 so that the orientation of each figure can be understood.

[0012] An example of the imaging chip 113 is a back-illuminated MOS (Metal Oxide Semiconductor) image sensor. The PD (photodiode) layer 106 is disposed on the back surface side of the wiring layer 108. The PD layer 106 has a plurality of PDs 104 arranged two-dimensionally and accumulating charges corresponding to incident light, and transistors 105 provided corresponding to the PDs 104.

[0013] A color filter 102 is provided via a passivation film 103 on the incident side of incident light in the PD layer 106. The color filter 102 has a plurality of types that transmit different wavelength regions, and has a specific arrangement corresponding to each of the PDs 104. The arrangement of the color filter 102 will be described later. A set of the color filter 102, the PD 104, and the transistor 105 forms one pixel.

[0014] On the incident side of incident light in the color filter 102, a microlens 101 is provided corresponding to each pixel. The microlens 101 condenses the incident light toward the corresponding PD 104.

[0015] The wiring layer 108 has a wiring 107 that transmits pixel signals from the PD layer 106 to the signal processing chip 111. The wiring 107 may be multilayered, and passive elements and active elements may be provided.

[0016] A plurality of bumps 109 are arranged on the surface of the wiring layer 108. The plurality of bumps 109 are aligned with a plurality of bumps 109 provided on the opposing surface of the signal processing chip 111, and the imaging chip 113 and the signal processing chip 111 are pressed or the like, so that the aligned bumps 109 are joined to each other and electrically connected.

[0017] Similarly, a plurality of bumps 109 are arranged on the opposing surfaces of the signal processing chip 111 and the memory chip 112. These bumps 109 are aligned with each other, and the signal processing chip 111 and the memory chip 112 are pressed or the like, so that the aligned bumps 109 are joined to each other and electrically connected.

[0018] Note that for the bonding between the bumps 109, not limited to Cu bump bonding by solid-phase diffusion, micro bump bonding by solder melting may also be employed. Also, for example, about one bump 109 may be provided for one block described later. Therefore, the size of the bump 109 may be larger than the pitch of the PD104. Also, in the peripheral region other than the pixel region where the pixels are arranged, bumps larger than the bump 109 corresponding to the pixel region may be provided together.

[0019] The signal processing chip 111 has TSVs (through-silicon vias) 110 that connect the circuits provided on the front and back surfaces to each other. The TSVs 110 are preferably provided in the peripheral region. Also, the TSVs 110 may be provided in the peripheral region of the imaging chip 113 and the memory chip 112 as well.

[0020] FIG. 2 is a diagram for explaining the pixel arrangement of the imaging chip 113. In particular, it shows how the imaging chip 113 is observed from the back side. (a) is a plan view schematically showing the imaging surface 200 which is the back surface of the imaging chip 113, and (b) is an enlarged plan view of a partial region 200a of the imaging surface 200. As shown in (b), a large number of pixels 201 are two-dimensionally arranged on the imaging surface 200.

[0021] Each pixel 201 has a color filter (not shown). The color filter consists of three types: red (R), green (G), and blue (B). The notations "R", "G", and "B" in (b) represent the types of color filters that the pixel 201 has. As shown in (b), on the imaging surface 200 of the image sensor 100, pixels 201 equipped with such color filters are arranged according to the so-called Bayer array.

[0022] A pixel 201 having a red filter photoelectrically converts the light in the red wavelength band of the incident light and outputs a received signal (photoelectric conversion signal). Similarly, a pixel 201 having a green filter photoelectrically converts the light in the green wavelength band of the incident light and outputs a received signal. Also, a pixel 201 having a blue filter photoelectrically converts the light in the blue wavelength band of the incident light and outputs a received signal.

[0023] The imaging device 100 is configured to be individually controllable for each block 202 composed of a total of four pixels 201 of 2×2 adjacent pixels. For example, when starting charge accumulation simultaneously for two different blocks 202, in one block 202, charge reading, that is, reading of the received light signal, is performed 1 / 30 second after the start of charge accumulation, and in the other block 202, charge reading can be performed 1 / 15 second after the start of charge accumulation. In other words, the imaging device 100 can set different exposure times (charge accumulation times, so-called shutter speeds) for each block 202 in one imaging.

[0024] In addition to the exposure time described above, the imaging device 100 can also vary the amplification factor (so-called ISO sensitivity) of the imaging signal for each block 202. The imaging device 100 can change the timing of starting charge accumulation and the timing of reading the received light signal for each block 202. That is, the imaging device 100 can change the frame rate during video imaging for each block 202.

[0025] Summarizing the above, the imaging device 100 is configured to be able to vary imaging conditions such as exposure time, amplification factor, and frame rate for each block 202. For example, if a readout line (not shown) for reading the imaging signal from a photoelectric conversion unit (not shown) provided in the pixel 201 is provided for each block 202 and configured to be able to read the imaging signal independently for each block 202, the exposure time (shutter speed) can be varied for each block 202.

[0026] Also, if an amplification circuit (not shown) for amplifying the imaging signal generated by the photoelectrically converted charge is provided independently for each block 202 and the amplification factor by the amplification circuit is configured to be controllable independently for each amplification circuit, the amplification factor (ISO sensitivity) of the signal can be varied for each block 202.

[0027] In addition to the imaging conditions described above, the imaging conditions that can be varied for each block 202 include the frame rate, gain, resolution (subsampling rate), the number of addition rows or columns for adding pixel signals, the charge accumulation time or number of accumulations, the number of bits for digitization, and the like. Further, the control parameter may be a parameter in image processing after obtaining an image signal from a pixel.

[0028] Also, for example, if a liquid crystal panel having compartments (one compartment corresponding to one block 202) that can be independently controlled for each block 202 is provided in the imaging device 100 and used as a dimming filter that can be turned on and off, it becomes possible to control the brightness (aperture value) for each block 202.

[0029] Note that the number of pixels 201 constituting the block 202 does not have to be the 4 pixels of 2×2 described above. The block 202 may have at least one pixel 201, or conversely, may have more than 4 pixels 201.

[0030] FIG. 3 is a circuit diagram of the imaging chip 113. In FIG. 3, a rectangle typically surrounded by a dotted line represents a circuit corresponding to one pixel 201. Also, a rectangle surrounded by a dashed-dotted line corresponds to one block 202 (202-1 to 202-4). Note that at least a part of each transistor described below corresponds to the transistor 105 in FIG. 1.

[0031] As described above, the reset transistor 303 of the pixel 201 is turned on / off in units of blocks 202. Also, the transfer transistor 302 of the pixel 201 is turned on / off in units of blocks 202. In the example shown in FIG. 3, a reset wiring 300-1 for turning on / off the four reset transistors 303 corresponding to the upper left block 202-1 is provided, and a TX wiring 307-1 for supplying a transfer pulse to the four transfer transistors 302 corresponding to the same block 202-1 is also provided.

[0032] Similarly, a reset wiring 300-3 for turning on / off four reset transistors 303 corresponding to the lower left block 202-3 is provided separately from the reset wiring 300-1. Also, a TX wiring 307-3 for supplying a transfer pulse to four transfer transistors 302 corresponding to the block 202-3 is provided separately from the TX wiring 307-1.

[0033] Similarly, for the upper right block 202-2 and the lower right block 202-4, a reset wiring 300-2 and a TX wiring 307-2, and a reset wiring 300-4 and a TX wiring 307-4 are provided in the respective blocks 202, respectively.

[0034] Sixteen PD104 corresponding to each pixel 201 are respectively connected to corresponding transfer transistors 302. A transfer pulse is supplied to the gate of each transfer transistor 302 via the TX wiring for each block 202. The drain of each transfer transistor 302 is connected to the source of a corresponding reset transistor 303, and a so-called floating diffusion FD between the drain of the transfer transistor 302 and the source of the reset transistor 303 is connected to the gate of a corresponding amplification transistor 304.

[0035] The drain of each reset transistor 303 is commonly connected to a Vdd wiring 310 to which a power supply voltage is supplied. A reset pulse is supplied to the gate of each reset transistor 303 via the reset wiring for each block 202.

[0036] The drain of each amplification transistor 304 is commonly connected to a Vdd wiring 310 to which a power supply voltage is supplied. Also, the source of each amplification transistor 304 is connected to the drain of a corresponding selection transistor 305. A selection pulse is supplied to the gate of each selection transistor 305 and is connected to a decoder wiring 308. The decoder wiring 308 is provided independently for each of the sixteen selection transistors 305.

[0037] Then, the sources of the respective selection transistors 305 are connected to a common output wiring 309. A load current source 311 supplies current to the output wiring 309. That is, the output wiring 309 for the selection transistors 305 is formed by a source follower. Note that the load current source 311 may be provided on the imaging chip 113 side or on the signal processing chip 111 side.

[0038] Here, the flow from the start of charge accumulation to the pixel output after the accumulation ends will be described. When a reset pulse is applied to the reset transistor 303 through the reset wiring for each block 202, and at the same time a transfer pulse is applied to the transfer transistor 302 through the TX wiring for each block 202 (202-1 to 202-4), the potentials of the PD104 and the floating diffusion FD are reset for each block 202.

[0039] When the application of the transfer pulse to each PD104 is released, each PD104 converts the incident light received into charge and accumulates it. Then, when the transfer pulse is applied again with no reset pulse applied, the accumulated charge is transferred to the floating diffusion FD, and the potential of the floating diffusion FD changes from the reset potential to the signal potential after charge accumulation.

[0040] Then, when a selection pulse is applied to the selection transistor 305 through the decoder wiring 308, the variation in the signal potential of the floating diffusion FD is transmitted to the output wiring 309 through the amplification transistor 304 and the selection transistor 305. As a result, the pixel signal corresponding to the reset potential and the signal potential is output from the unit pixel to the output wiring 309.

[0041] As described above, for the four pixels forming block 202, the reset wiring and the TX wiring are common. That is, the reset pulse and the transfer pulse are simultaneously applied to the four pixels within the same block 202, respectively. Therefore, all the pixels 201 forming a certain block 202 start charge accumulation at the same timing and end charge accumulation at the same timing. However, the pixel signals corresponding to the accumulated charges are selectively output from the output wiring 309 by sequentially applying selection pulses to the respective selection transistors 305.

[0042] In this way, the charge accumulation start timing can be controlled for each block 202. In other words, imaging can be performed at different timings between different blocks 202.

[0043] FIG. 4 is a block diagram showing a functional configuration example of the imaging device 100. The analog multiplexer 411 sequentially selects the 16 PDs 104 forming block 202 and outputs the respective pixel signals to the output wiring 309 provided corresponding to the block 202. The multiplexer 411 is formed on the imaging chip 113 together with the PD 104.

[0044] The pixel signals output via the multiplexer 411 are subjected to correlated double sampling (CDS) and analog / digital (A / D) conversion by a signal processing circuit 412 that performs CDS and A / D conversion and is formed on the signal processing chip 111. The A / D converted pixel signals are delivered to the demultiplexer 413 and stored in the pixel memories 414 corresponding to the respective pixels. The demultiplexer 413 and the pixel memories 414 are formed on the memory chip 112.

[0045] The arithmetic circuit 415 processes the pixel signals stored in the pixel memories 414 and delivers them to the subsequent image processing unit. The arithmetic circuit 415 may be provided on the signal processing chip 111 or on the memory chip 112. Note that FIG. 4 shows the connection for four blocks 202, but actually these exist for each of the four blocks 202 and operate in parallel.

[0046] However, the arithmetic circuit 415 does not necessarily exist for every four blocks 202. For example, one arithmetic circuit 415 may sequentially process while referring to the values of the pixel memories 414 corresponding to each of the four blocks 202 in order.

[0047] As described above, output wirings 309 are provided corresponding to each of the blocks 202. Since the imaging device 100 stacks the imaging chip 113, the signal processing chip 111, and the memory chip 112, by using electrical connections between chips using the bumps 109 for these output wirings 309, the wirings can be routed without increasing the chips in the plane direction.

[0048] <Block Configuration Example of Electronic Device> FIG. 5 is an explanatory diagram showing a block configuration example of an electronic device. The electronic device 500 is, for example, a lens-integrated camera. The electronic device 500 includes an imaging optical system 501, an imaging device 100, a control unit 502, a liquid crystal monitor 503, a memory card 504, an operation unit 505, a DRAM 506, a flash memory 507, and a recording unit 508. The control unit 502 includes a compression unit that compresses moving image data as will be described later. Therefore, among the electronic device 500, at least the configuration including the control unit 502 serves as a moving image compression device, a decompression device, and a playback device. Further, the memory card 504, the DRAM 506, and the flash memory 507 constitute a storage device 1202 to be described later.

[0049] The imaging optical system 501 is composed of a plurality of lenses and forms a subject image on the imaging surface 200 of the imaging device 100. In FIG. 5, for convenience, the imaging optical system 501 is illustrated as a single lens.

[0050] The imaging device 100 is, for example, an imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and images a subject image formed by the imaging optical system 501 to output an imaging signal. The control unit 502 is an electronic circuit that controls each part of the electronic device 500, and is composed of a processor and its peripheral circuits.

[0051] A flash memory 507, which is a non-volatile storage medium, has a predetermined control program written therein in advance. The processor of the control unit 502 reads and executes the control program from the flash memory 507 to control each part. This control program uses a DRAM 506, which is a volatile storage medium, as a working area.

[0052] The liquid crystal monitor 503 is a display device using a liquid crystal panel. The control unit 502 repeatedly causes the imaging device 100 to image a subject image at a predetermined cycle (for example, 1 / 60 second). Then, various image processes are performed on the imaging signal output from the imaging device 100 to create a so-called through image, which is displayed on the liquid crystal monitor 503. On the liquid crystal monitor 503, in addition to the above through image, for example, a setting screen for setting imaging conditions is displayed.

[0053] Based on the imaging signal output from the imaging device 100, the control unit 502 creates an image file to be described later and records the image file on a memory card 504, which is a portable recording medium. The operation unit 505 has various operation members such as push buttons, and outputs an operation signal to the control unit 502 in response to the operation of these operation members.

[0054] The recording unit 508 is composed of, for example, a microphone, converts ambient sound into an audio signal, and inputs it to the control unit 502. Note that the control unit 502 may record the video file not on the memory card 504, which is a portable recording medium, but on a recording medium (not shown) such as an SSD (Solid State Drive) or a hard disk built in the electronic device 500.

[0055] <Relationship between imaging surface 200 and subject image> FIG. 6 is an explanatory diagram showing the relationship between imaging surface 200 and subject image. (a) schematically shows the imaging surface 200 (imaging range) of imaging device 100 and subject image 601. In (a), control unit 502 captures subject image 601. The imaging in (a) may also serve as imaging for creating, for example, a live view image (so-called through image).

[0056] Control unit 502 executes a predetermined image analysis process on subject image 601 obtained by the imaging in (a). The image analysis process is a process of detecting a main subject by, for example, a well-known subject detection technique (a technique of calculating feature amounts and detecting a range where a predetermined subject exists). In Example 1, except for the main subject, it is regarded as the background. Since the main subject is detected by the image analysis process, imaging surface 200 is divided into main subject area 602 where the main subject exists and background area 603 where the background exists.

[0057] Note that in (a), an area roughly including subject image 601 is shown as main subject area 602, but main subject area 602 may have a shape along the outer shape of subject image 601. That is, main subject area 602 may be set so as to include as little as possible things other than subject image 601.

[0058] Control unit 502 sets different imaging conditions for each block 202 in main subject area 602 and each block 202 in background area 603. For example, a faster shutter speed is set for each block 202 in the former than for each block 202 in the latter. By doing so, in the imaging of (c) imaged after the imaging of (a), image blur is less likely to occur in main subject area 602.

[0059] In addition, when the main subject area 602 is in a backlit state due to the influence of a light source such as the sun existing in the background area 603, the control unit 502 sets a relatively high ISO sensitivity or a low shutter speed for each of the former blocks 202. Also, the control unit 502 sets a relatively low ISO sensitivity or a high shutter speed for each of the latter blocks 202. By doing so, in the imaging of (c), it is possible to prevent the blackening of the backlit main subject area 602 and the overexposure of the background area 603 with a large amount of light.

[0060] Note that the image analysis process may be a process different from the process of detecting the main subject area 602 and the background area 603 described above. For example, it may be a process of detecting a portion with a brightness equal to or higher than a certain level (too bright portion) or a portion with a brightness lower than a certain level (too dark portion) in the entire imaging surface 200. When the image analysis process is such a process, the control unit 502 may set the shutter speed and ISO sensitivity so that the exposure value (Ev value) of the blocks 202 included in the former area is lower than that of the blocks 202 included in other areas.

[0061] Also, for the blocks 202 included in the latter area, the control unit 502 sets the shutter speed and ISO sensitivity so that the exposure value (Ev value) is higher than that of the blocks 202 included in other areas. By doing so, the dynamic range of the image obtained by the imaging of (c) can be made wider than the original dynamic range of the imaging device 100.

[0062] (b) of FIG. 6 shows an example of the mask information 604 corresponding to the imaging surface 200 shown in (a). "1" is stored at the position of the block 202 belonging to the main subject area 602, and "2" is stored at the position of the block 202 belonging to the background area 603, respectively.

[0063] The control unit 502 performs image analysis processing on the image data of the first frame and detects the main subject area 602. As a result, the frame captured in (a) is divided into the main subject area 602 and the background area 603 as shown in (b). The control unit 502 sets different imaging conditions for each block 202 in the main subject area 602 and each block 202 in the background area 603, performs the imaging in (c), and creates image data. An example of the mask information 604 at this time is shown in (d).

[0064] Since the mask information 604 in (b) corresponding to the result of the imaging in (a) and the mask information 604 in (d) corresponding to the result of the imaging in (c) are captured at different times (there is a time difference), for example, when the subject is moving or when the user moves the electronic device 500, these two pieces of mask information 604 will have different contents. In other words, the mask information 604 is dynamic information that changes over time. Therefore, different imaging conditions are set for each block 202 for each frame.

[0065] Hereinafter, embodiments of video compression, video file generation, decompression, and playback using the above-described imaging device 100 will be described.

Embodiment

[0066] <Video Compression and Decompression Example> FIG. 7 is an explanatory diagram showing a video compression and decompression example according to Embodiment 1. The electronic device 500 includes the above-described imaging device 100 and a control unit 502. The control unit 502 includes a first generation unit 701, a compression / decompression unit 702, a synthesis unit 703, and a playback unit 704. As described above, the imaging device 100 has a plurality of imaging regions for imaging a subject. The imaging region is a set of pixels of at least one pixel or more, for example, one or more of the above-described blocks 202. Imaging conditions (for example, frame rate, exposure time, ISO sensitivity) can be set for each block 202 in the imaging region.

[0067] Here, among the imaging surface 200, the imaging area where the first frame rate (for example, 30 [fps]) is set is referred to as the "first imaging area", and the imaging area where the second frame rate (for example, 60 [fps]), which is faster than the first frame rate, is set is referred to as the "second imaging area". Note that the values of the first frame rate and the second frame rate are examples, and other values may be used as long as the second frame rate is faster than the first frame rate. If the second frame rate is a multiple of the first frame rate, frames output from the first imaging area and the second imaging area can be obtained at the imaging timing of the first frame rate.

[0068] The imaging device 100 images a subject and outputs input moving image data 710 to the first generation unit 701. The area of the image data output from a certain imaging area of the imaging device 100 is referred to as an "image area" (corresponding to the imaging area).

[0069] For example, when the entire imaging surface 200 is the first imaging area set at the first frame rate (30 [fps]), the image data of the first image area a1 (hatched) output from the first imaging area (the entire imaging surface 200) by imaging at the first frame rate (30 [fps]) becomes one frame by image processing. This frame is referred to as the "first frame 711".

[0070] Specifically, for example, when taking a fixed-point photograph of a landscape, the first frame 711 is generated as the image data of the first image area a1 of only the landscape by imaging at the first frame rate (30 [fps]).

[0071] Also, for example, when the entire imaging surface 200 is a first imaging region set to a first frame rate (30 [fps]), and when the imaging region in which a specific subject is detected is changed from the first imaging region to a second imaging region set to a second frame rate (60 [fps]), a combination of the image data of the first image region a1 (hatched) output from the first imaging region by imaging at the first frame rate (30 [fps]) and the image data of the second image region a2 output from the second imaging region is also the first frame 711.

[0072] Specifically, for example, when a specific subject (train) is detected during fixed-point shooting of a landscape, the first frame 711 is generated as a combination of the image data of the landscape (first image region a1) excluding the train obtained at the first frame rate (30 [fps]) by imaging at the first frame rate (30 [fps]) and the image data of the train (second image region a2) obtained at the second frame rate (60 [fps]).

[0073] Also in this case, the image data of the second image region a2 output from the second imaging region of the imaging surface 200 by imaging at the second frame rate (60 [fps]) is referred to as "image data 712". In this case, the image region from which the image data of the subject is not output from the first imaging region is referred to as "missing region 712x".

[0074] Specifically, for example, when a specific subject (train) is detected during fixed-point shooting of a landscape, the image data of the train (second image region a2) obtained at the second frame rate (60 [fps]) by imaging at the second frame rate (60 [fps]) is the image data 712.

[0075] Note that there may be three or more imaging regions with different frame rates set. In this case, for the third imaging region and subsequent regions, a frame rate different from the first frame rate and the second frame rate can be set.

[0076] The first generation unit 701 complements the image data 712 among the input video data 710 input from the imaging device 100. Specifically, for example, the first generation unit 701 complements the missing area 712x where no image signal is output from the first imaging area of the imaging device 100 with a specific color. In this example, the specific color is black, and it is also painted black in FIG. 7. Note that the specific color may be a color other than black, or a specific pattern. Also, the specific color may be not a single color but a plurality of colors. Also, the pixel area around the second image area a2 may be made the same color as the boundary of the second image area a2. Also, the missing area 712x complemented with the specific color is referred to as the "complemented area 712y".

[0077] The image data obtained by combining the image data 712 and the complemented area 712y through image processing is referred to as the second frame 713. Also, the video data composed of the first frame 711 group is referred to as the first video data 721, and the video data composed of the second frame 713 group is referred to as the second video data 722. The first generation unit 701 outputs the first video data 721 and the second video data 722 to the compression / decompression unit 702.

[0078] The compression / decompression unit 702 compresses the first video data 721 and the second video data 722 respectively and stores them in a storage device (for example, a memory card 504, a flash memory 507). The compression / decompression unit 702 compresses, for example, by a hybrid encoding that combines entropy encoding with motion compensation frame-interpolation (Motion Compensation: MC) and discrete cosine transform (Discrete Cosine Transform: DCT).

[0079] The compression and decompression unit 702 performs compression processing that does not require motion detection or motion compensation on the first image region a1 indicated by hatching among the first frames 711 that make up the first video data 721. For the image data 712 of the second image region a2 from which the hatched specific subject image is output, the compression and decompression unit 702 compresses it by the hybrid encoding described above. In this way, for the first image region a1 other than the specific subject image, since motion detection and motion compensation are not performed, it is possible to reduce the processing load of video compression.

[0080] Note that for the first image region a1, assuming that there is no camera shake in the imaging device or that the subject is not moving, compression processing that does not require motion detection or motion compensation is to be performed. However, when there is camera shake or subject movement, the compression and decompression unit 702 may compress the first image region a1 by the hybrid encoding described above.

[0081] Similarly, the compression and decompression unit 702 performs compression processing that does not require motion detection or motion compensation on the complementary region 712y indicated by black filling among the second frames 713 that make up the second video data 722. For the image data 712 of the second image region a2 from which the hatched specific subject image is output, the compression and decompression unit 702 compresses it by the hybrid encoding described above. In this way, for the complementary region 712y (black filling) other than the specific subject image, since motion detection and motion compensation are not performed, it is possible to reduce the processing load of video compression. Also, when there is camera shake or subject movement, the compression and decompression unit 702 may compress the complementary region 712y by the hybrid encoding described above.

[0082] In this way, the second frame 713 obtained at the second frame rate (60 [fps]) has the same size as the first frame 711 obtained at the first frame rate (30 [fps]). Therefore, since the same compression processing as the compression processing of the first frame 711 is applied to the second frame 713, there is no need to apply other compression processing that conforms to the size of the image data 712.

[0083] In addition, when there is a video playback instruction or an expansion instruction, the compression and expansion unit 702 expands the compressed first video data 721 and second video data 722 to restore them to the original first video data 721 and second video data 722.

[0084] The composition unit 703 refers to the first frame 711 that is one frame earlier in time than the second frame 713, and copies, that is, composes, the first frame 711 to the second frame 713. Specifically, the composition unit 703 generates another first frame 711 to be composed into the second frame by copying, and composes the generated first frame and the second frame. The composed frame is referred to as the "third frame 730". The third frame 730 is a frame in which the specific subject image (second image area a2) in the second frame 713 is superimposed on the subject image of the first frame 711. The composition unit 703 outputs video data (hereinafter, fourth video data) 740 including the first frame 711 output at an imaging rate of 30 [fps] and the third frame 730 which is a composed frame to the playback unit 704. Note that when there is no composition instruction, for example, when it is desired to play back a video at 30 fps, the composition unit 703 does not execute the composition process.

[0085] The playback unit 704 plays back the fourth video data 740 and displays the video on the liquid crystal monitor 503. As described above, the input video data 710 as it is cannot be compressed by the compression and expansion unit 702. Therefore, the first generation unit 701 generates second video data 722 composed of a plurality of second frames 713 by complementing the image data 712 in the complement area 712y. The compression and expansion unit 702 separately compresses and expands the first video data 721 and the second video data 722.

[0086] Thereby, the general-purpose compression and expansion unit 702 enables the compression of the second video data 722 in the same manner as normal video data (first video data 721) in the compression and expansion unit 702. Note that when the composition process is not executed by the composition unit 703, the playback unit 704 plays back the first video data 721 with a frame rate of 30 [fps] and displays the video on the liquid crystal monitor 503.

[0087] In the above example, the entire imaging surface 200 is the first imaging area set at the first frame rate (30 [fps]), and the case where the imaging area in which a specific subject is detected is changed from the first imaging area to the second imaging area set at the second frame rate (60 [fps]) has been described. However, the setting of the imaging conditions for the imaging area of the imaging surface 200 is not limited to this.

[0088] For example, when a plurality of first imaging areas set at the first frame rate (30 [fps]) and a plurality of second imaging areas set at the second frame rate (60 [fps]) are mixed on the imaging surface 200, such as in a staggered arrangement, the image data combined by a plurality of first image areas a1 corresponding to the plurality of first imaging areas becomes the first frame F711. Also, in this case, the image data combined by a plurality of second image areas a2 corresponding to the plurality of second imaging areas becomes the "second frame F712". In the staggered arrangement, the frame rates of the first imaging area and the second imaging area may be the same, and other imaging conditions such as the exposure time, ISO sensitivity, and decimation rate may be set differently between the first imaging area and the second imaging area.

[0089] <Example of the file format of a video file> FIG. 8 is an explanatory diagram showing an example of the file format of a video file. In FIG. 8, for example, the case where a file format conforming to MPEG4 (Moving Picture Experts Group phase 4) is applied will be described as an example.

[0090] The video file 800 is a collection of data called boxes, and for example, has a header part 801 and a data part 802. The header part 801 includes, as boxes, ftyp811, uuid812, and moov813. The data part 802 includes, as a box, mdat820.

[0091] The ftyp811 is a box that stores information indicating the type of the video file 800, and is placed at a position before other boxes within the video file 800. The uuid812 is a box that stores a universal unique identifier and is extensible by the user. In Example 1, for example, frame rate identification information for identifying whether the video data in this video file 800 has a frame rate of only the first frame rate (e.g., 30 [fps]) or video data (the first video data 721 and the second video data 722) including the first frame rate and the second frame rate (60 [fps]) may be written into the uuid812. Thereby, it is possible to specify which video data has which frame rate during extension, composition, and playback.

[0092] The moov813 is a box that stores metadata related to various media such as video, audio, and text. The mdat820 is a box that stores data of various media such as video, audio, and text.

[0093] Next, the boxes within the moov813 will be specifically described. The moov813 has a uuid831, a udta832, an mvhd833, trak834a, 834b, and additional information 835. When not distinguishing between trak834a and 834b, it is simply denoted as trak834. Similarly, when not distinguishing between tkhd841a etc. of the data within trak834a and tkhd841b etc. of the data within trak834b, it is simply referred to as tkhd841.

[0094] The uuid831 is, like the uuid812, a box that stores a universal unique identifier and is extensible by the user. In Example 1, for example, during the generation of the video file 800, frame type identification information for identifying whether each of the frames in this video file 800 is the first frame 711 or the second frame 713 is written into the uuid831 in association with the frame numbers.

[0095] In addition, information indicating the storage positions of the compressed data of the first video data 721 and the compressed data of the second video data 722 may be written into uuid831. Specifically, for example, as information indicating the storage position of the compressed data of the first video data 721, (SOM (Start Of Movie) 850a, EOM (End Of Movie) 854a) is written, and as information indicating the storage position of the compressed data of the second video data 722, (SOM850b, EOM854b) is written. Thereby, it is possible to specify which video data is stored at which storage position during decompression, composition, and playback.

[0096] Note that the storage positions of the compressed data can be specified by stsz847a, 847b and stco848a, 848b which will be described later. Therefore, instead of SOM850a and EOM854a, the addresses of the compressed data of the first video data 721 specified from stsz847a, 847b and stco848a, 848b may be associated with the first frame rate information indicating the first frame rate and set in stsz847a, 847b and stco848a, 848b as information indicating the storage position.

[0097] Similarly, instead of SOM850b and EOM854b, the addresses of the compressed data of the second video data 722 specified from stsz847a, 847b and stco848a, 848b may be associated with the second frame rate information indicating the second frame rate and set in stsz847a, 847b and stco848a, 848b as information indicating the storage position.

[0098] udta832 is a box for storing user data. Examples of user data include an identification code of an electronic device and location information of the electronic device.

[0099] mvhd833 is a box that stores the time scale and the duration for each trak834. The time scale is the frame rate or the sampling frequency. The duration is the length based on the time scale. Dividing the duration by the time scale gives the time length of the media specified by that trak834.

[0100] trak834 is a box set for each type of media (video, audio, text). In this embodiment, moov includes trak834a and 834b. trak834a is, for example, a box that stores metadata regarding the video, audio, and text of the first video data 721 output at an imaging of 30 [fps].

[0101] trak834a is set for each of the video, audio, and text of the first video data 721. trak834b is, for example, a box that stores metadata regarding the video, audio, and text of the second video data 722 output at an imaging of 60 [fps]. trak834b is also set for each of the video, audio, and text of the second video data 722.

[0102] Additional information 835 is a box that includes imaging condition information and insertion position information. The imaging condition information is information indicating the storage position of the media within the video file 800 for each imaging condition (such as the frame rate like 30 [fps], 60 [fps]). The insertion position information is information indicating the position where the data of the media with the higher frame rate (the second video data 722) is inserted into the data of the media with the lower frame rate (the first video data 721).

[0103] Next, the box within trak834 will be specifically described. trak834a and 834b each have tkhd841a, 841b, edts842a, 842b, tref843a, 843b, stsc844a, 844b, stts845a, 845b, stss846a, 846b, stsz847a, 847b, and stco848a, 848b. When not distinguishing between tkhd841a~stco848a and tkhd841b~stco848b, they are simply denoted as tkhd841~stco848.

[0104] tkhd841 is a box that stores the basic attributes of trak834, such as the playback time, display resolution, and identification code that determines the type of media of trak834. For example, if trak834 is a video, the media ID = 1; if it is audio, the media ID = 2; if it is text, the media ID = 3.

[0105] edts842 is a box that stores, as the edit list of trak834, the playback start position of trak834 and the playback time from that playback position. tref843 is a box that stores the reference information between trak834s. For example, when a video trak834 references a text trak834 as a chapter, the media ID = 3 indicating the text trak834 is stored in the tref843 of the video trak834, and "chap" is stored as the identification code for referencing the text trak834 as a chapter.

[0106] stsc844 is a box that stores the number of samples within one chunk. A chunk is a collection of media data for the number of samples and is stored in mdat820. For example, when the media is a video, the samples within a chunk are frames. If the number of samples is "3", it means that 3 frames are stored within one chunk.

[0107] stts845 is a box that stores the playback time for each chunk within its trak834 or for samples within a chunk. stss846 is a box that stores information regarding the interval of key frames (I pictures). When the GOP (Group of Pictures) is "5", "1,6,11,···" is stored in stss846.

[0108] stsz847 is a box that stores the data size for each sample within mdat820. stco848 is a box that stores the offset from the start address of the video file 800 for each chunk within mdat820. By referring to stsz847 and stco848, it becomes possible to identify the position of the media data (frames, audio data, text (chapters)) within mdat820.

[0109] mdat820 is a box that stores media-specific chunks. SOM850a,850b (when not distinguishing, denoted as SOM850) are identifiers that indicate the start position of the storage of a chunk group under certain imaging conditions. Also, EOM854a,854b (when not distinguishing, denoted as EOM854) are identifiers that indicate the end position of the storage of a chunk group under certain imaging conditions.

[0110] In Figure 8, mdat820 stores video chunk 851-1, audio chunk 852-1, text chunk 853-1, …, video chunk 851-2, audio chunk 852-2, text chunk 853-2, …, video chunk 851-3, audio chunk 852-3, text chunk 853-3.

[0111] In this example, since video shooting is performed under two imaging conditions (30 [fps], 60 [fps]), the chunks are divided for each imaging condition. Specifically, for example, the chunk group obtained at the imaging timing of 30 [fps] is stored from SOM850a to EOM854a, and the chunk group obtained at the imaging timing of 60 [fps] is stored from SOM850b to EOM854b.

[0112] Video chunk 851-1 stores the compressed frames of the first frame 711 before specific subject detection, which are samples output at 30 [fps] imaging, that is, the compressed frames 861-s1, 861-s2, 861-s3. Video chunk 851-2 stores the compressed frames of the first frame 711 at the time of specific subject detection, which are samples output at 30 [fps] imaging, that is, the compressed frames 862-s1, 862-s2, 862-s3. Since the imaging timing of frames 862-s1, 862-s2, 862-s3 overlaps with 60 [fps], the specific subject image (second image area a2) at 60 [fps] is included.

[0113] Video chunk 851-3 stores the compressed frames of the second frame 713 at the time of specific subject detection, which are samples output at 60 [fps] imaging, that is, the compressed frames 863-s1, 863-s2, 863-s3.

[0114] <Additional Information> FIG. 9 is an explanatory diagram showing the relationship between a frame and additional information 835. (A) shows an example of the data structure of frame F. Frame F has a frame number 901 and frame data 902. The frame data 902 is image data generated by imaging.

[0115] (B) shows a series of compressed frames. In (B), the compressed frames are arranged in time series from left (oldest) to right (newest). #1a to #6a are the frame numbers of the compressed frames 861-s1, 861-s2, 861-s3, 862-s1, 862-s2, 862-s3 output at 30 [fps] imaging. #1b to #3b are the frame numbers of the compressed frames 863-s1, 863-s2, 863-s3 output at 60 [fps] imaging.

[0116] (C) shows an example of the data structure of the additional information 835. The additional information 835 has imaging condition information 910 and insertion position information 920. As described above, the imaging condition information 910 is information indicating the storage position of media in the video file 800 for each imaging condition (for example, frame rate such as 30 [fps] or 60 [fps]). The imaging condition information 910 has frame rate information 911 and position information 912.

[0117] The frame rate information 911 is, for example, a frame rate such as 30 [fps] or 60 [fps]. The position information 912 is information indicating the storage position of the compressed frame in the video file 800 and is specified by referring to stsz847 and stco848. Specifically, for example, the value Pa of the position information 912 of the compressed frame when the frame rate information 911 is 30 [fps] indicates an address in the range of SOM850a to EOM854a. Similarly, the value Pb of the position information 912 of the compressed frame when the frame rate information 911 is 60 [fps] indicates an address in the range of SOM850b to EOM854b.

[0118] The insertion position information 920 is information indicating the position where the data of the media with the higher frame rate (60 [fps]) (the second video data 722) is inserted into the data of the media with the lower frame rate (30 [fps]) (the first video data 721). The insertion position information 920 has an insertion frame number 921 and an insertion destination 922. The insertion frame number 921 indicates the frame number of the compressed frame to be inserted. In this example, the compressed frames 863-s1, 863-s2, and 863-s3 specified by frame numbers #1b to #3b are the compressed frames to be inserted.

[0119] The insertion destination 922 indicates the insertion position of the compressed frame specified by the insertion frame number 921. Specifically, the insertion destination 922 is specified, for example, between two frame numbers. For example, the compressed frame 863-s1 with the insertion frame number #1b is inserted between the compressed frame 861-s3 and the compressed frame 862-s1 specified by the two frame numbers (#3a, #4a) of the insertion destination 922. In FIG. 9, the insertion destination 922 is specified using the frame number, but it may also be specified by referring to the addresses (stsz847 and stco848).

[0120] In FIGS. 8 and 9, an example of storing the compressed data of the first frame 711 and the compressed data of the second frame 713 in one video file 800 has been described. However, a video file compressed from the first frame 711 and a video file compressed from the second frame 713 may be generated separately. In this case, the header part 801 of both video files 800 stores the association information by which one video file 800 is associated with the other video file 800. The association information is stored, for example, in the uuid812, 831 and mvhd833 of the header part 801.

[0121] Thereby, similar to the case of collecting in one video file 800, decompression, synthesis, and playback can be performed. For example, when the first frame rate is selected, the video file compressed from the first frame 711 is decompressed and played back. When the second frame rate is selected, the video file 800 compressed from the first frame 711 and the video file 800 compressed from the second frame 713 are decompressed, and synthesis and playback are executed.

[0122] Note that if the additional information 835 is stored in the moov813, it may be stored in other boxes (831 to 834).

[0123] <Example of synthesis process> FIG. 10 is an explanatory diagram showing a first example of the synthesis process in the synthesis unit 703 shown in FIG. 7. In the first example of the synthesis process, while the electronic device 500 is performing fixed-point shooting of a landscape including a paddy field, mountains, and the sky, a moving train is captured as a specific subject. The train as the specific subject is identified by the well-known subject detection technology described above. The captured frames F are set as frames F1, F2-60, F3, F4-60, and F5 in chronological order. Here, it is assumed that the train travels from right to left within frames F1, F2-60, F3, F4-60, and F5.

[0124] Frames F1, F3, and F5 are the first frame 711 including the image data of the first image area a1 output by imaging at the first frame rate of 30 [fps] in the first imaging area and the image data of the second image area a2 output by imaging at the second frame rate of 60 [fps] in the second imaging area. Frames F2-60 and F4-60 are the second frames 713 including the image data of the second image area a2 output by imaging at the second frame rate of 60 [fps] in the second imaging area and having the background complemented with black painting.

[0125] Specifically, for example, frames F1, F3, and F5 are the first frame 711 in which a landscape including a paddy field, mountains, and the sky is imaged in the first image area a1 and a moving train is imaged as a specific subject in the second image area a2. Frames F2-60 and F4-60 are the frames in which the train is imaged in the second image area a2.

[0126] That is, in frames F1, F2-60, F3, F4-60, and F5, the image data of the second image area a2 in which the train is imaged is the image data output by imaging in the second imaging area (60 [fps]). Also, in frames F1, F3, and F5, the image data of the first image area a1 in which the landscape is imaged is the image data output by imaging in the first imaging area (30 [fps]). Since the first image area a1 is output by imaging at the first frame rate (30 [fps]), the complementary area 712y of frames F2-60 and F4-60 output by imaging at the second frame rate (60 [fps]) is filled with a specific color (black).

[0127] Frames F1, F2-60, F3, F4-60, F5... correspond to the first video data 721 and the second video data 722 described above. Since the second video data 722 includes the second frame 713 in which the complement region 712y is painted black, the synthesizing unit 703 synthesizes the first video data 721 and the second video data 722.

[0128] Specifically, for example, the synthesizing unit 703 copies the image data (scenery excluding the train) of the first image region a1 of the frame F1 that is one frame earlier in time than the frame F2-60 to the image data (train) of the second image region a2 of the frame F2-60. Thereby, the synthesizing unit 703 generates the frame F2 which is the third frame 730.

[0129] Similarly for the frame F4-60, the synthesizing unit 703 copies the image data (scenery excluding the train) of the first image region a1 of the frame F3 that is one frame earlier in time than the frame F4-60 to the image data (train) of the second image region a2 of the frame F4-60. Thereby, the synthesizing unit 703 generates the frame F4 which is the third frame 730. Then, the synthesizing unit 703 outputs the fourth video data 740 including the frames F1 to F5.

[0130] In this way, by setting the first image regions a1 of the frames F1 and F3 with the frame rate of one frame earlier in time in the complement regions 712y of the frames F2-60 and F4-60, the difference between the frames F1 and F2 and the difference between the frames F3 and F4 for the first image region a1 can be made almost zero. Thereby, a video without a sense of discomfort can be played.

[0131] Therefore, it becomes possible to reproduce the fourth video data 740 which is a frame sequence in which the first frame 711 and the third frame 730 are mixed. Also, the first video data 721 and the second video data 722 can be respectively decompressed by the conventional compression / decompression unit 702, and the processing load of the decompression process can be reduced. Also, when reproducing at 30 [fps], since the compression / decompression unit 702 only decompresses the first video data 721 and synthesis by the synthesis unit 703 is unnecessary, the efficiency of the reproduction process can be improved.

[0132] In frame F2, the image data of the first image area a1 of frame F1 (the scenery excluding the train) is replicated. Therefore, the part that was originally the second image area a2 of frame F1 (the end of the train) is not replicated to frame F2. For this reason, frame F2 has a complementary image part Da1 where nothing is output.

[0133] Similarly, in frame F4, the image data of the first image area a1 of frame F3 (the scenery excluding the train) is replicated. Therefore, the part that was originally the second image area a2 of frame F3 (the end of the train) is not replicated to frame F4. For this reason, frame F4 has a complementary image part Da3 where nothing is output.

[0134] In the first embodiment, for the complementary image parts Da1, Da3, the synthesis unit 703 may leave them in a specific color that is filled in, or may perform interpolation processing using surrounding pixels. Thereby, video compression is possible, and frames F2, F4,... with less discomfort can be reproduced.

[0135] FIG. 11 is an explanatory diagram showing a second example of the synthesis process in the synthesis unit 703 shown in FIG. 7. In the second example of the synthesis process, the electronic device 500 is, for example, a drive recorder, and photographs a vehicle traveling ahead (preceding vehicle) and the scenery. In this case, the preceding vehicle is a specific subject to be followed, and the scenery changes by self-driving. The frames F generated by the photographing are frames F6, F7-60, F8, F9-60, F10 in chronological order.

[0136] Frames F6, F8, and F10 are the first frame 711 that includes the image data of the first image area a1 output by imaging the first imaging area at the first frame rate of 30 [fps] and the image data 712 of the second image area a2 output by imaging the second imaging area at the second frame rate of 60 [fps]. Frames F7-60 and F9-60 are the image data 712 of the second image area a2 output by imaging the second imaging area at the second frame rate of 60 [fps].

[0137] Specifically, for example, frames F6, F8, and F10 are the first frame 711 in which a preceding vehicle is imaged in the first image area a1 and a changing landscape is imaged in the second image area a2. Frames F7-60 and F9-60 are frames in which a landscape is imaged in the second image area a2.

[0138] That is, in frames F6, F7-60, F8, F9-60, and F10, the image data of the second image area a2 in which a landscape is imaged is the image data output by imaging the second imaging area (60 [fps]). Also, in frames F6, F8, and F10, the image data of the first image area a1 in which a preceding vehicle is imaged is the video data output by imaging the first imaging area (30 [fps]). Since the first imaging area is output by imaging at the first frame rate (30 [fps]), the first image area a1 of frames F7-60 and F9-60 output by imaging at the second frame rate (60 [fps]) is blacked out by the first generation unit 701 during compression.

[0139] The synthesizing unit 703 copies the image data (landscape-excluding preceding vehicle) of the first image area a1 of the frame F6 that is one frame earlier in time than frame F7-60 to the image data (landscape) of the second image area a2 of frame F7-60. Thereby, the synthesizing unit 703 generates frame F7 which is the third frame 730.

[0140] Similarly for frame F9, the synthesizing unit 703 copies the image data of the first image area a1 of frame F8 (the preceding vehicle excluding the scenery), which is one frame earlier in time than frame F9-60, to the image data of the second image area a2 of frame F9-60 (the scenery). Thereby, the synthesizing unit 703 generates frame F9, which is the third frame 730. Then, the synthesizing unit 703 outputs the fourth video data 740 including frames F6 to F10.

[0141] In this way, by setting the second image areas a2 of frames F6 and F8 at the previous frame rate one frame earlier in time in the complementary areas 712y of frames F7-60 and F9-60, the difference between frames F6 and F7 and the difference between frames F8 and F9 for the first image area a1 can be made zero.

[0142] Therefore, it becomes possible to reproduce the fourth video data 740, which is a frame sequence in which the first frame 711 and the image data 712 are mixed. Also, the first video data 721 and the second video data 722 can be respectively decompressed by the conventional compression / decompression unit 702, and the processing load of the decompression process can be reduced. Further, when reproducing at 30 [fps], the compression / decompression unit 702 only decompresses the first video data 721, and since synthesis by the synthesizing unit 703 is not required, the efficiency of the reproduction process can be improved.

[0143] <Configuration example of control unit 502> FIG. 12 is a block diagram showing a configuration example of the control unit 502 shown in FIG. 5. The control unit 502 includes a preprocessing unit 1210, a first generation unit 701, an acquisition unit 1220, a compression / decompression unit 702, a specifying unit 1240, a synthesizing unit 703, and a reproduction unit 704. The control unit 502 is composed of a processor 1201, a storage device 1202, an integrated circuit 1203, and a bus 1204 connecting these. Note that the storage device 1202, the decompression unit 1234, the specifying unit 1240, the synthesizing unit 703, and the reproduction unit 704 may be mounted on another device accessible to the electronic device 500.

[0144] The preprocessing unit 1210, the first generation unit 701, the acquisition unit 1220, the compression / decompression unit 702, the specification unit 1240, the synthesis unit 703, and the playback unit 704 may be realized by causing the processor 1201 to execute a program stored in the storage device 1202, or may be realized by an integrated circuit 1203 such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). Further, the processor 1201 may use the storage device 1202 as a work area. Also, the integrated circuit 1203 may use the storage device 1202 as a buffer for temporarily holding various data including image data.

[0145] Note that a device including at least the compression unit 1231 in the compression / decompression unit 702 becomes a video compression device. A device including at least the second generation unit 1232 in the compression / decompression unit 702 becomes a generation device. Also, a device including at least the decompression unit 1234 in the compression / decompression unit 702 becomes a decompression device. Also, a device including at least the playback unit 704 becomes a playback device.

[0146] The preprocessing unit 1210 executes preprocessing for generating a video file for the input video data 710 from the imaging device 100. Specifically, for example, the preprocessing unit 1210 includes a detection unit 1211 and a setting unit 1212. The detection unit 1211 detects a specific subject by the well-known subject detection technique described above.

[0147] The setting unit 1212 changes the imaging area in which the specific subject is detected on the imaging surface 200 of the imaging device 100 from the first frame rate (for example, 30 [fps]) to the second frame rate (for example, 60 [fps]).

[0148] Specifically, for example, the setting unit 1212 detects a motion vector of a specific subject from the difference between the imaging area where the specific subject is detected in the input frame and the imaging area where the specific subject is detected in the input frame that has already been processed, and predicts the imaging area of the specific subject in the next input frame. The setting unit 1212 changes the predicted imaging area to the second frame rate. The setting unit 1212 adds information indicating the image area at the first frame rate (for example, 30 [fps]) and the image area at the second frame rate (for example, 60 [fps]) to the frame F.

[0149] The first generation unit 701 complements a missing area 712x that was output during imaging at the second frame rate in the image data 712, which is an image area at the second frame rate in which a specific subject is imaged, with a specific color to obtain a complemented area 712y. Specifically, for example, in frames F2-60 and F4-60 of FIG. 10, the image area other than the second image area a2, which is the specific subject image output during imaging at 60 [fps] (corresponding to the background), is the complemented area 712y.

[0150] Also, in frames F7-60 and F9-60 of FIG. 11, the image area other than the second image area a2, which is the changing scenery imaged at 60 [fps] (corresponding to the preceding vehicle), is the complemented area 712y. The first generation unit 701 sets the missing area 712x to a specific color and eliminates the missing area 712x.

[0151] In this way, the image data of the complemented area 712y, which is a specific color, is data that is not based on the output from the second imaging area and is configured as predetermined data that is independent of the output data from the second imaging area.

[0152] The acquisition unit 1220 acquires the input video data 710 output from the preprocessing unit 1210, or the first video data 721 and the second video data 722, stores them in the storage device 1202, and outputs a plurality of frames to the compression / decompression unit 702 one by one in time series order at a predetermined timing. Specifically, for example, when no specific subject is detected, the acquisition unit 1220 acquires the input video data 710 from the preprocessing unit, and when a specific subject is detected, it acquires the first video data 721 and the second video data 722.

[0153] The compression / decompression unit 702 includes a compression unit 1231, a second generation unit 1232, a selection unit 1233, a decompression unit 1234, and a storage unit 1235. The compression unit 1231 compresses the video data from the acquisition unit 1220. Specifically, for example, when the compression unit 1231 acquires video data in which no specific subject is detected, since each frame is the first image area a1, it performs a compression process that does not require motion detection or motion compensation.

[0154] Also, when the compression unit 1231 acquires the first video data 721 and the second video data 722, it compresses the first video data 721 and the second video data 722 respectively. Specifically, for example, in the case of the first video data 721, for the image data in the first image area a1, it performs a compression process that does not require motion detection or motion compensation, and for the image data in the second image area a2 where a specific subject is imaged, it compresses it by the above-described hybrid encoding. In this way, for areas other than the specific subject image, since motion detection and motion compensation are not performed, the processing load of video compression is reduced.

[0155] Also, in the case of the second video data 722, the compression unit 1231 performs a compression process that does not require motion detection or motion compensation on the image data of the complementary region 712y (black-filled), and compresses the image data of the second image region a2 in which a specific subject is imaged by the above-described hybrid encoding. In this way, for the complementary region 712y other than the specific subject image, since motion detection and motion compensation are not performed, the processing load of video compression is reduced. Also, since the complementary region 712y exists, the second frame 713 can be applied with the normal video compression process in the same manner as the first frame 711.

[0156] In this way, the second frame 713 obtained at the second frame rate (60 [fps]) has the same size as the first frame 711 obtained at the first frame rate (30 [fps]). Therefore, since the same compression process as the compression process of the first frame 711 is applied to the second frame 713, there is no need to apply another compression process that matches the size of the image data 712. That is, the compression unit 1231 can apply the compression process applied in the first frame 711 to the second frame 713 as it is. For this reason, there is no need to implement another compression process for the image data 712.

[0157] The second generation unit 1232 generates a video file 800 including the video data (compressed data) compressed by the compression unit 1231. Specifically, for example, the second generation unit 1232 generates the video file 800 according to the file format shown in FIG. 8. The storage unit 1235 stores the generated video file 800 in the storage device 1202.

[0158] Note that the compression unit 1231 may store the compressed data in a buffer memory, and the second generation unit 1232 may read out the compressed data stored in the buffer memory and generate the video file 800.

[0159] The selection unit 1233 receives a playback instruction for the video file 800 from the operation unit 505, reads out the video file 800 to be decompressed from the storage device 1202, and passes it to the decompression unit 1234. The decompression unit 1234 decompresses the video file 800 passed from the selection unit 1233 according to the file format.

[0160] That is, the decompression unit 1234 performs general-purpose decompression processing. Specifically, for example, the decompression unit 1234 performs variable-length decoding processing, inverse quantization, and inverse transformation on the input compressed frame, and applies intra-frame prediction or inter-frame prediction to decompress the compressed frame into the original frame.

[0161] The video file 800 includes a video file 800 in which video data in which no specific subject is detected is compressed, and a video file 800 in which the first video data 721 and the second video data 722 are compressed. The former video file 800 is, in this example, video data obtained by imaging at a frame rate of 30 [fps], for example, fixed-point imaging of only a landscape where a train is not passing. Therefore, when the selection unit 1233 receives a selection of a playback instruction for the video file 800, the decompression unit 1234 decompresses the video file 800 according to the file format.

[0162] On the other hand, the video file 800 in which the first video data 721 and the second video data 722 are compressed includes the compressed video data of the first video data 721 and the second video data 722. Therefore, when the selection unit 1233 receives a selection of a playback instruction for the video file 800 in which the first video data 721 and the second video data 722 are compressed, it specifies the frame rate (for example, 30 [fps], 60 [fps]) selected by the playback instruction.

[0163] When the selected frame rate is 30 [fps], the selection unit 1233 passes, as the compressed data of the first video data 721, the chunk group existing between SOM850a and EOM854a in the mdat820 of the video file 800 to the decompression unit 1234. Thereby, the decompression unit 1234 can decompress the compressed data of the first video data 721 into the first video data 721.

[0164] When the selected frame rate is 60 [fps], the selection unit 1233 passes, as the compressed data of the first video data 721, the chunk group existing between SOM850a and EOM854a in the mdat820 of the video file 800 to the decompression unit 1234, and also passes, as the compressed data of the second video data 722, the chunk group existing between SOM850b and EOM854b in the mdat820 of the video file 800 to the decompression unit 1234. Thereby, the decompression unit 1234 can decompress the compressed data of the first video data 721 into the first video data 721 and decompress the compressed data of the second video data 722 into the second video data 722.

[0165] Note that, as described above, when there are two pieces of compressed data to be decompressed, the decompression unit 1234 may decompress them in the order of the compressed data of the first video data 721 and the compressed data of the second video data 722 (the reverse order is also possible), or may decompress the compressed data of the first video data 721 and the compressed data of the second video data 722 in parallel.

[0166] When the first video data 721 and the second video data 722 are decompressed by the decompression unit 1234, the specifying unit 1240 specifies a difference region based on the first frame 711 in the first video data 721 (for example, frame F1 in FIG. 10) and the second frame 713 in the second video data 722 (for example, frame F2-60 in FIG. 10).

[0167] The differential region is a region indicating the difference between a second image region a2 corresponding to the second imaging region in the first frame 711 and a second image region a2 corresponding to the second imaging region in the second frame 713. The differential region between frame F1 and frame F2-60 is the region Da1 of the white dotted rectangle behind the train in frame F2-60. The differential region between frame F3 and frame F4-60 is the region Da3 of the white dotted rectangle behind the train in frame F4-60.

[0168] As shown in FIGS. 7 to 11, the synthesizing unit 703 replicates and synthesizes the first frame 711 (for example, frame F1 in FIG. 10) including the image data of the first image region a1 one frame before in time into the second frame 713 (for example, frame F2-60 in FIG. 10) to generate a third frame 730 (for example, frame F2 in FIG. 10). Note that the synthesizing unit 703 may replicate the image data (the end part of the train) of the second image region a2 at the same position as the differential region in the first frame 711 in the differential regions (Da1, Da3) specified by the specifying unit 1240. Thereby, the difference between the temporally continuous first frame 711 and the third frame 730 can be made substantially zero. Therefore, a video without a sense of incongruity can be played.

[0169] Note that, in the specifying unit 1240 and the synthesizing unit 703, the insertion position into the first moving image data 721 of frame F2-60 is specified by the insertion position information 920 of the additional information 835. For example, assuming that the frame numbers of frames F1 and F3 are #4a and #5a respectively, and the frame number of frame F2-60 is #2b, the insertion position 922 of the value #2b of the insertion frame number 921 is (#4a, #5a). Therefore, the insertion position of frame F2-60 is specified to be between frames F1 and F3.

[0170] <Configuration Example of Compression Unit 1231> FIG. 13 is a block diagram showing a configuration example of the compression unit 1231. As described above, the compression unit 1231 compresses each frame F from the acquisition unit 1220 by hybrid encoding that combines entropy encoding with motion compensation inter-frame prediction (MC) and discrete cosine transform (DCT).

[0171] The compression unit 1231 includes a subtraction unit 1301, a DCT unit 1302, a quantization unit 1303, an entropy encoding unit 1304, a coding amount control unit 1305, an inverse quantization unit 1306, an inverse DCT unit 1307, a generation unit 1308, a frame memory 1309, a motion detection unit 1310, a motion compensation unit 1311, and a compression control unit 1312. The subtraction unit 1301 to the motion compensation unit 1311 have the same configuration as an existing compressor.

[0172] Specifically, for example, the subtraction unit 1301 subtracts a predicted frame from the motion compensation unit 1311 that predicts the input frame from the input frame and outputs difference data. The DCT unit 1302 performs a discrete cosine transform on the difference data from the subtraction unit 1301.

[0173] The quantization unit 1303 quantizes the discrete cosine transform difference data. The entropy encoding unit 1304 entropy-encodes the quantized difference data and also entropy-encodes the motion vector from the motion detection unit 1310.

[0174] The coding amount control unit 1305 controls the quantization by the quantization unit 1303. The inverse quantization unit 1306 inverse-quantizes the difference data quantized by the quantization unit 1303 to obtain discrete cosine transform difference data. The inverse DCT unit 1307 performs an inverse discrete cosine transform on the inverse-quantized difference data.

[0175] The generation unit 1308 adds the inverse discrete cosine transformed differential data and the predicted frame from the motion compensation unit 1311 to generate a reference frame that a frame input later in time than the input frame refers to. The frame memory 1309 holds the reference frame obtained from the generation unit 1308. The motion detection unit 1310 detects a motion vector using the input frame and the reference frame. The motion compensation unit 1311 generates a predicted frame using the reference frame and the motion vector.

[0176] Specifically, for example, the motion compensation unit 1311 performs motion compensation on the frame output by imaging at the second frame rate using a specific reference frame among a plurality of reference frames held in the frame memory 1309 and the motion vector. By setting a specific reference frame as the reference frame, it is possible to suppress high-load motion compensation using other reference frames other than the specific reference frame. Further, by setting the specific reference frame as a single reference frame obtained from the frame one time before the input frame in time, high-load motion compensation can be avoided and the processing load of motion compensation can be reduced.

[0177] The compression control unit 1312 controls the motion detection unit 1310 and the motion compensation unit 1311. Specifically, for example, the compression control unit 1312 executes a first compression control method for setting a specific motion vector indicating no motion in the motion detection unit 1310 and a second compression control method for skipping the motion detection itself.

[0178] The first compression control method will be described. In the case of the first video data 721, the compression control unit 1312 controls the motion detection unit 1310, and for the first image area a1 output by imaging at the first frame rate (for example, 30 [fps]), instead of detecting the motion vector, a specific motion vector indicating no motion is set and output to the motion compensation unit 1311. Also, the compression control unit 1312 controls the motion detection unit 1310, and for the second image area a2 output by imaging at the second frame rate (for example, 60 [fps]), the motion vector is detected and output to the motion compensation unit 1311. The specific motion vector is a motion vector whose direction is not defined and whose motion amount is 0. Thus, for the first image area a1 output by imaging at the first frame rate (for example, 30 [fps]), the motion vector is not detected.

[0179] In this case, the compression control unit 1312 controls the motion compensation unit 1311 to perform motion compensation on the image data of the first image area a1 based on the specific motion vector and the reference frame. For the image data of the second image area a2, the compression control unit 1312 performs motion compensation based on the motion vector detected by the motion detection unit 1310. In the case of the second video data 722, the first image area a1 output by imaging at the first frame rate (for example, 30 [fps]) may be replaced with an area filled with a specific color.

[0180] The second compression control method will be described. In the case of the first video data 721, the compression control unit 1312 controls the motion detection unit 1310 not to detect the motion vector for the image data of the complementary area 712y. Also, the compression control unit 1312 controls the motion detection unit 1310 to detect the motion vector for the second image area a2 output by imaging at the second frame rate (for example, 60 [fps]).

[0181] In this case, the compression control unit 1312 controls the motion compensation unit 1311 to perform motion compensation on the image data of the first image region a1 based on the reference frame. That is, since there is no motion vector, the compression control unit 1312 controls the motion compensation unit 1311 to determine the reference frame for the image data of the complementary region 712y as a predicted frame that predicts the frame one time later in time from the input frame.

[0182] Also, the compression control unit 1312 controls the motion compensation unit 1311 to perform motion compensation on the image data of the second image region a2 based on the reference frame and the motion vector detected by the motion detection unit 1310. In the case of the second video data 722, the first image region a1 output by imaging at the first frame rate (for example, 30 [fps]) may be replaced with the complementary region 712y.

[0183] According to the first compression control method, since the motion vector is a specific motion vector, motion detection in the first image region a1 and the complementary region 712y is simplified. Therefore, the processing load of video compression is reduced. Also, according to the second compression control method, since motion detection itself is not performed for the first image region a1 and the complementary region 712y, the processing load of video compression is reduced more than the first compression control method.

[0184] <Example of the operation processing procedure of the control unit 502> FIG. 14 is a sequence diagram showing an example of the operation processing procedure of the control unit 502. In FIG. 14, for convenience of explanation, the acquisition unit 1220 is omitted. The preprocessing unit 1210 sets the imaging conditions for the entire imaging surface 200 of the imaging device 100 to the first frame rate (for example, 30 [fps]) automatically when, for example, the user operates the operation unit 505 or when a specific subject is not detected in step S1412 (step S1412: Yes) (step S1401).

[0185] As a result, in the imaging device 100, the imaging conditions for the entire imaging surface 200 are set to the first frame rate (step S1402). The imaging device 100 images the subject at the first frame rate and outputs the input video data 710 to the preprocessing unit 1210 (step S1403).

[0186] When the input video data 710 is input to the preprocessing unit 1210 (step S1403), the preprocessing unit 1210 executes a setting process (step S1404). The setting process (step S1404) sets the frame rate for each frame of the input video data 710. For example, an image area with the first frame rate (e.g., 30 [fps]) added is recognized as the first image area a1, and an image area with the second frame rate (e.g., 60 [fps]) added is recognized as the second image area a2.

[0187] Also, the preprocessing unit 1210 outputs the input video data 710 to the first generation unit 701 (step S1405). Further, when the preprocessing unit 1210 does not detect an image area with the second frame rate of the next input frame in the setting process (step S1404) (step S1406: No), it waits for the input of the input video data 710 in step S1403. On the other hand, when the preprocessing unit 1210 detects an image area with the second frame rate of the next input frame in the setting process (step S1404) (step S1406: Yes), it changes the setting of the second image area a2 including the specific subject to the second frame rate (e.g., 60 [fps]) (step S1407).

[0188] Then, according to the setting change content in step S1407, the imaging conditions for the second imaging area in the entire imaging surface 200 are set to the second frame rate. As a result, the imaging device 100 images the subject at the first frame rate in the first imaging area and at the second frame rate in the second imaging area, and outputs the input video data 710 to the preprocessing unit 1210 (step S1409).

[0189] When the input video data 710 is input (step S1409), the preprocessing unit 1210 executes setting processing (step S1410). The setting processing (step S1410) is the same processing as the setting processing (step S1404). Details of the setting processing (step S1410) will be described later with reference to FIG. 15. The preprocessing unit 1210 outputs the input video data 710 to the first generation unit 701 (step S1411).

[0190] When the specific subject becomes undetected (step S1412: Yes), the preprocessing unit 1210 returns to step S1401 and changes the setting of the entire imaging surface 200 to the first frame rate (step S1401). On the other hand, when the specific subject continues to be detected (step S1412: No), the preprocessing unit 1210 returns to step S1407 and changes the second image area a2 corresponding to the detection position of the specific subject to the second frame rate (step S1407). In this case, for the image area where the specific subject is no longer detected, the preprocessing unit 1210 changes the setting to the first frame rate.

[0191] When the input video data 710 is input (step S1405), the first generation unit 701 executes complement processing (step S1413). In the complement processing (step S1413), the first generation unit 701 specifies that each frame of the input video data 710 is only the first frame 711 with reference to the frame rate of each frame.

[0192] Therefore, since the specific subject is not imaged, the image data 712 does not exist. Therefore, the first generation unit 701 does not complement the image data 712. Details of the complement processing (step S1413) will be described later with reference to FIG. 18. The first generation unit 701 outputs the input video data 710 to the compression unit 1231 (step S1414).

[0193] Also, when the input video data 710 is input (step S1411), the first generation unit 701 executes a complement processing (step S1415). In the complement processing (step S1415), the first generation unit 701 refers to the frame rate of each frame and determines that each frame of the input video data 710 includes the first frame 711 and the image data 712.

[0194] Therefore, since the specific subject is imaged in the first frame 711 and the image data 712, the first generation unit 701 generates the second frame 713. Details of the complement processing (step S1415) will be described later with reference to FIG. 18. The first generation unit 701 outputs the first frame 711 and the second frame 713 to the compression unit 1231 (step S1416).

[0195] When the input video data 710 is input (step S1414), the compression unit 1231 and the second generation unit 1232 execute a video file generation process for the input video data 710 (step S1417). Since the input video data 710 is composed of only the first frame 711, the compression unit 1231 executes compression encoding that does not require motion detection or motion compensation in the compression process (step S1417). Details of the video file generation process (step S1417) will be described later with reference to FIGS. 18 to 24.

[0196] Also, when the first video data 721 and the second video data 722 are input (step S1416), the compression unit 1231 and the second generation unit 1232 execute a video file generation process for the first video data 721 and the second video data 722 (step S1418). The first video data 721 is composed of the first frame 711, and the second video data 722 is composed of the second frame 713.

[0197] In the compression unit 1231, in the moving image file generation process (step S1418), when the object to be compressed is the first moving image data 721, for the image data of the first image area a1, a compression process that does not require motion detection or motion compensation is executed, and for the image data of the second image area a2 in which a specific subject is imaged, compression is performed by the above-described hybrid encoding. In this way, for areas other than the specific subject image, motion detection and motion compensation are not executed, so that the processing load of moving image compression is reduced.

[0198] Also, when the object to be compressed is the second moving image data 722, the compression unit 1231 also executes a compression process that does not require motion detection or motion compensation for the image data of the complementary area 712y (black filling), and for the image data of the second image area a2 in which a specific subject is imaged, compression is performed by the above-described hybrid encoding. In this way, for areas other than the specific subject image, motion detection and motion compensation are not executed, so that the processing load of moving image compression is reduced. Details of the moving image file generation process (step S1418) will be described later with reference to FIGS. 18 to 24.

[0199] <Setting process (steps S1404, S1410)> FIG. 15 is a flowchart showing a detailed processing procedure example of the setting process (steps S1404, S1410) shown in FIG. 14. In FIG. 15, a first frame rate (e.g., 30 [[fps]]) is set in advance in the imaging device 100, and an image area of a second frame rate (e.g., 60 [[fps]]) is tracked by the subject detection technique of the detection unit 1211 and fed back to the imaging device 100. Note that the image areas of the first frame rate and the second frame rate may be fixed at all times.

[0200] The preprocessing unit 1210 waits for the input of a frame constituting the input moving image data 710 (step S1501), and when a frame is input (step S1501: Yes), determines whether a specific subject such as a main subject is detected by the detection unit 1211 (step S1502). If no specific subject is detected (step S1502: No), the process proceeds to step S1504.

[0201] On the other hand, when a specific subject is detected (step S1502: Yes), the preprocessing unit 1210 causes the detection unit 1211 to compare the frame one frame before in time (for example, the reference frame) with the input frame to detect a motion vector, predict an image area at the second frame rate in the next input frame, output it to the imaging device 100, and proceed to step S1504 (step S1503). As a result, the imaging device 100 can set the imaging conditions of the block 202 constituting the imaging area corresponding to the predicted image area to the second frame rate and set the imaging conditions of the remaining blocks 202 to the first frame rate to image the subject.

[0202] Then, the preprocessing unit 1210 executes frame rate setting processing on the input frame (step S1504) and returns to step S1501. The frame rate setting processing (step S1505) is the processing for setting the above-described frame rate to the frame F, which will be described in detail with reference to FIG. 16.

[0203] When there is no input of the frame F (step S1501: No), since the input of the input video data 710 has ended, the preprocessing unit 1210 ends the setting processing (steps S1404, S1410).

[0204] <Frame rate setting processing (step S1504)> FIG. 16 is a flowchart showing a detailed processing procedure example of the frame rate setting processing (step S1504) shown in FIG. 15. When a frame is input (step S1601), the preprocessing unit 1210 determines whether there is an unselected image area in the input frame (step S1602). When there is an unselected image area (step S1602: Yes), the preprocessing unit 1210 selects one unselected image area (step S1603) and determines whether the detection flag of the specific subject is ON (step S1604). The detection flag is information indicating the presence or absence of detection of the specific subject, and the default is OFF (non-detection).

[0205] When a specific subject is detected in step S1406 of FIG. 14 (step S1406: Yes), the preprocessing unit 1210 changes the detection flag from OFF to ON (detection in progress). When the specific subject is not detected in step S1412 (step S1412: Yes), the preprocessing unit 1210 changes the detection flag from ON to OFF.

[0206] When the detection flag is OFF (step S1604: No), information indicating the first frame rate is set in the input frame for the selected image area (step S1605), and the process returns to step S1602. On the other hand, when the detection flag is ON (step S1604: Yes), the preprocessing unit 1210 determines whether the selected image area is an image area where a specific subject image exists (step S1606).

[0207] When no specific subject image exists (step S1606: No), the process returns to step S1602. On the other hand, when a specific subject image exists (step S1606: Yes), the preprocessing unit 1210 sets information indicating the second frame rate in the input frame for the selected image area (step S1607), and the process returns to step S1602.

[0208] In step S1602, when there is no unselected image area (step S1602: No), the preprocessing unit 1210 ends the frame rate setting process. After that, the preprocessing unit 1210 sets the frame rate in the imaging device 100 (steps S1401, S1407).

[0209] By setting information indicating the frame rate for each image area of each frame, the preprocessing unit 1210 can identify which frame rate should be set for the imaging area of the imaging device 100 corresponding to which image area. Also, the first generation unit 701 and the compression unit 1231 can identify the frame rate of each image area of the input frame F.

[0210] <Complementary processing (steps S1413, S1415)> FIG. 17 is a flowchart showing an example of the complement processing procedure by the first generation unit 701. When the first generation unit 701 receives the input of the frame F (step S1701), it refers to the frame rate of the input frame (step S1702). If it is not only the second frame rate (60 [fps]) (step S1703: No), the first generation unit 701 ends without executing the complement processing. If it is only the second frame rate (60 [fps]) (step S1703: Yes), since the input frame is the image data 712, the first generation unit 701 executes the complement processing to make the input frame into the second frame 713 (step S1704). Thereby, the frames F2-60, F4-60 shown in FIG. 10, and the frames F7-60, F9-60 shown in FIG. 11 can be generated.

[0211] <Video file generation process (steps S1417, S1418)> FIG. 18 is a flowchart showing a detailed processing procedure example of the video file generation process (steps S1417, S1418) shown in FIG. 14. The compression unit 1231 will separately compress the first video data 721 composed of the first frame 711 and the second video data 722 composed of the second frame 713. When the compression unit 1231 receives the input of the frame F (step S1801), it performs compression encoding on the input frame (step S1802). The detailed control content of the compression encoding will be described later with reference to FIGS. 19 to 24.

[0212] After that, the second generation unit 1232 generates metadata such as uuid831, udta832, mvhd833, trak834 shown in FIG. 8 from the compression-encoded data (step S1803). Note that the second generation unit 1232 may execute step S1803 before the compression encoding (step S1802) for the metadata that requires information before compression.

[0213] The second generation unit 1232 refers to the information indicating the frame rate given to the frame F, generates imaging condition information 910 (step S1804), refers to the position information of the chunks (stsz847 and stco848), specifies the insertion destination of the second frame 713, and generates insertion position information (step S1805). Additional information 835 is generated by steps S1804 and S1805. Then, the second generation unit 1232 combines the header part 801 and the data part 802 to generate a video file 800 (step S1806) and stores it in the storage device 1202 (step S1807).

[0214] <Example of Compression Processing: First Compression Control Method> Next, the compression encoding by the compression unit 1231 shown in FIG. 18 (step S1802) will be described separately for the first compression control method and the second compression control method.

[0215] FIG. 19 is a flowchart showing an example of the compression control processing procedure of the first compression control method by the compression control unit 1312. The compression control unit 1312 acquires an input frame (the first frame 711 or the second frame 713) (step S1901) and selects an unselected image area from the acquired input frame (step S1902). Then, the compression control unit 1312 refers to the frame rate of the selected image area from the input frame (step S1903).

[0216] When the input frame is the first frame 711, the selected image area is the first image area a1 output by imaging at the first frame rate or the second image area a2 output by imaging at the second frame rate. When the input frame is the second frame 713, the selected image area is the complementary area 712y corresponding to the first image area a1 output by imaging at the first frame rate or the second image area a2 output by imaging at the second frame rate.

[0217] When the frame rate of the selected image area is the second frame rate (step S1903: second FR), the compression control unit 1312 outputs the image data of the selected image area to the motion detection unit 1310 (step S1904). Thereby, the motion detection unit 1310 detects a motion vector using a reference frame as usual for the selected image area with the second frame rate.

[0218] On the other hand, when the frame rate of the selected image area is the first frame rate (step S1903: first FR), the compression control unit 1312 sets a skip flag for the selected image area with the first frame rate and outputs it to the motion detection unit 1310 (step S1905). Thereby, the motion detection unit 1310 sets a specific motion vector indicating no motion for the selected image area with the first frame rate.

[0219] After step S1904 or S1905, the compression control unit 1312 determines whether there is an unselected image area in the acquired input frame (step S1906). If there is an unselected image area (step S1906: Yes), the process returns to step S1902. On the other hand, if there is no unselected image area (step S1906: No), the compression control unit 1312 ends the series of processes.

[0220] FIG. 20 is a flowchart showing an example of a motion detection processing procedure of the first compression control method by the motion detection unit 1310. The motion detection unit 1310 acquires a reference frame that is one frame earlier in time than the input frame from the frame memory 1309 (step S2001) and waits for the input of the selected image area output in step S1904 or S1905 of FIG. 19 (step S2002: No).

[0221] When a selected image area is input (step S2002: Yes), the motion detection unit 1310 acquires the image data of the image area at the same location as the selected image area from the reference frame (step S2003). Then, the motion detection unit 1310 determines whether there is a skip flag in the selected image area (step S2004). If there is no skip flag (step S2004: No), the frame rate of the selected image area is the second frame rate. Therefore, the motion detection unit 1310 detects a motion vector using the image data of the selected image area and the image data of the image area of the reference frame acquired in step S2003 (step S2005).

[0222] On the other hand, if there is a skip flag (step S2004: Yes), the motion detection unit 1310 sets a specific motion vector indicating no motion (step S2006). As a result, in the motion detection process by the motion detection unit 1310, since a specific motion vector indicating no motion is always used, the processing load of motion detection is reduced for the selected image area at the first frame rate. Then, the motion detection unit 1310 outputs the motion vector obtained in step S2005 or S2006 to the motion compensation unit 1311 (step S2007), and ends a series of processes.

[0223] FIG. 21 is a flowchart showing an example of a motion compensation processing procedure of the first compression control method by the motion compensation unit 1311. The motion compensation unit 1311 acquires a reference frame from the frame memory 1309 (step S2101). The motion compensation unit 1311 acquires the image area at the same location as the selected image area from the reference frame (step S2102).

[0224] Then, the motion compensation unit 1311 performs motion compensation using the motion vector for the selected image area from the motion detection unit 1310 and the image area of the reference frame acquired in step S2102 (step S2103). Thereby, the motion compensation unit 1311 can generate predicted image data in the selected image area.

[0225] Then, the motion compensation unit 1311 determines whether the motion compensation for all selected image regions has been completed (step S2104). Specifically, for example, when the compression control unit 1312 determines in step S1906 that there is an unselected image region (step S1906: Yes), the motion compensation unit 1311 determines that the motion compensation for all selected image regions has not been completed (step S2104: No), and returns to step S2102.

[0226] On the other hand, when the compression control unit 1312 determines in step S1906 that there is no unselected image region (step S1906: No), the motion compensation unit 1311 determines that the motion compensation for all selected image regions has been completed (step S2104: Yes). Then, the motion compensation unit 1311 outputs a predicted frame obtained by combining predicted image data for all selected image regions to the subtraction unit 1301 and the generation unit 1308 (step S2105), and ends a series of processes.

[0227] <Compression processing example: Second compression control method> FIG. 22 is a flowchart showing an example of a compression control processing procedure of the second compression control method by the compression control unit 1312. The compression control unit 1312 acquires an input frame (step S2201), and selects an unselected image region from the acquired input frame (step S2202). Then, the compression control unit 1312 refers to the frame rate of the input frame for the selected image region (step S2203).

[0228] When the frame rate of the selected image region is the second frame rate (step S2203: Second FR), the compression control unit 1312 outputs the selected image region to the motion detection unit 1310 (step S2204). Thereby, the motion detection unit 1310 detects a motion vector using a reference frame for the selected image region with the second frame rate as usual.

[0229] On the one hand, when the frame rate of the selected image region is the first frame rate (step S2203: first FR), the compression control unit 1312 sets a skip flag for the selected image region at the first frame rate and outputs it to the motion detection unit 1310 (step S2205). As a result, the motion detection unit 1310 will not perform motion detection on the selected image region at the first frame rate. Then, the compression control unit 1312 issues a motion compensation stop instruction for the selected image region and outputs it to the motion compensation unit 1311 (step S2206). Thereby, the execution of motion compensation for the selected image region can be stopped.

[0230] After step S2204 or S2206, the compression control unit 1312 determines whether there is an unselected image region in the acquired input frame (step S2207). If there is an unselected image region (step S2207: Yes), the process returns to step S2202. On the other hand, if there is no unselected image region (step S2207: No), the compression control unit 1312 ends a series of processes.

[0231] FIG. 23 is a flowchart showing an example of a motion detection processing procedure of the second compression control method by the motion detection unit 1310. The motion detection unit 1310 acquires a reference frame that is one time earlier than the input frame F from the frame memory 1309 (step S2301) and waits for the input of the selected image region output in step S2204 or S2205 of FIG. 22 (step S2302: No).

[0232] When a selected image area is input (step S2302: Yes), the motion detection unit 1310 acquires the image data of the image area at the same location as the selected image area from the reference frame (step S2303). Then, the motion detection unit 1310 determines whether there is a skip flag in the selected image area (step S2304). When there is no skip flag (step S2304: No), the frame rate of the selected image area is the second frame rate. Therefore, the motion detection unit 1310 uses the image data of the selected image area and the image data of the image area of the reference frame acquired in step S2003 to detect a motion vector (step S2305).

[0233] Then, the motion detection unit 1310 outputs the motion vector obtained in step S2305 to the motion compensation unit 1311 (step S2306) and ends a series of processes. On the other hand, when there is a skip flag (step S2304: Yes), the motion detection unit 1310 ends a series of processes without performing motion detection.

[0234] FIG. 24 is a flowchart showing an example of a motion compensation processing procedure of the second compression control method by the motion compensation unit 1311. The motion compensation unit 1311 acquires a reference frame from the frame memory 1309 (step S2401). The motion compensation unit 1311 acquires an image area at the same location as the selected image area from the reference frame (step S2402).

[0235] Then, the motion compensation unit 1311 determines whether the trigger input for motion compensation for the selected image area is either a motion vector or a motion compensation stop instruction (step S2403). When the trigger input is a motion vector (step S2403: motion vector), the motion compensation unit 1311 performs motion compensation using the motion vector for the selected image area from the motion detection unit 1310 and the image area of the reference frame acquired in step S2402 (step S2404). Thereby, the motion compensation unit 1311 can generate predicted image data in the selected image area.

[0236] On the other hand, when the trigger input is a motion compensation stop instruction (step S2403: motion compensation stop instruction), the motion compensation unit 1311 determines the image data of the acquired image area as the image data of the predicted image area (predicted image data) (step S2405).

[0237] Then, after step S2404 or S2405, the motion compensation unit 1311 determines whether the motion compensation of all selected image areas has been completed (step S2406). Specifically, for example, when the compression control unit 1312 determines in step S2207 that there is an unselected image area (step S2207: Yes), the motion compensation unit 1311 determines that the motion compensation of all selected image areas has not been completed (step S2406: No), and returns to step S2402.

[0238] On the other hand, when the compression control unit 1312 determines in step S2207 that there is no unselected image area (step S2207: No), the motion compensation unit 1311 determines that the motion compensation of all selected image areas has been completed (step S2406: Yes). Then, the motion compensation unit 1311 outputs a predicted frame obtained by combining the predicted image data for all selected image areas to the subtraction unit 1301 and the generation unit 1308 (step S2407), and ends a series of processes.

[0239] <Process from decompression to playback> FIG. 25 is a flowchart showing an example of a processing procedure from decompression to playback. The selection unit 1233 waits for a selection of a playback instruction from the operation unit 505 (step S2501: No). When a selection of a playback instruction is made (step S2501: Yes), the selection unit 1233 determines whether the frame rate of the video file 800 to be played back is selectable (step S2502). If it is not selectable (step S2502: No), the video file 800 is a video file 800 in which only a frame group at the first frame rate (30 [fps]) is compressed. In this case, the decompression unit 1234 decompresses the video file 800 (step S2504), and proceeds to step S2508.

[0240] On the other hand, if it is selectable in step S2502 (step S2502: Yes), the selection unit 1233 determines whether the selected frame rate is the first frame rate (30 [fps]) (step S2503). If the first frame rate (30 [fps]) is selected (step S2503: Yes), the video file 800 to be played back is the video file 800 obtained by compressing the first video data 721. Therefore, the decompression unit 1234 decompresses the video file 800 (step S2504) and proceeds to step S2508.

[0241] On the other hand, if the second frame rate (60 [fps]) is selected (step S2503: No), the video file 800 to be played back is the video file 800 obtained by compressing the first video data 721 and the second video data 722. Therefore, the decompression unit 1234 decompresses the video file 800 and outputs the first video data 721 and the second video data 722 (step S2505).

[0242] Then, the specifying unit 1240 specifies the difference region by referring to the first video data 721 and the second video data 722 decompressed in step S2505 (step S2506). After that, as shown in FIGS. 10 and 11, the combining unit 703 performs a combining process on the first video data 721 and the second video data 722 (step S2507). Details of the combining process (step S2507) will be described later with reference to FIG. 26. Finally, the playback unit 704 plays back the video data obtained in the combining process (step S2507) or step S2504 on the liquid crystal monitor (step S2508).

[0243] <Combining process (step S2507)> FIG. 26 is a flowchart showing a detailed processing procedure example of the synthesis process (step S2507) shown in FIG. 25. The synthesis unit 703 sets the output order of the frames F according to the insertion position information 920 (step S2601). Next, the synthesis unit 703 determines whether there is a remaining frame that has not been output to the playback unit 704 (step S2602). If there is a remaining frame (step S2602: Yes), the synthesis unit 703 acquires the frames in the output order (step S2603).

[0244] For example, the synthesis unit 703 refers to the frame type identification information written in uuid831 and determines whether the acquired frame is the second frame 713 (step S2604). If it is not the second frame 713 (step S2604: No), since the acquired frame is the first frame 711, the synthesis unit 703 outputs the acquired frame as a playback target to the playback unit 704 and writes it to the buffer (step S2605). Then, it returns to step S2602.

[0245] On the other hand, in step S2604, if the acquired frame is the second frame 713 (step S2604: Yes), the synthesis unit 703 synthesizes the frame in the buffer and the acquired frame to generate the third frame 730, and outputs it to the playback unit 704 as a playback target (step S2606). Then, it returns to step S2602. In step S2602, if there is no remaining frame (step S2602: No), the synthesis unit 703 ends the synthesis process (step S2507).

[0246] Thereby, as shown in FIGS. 10 and 11, the synthesis unit 703 can synthesize the second frame 713 and the first frame 711 one time earlier in time to the third frame 730 including the first image area a1 and the second image area a2. Therefore, the difference in the frame rate within one frame can be absorbed.

[0247] (1-1) In this way, the video compression device generates a plurality of first frames based on the data output from the first imaging area, generates a plurality of second frames based on the data output from the second imaging area, compresses the plurality of first frames 711, and compresses the plurality of second frames 713. Thereby, when compressing video data with different frame rates for each image area, the video data can be compressed separately.

[0248] (1-2) Also, in the above (1-1), the video compression device generates the first frame 711 based on the data output from the first imaging area and the data output from the second imaging area. Thereby, a frame without defects can be generated by the output from a plurality of imaging areas.

[0249] (1-3) Also, in the above (1-1), the video compression device generates the second frame 713 based on the data output from the second imaging area and the data not based on the output from the imaging device 100. Thus, the data not based on the output from the imaging device 100 is not the data from the first imaging area, but for example, the data obtained by image processing for the missing area 712x. Therefore, the second frame 713 can be compressed in the same way as the first frame 711.

[0250] (1-4) Also, in the above (1-3), the video compression device generates the second frame 713 based on the data output from the second imaging area and predetermined data. The predetermined data is, for example, the data obtained by image processing for the missing area 712x. Therefore, the second frame 713 can be compressed in the same way as the first frame 711.

[0251] (1-5) Further, in the above (1-4), the video compression device generates the second frame 713 by complementing the area (defective area 712x) where no data was output from the first imaging area for the data output from the second imaging area. Thereby, the second frame 713 can be compressed in the same manner as the first frame 711 by supplementing the defective area 712x.

[0252] (1-6) Further, in the above (1-5), the video compression device generates the second frame 713 by complementing the area where no data was output from the first imaging area with a specific color for the data output from the second imaging area. Thereby, the compression efficiency can be improved.

[0253] (1-7) Further, in the above (1-3) to (1-6), the video compression device detects a motion vector for the image data of the area generated based on the data output from the second imaging area among the second frames. Thereby, for example, for the image data of the first image area a1 and the complementary area 712y, instead of detecting the motion vector, a specific motion vector is set, so that motion detection is not executed, and the load of the compression process can be reduced.

[0254] (1-8) Further, in the above (1-7), the video compression device does not detect a motion vector for the image data of the area other than the area generated based on the data output from the second imaging area. Thereby, for example, by not performing motion detection for the image data of the first image area a1 and the complementary area 712y, the load of the compression process can be reduced.

[0255] (1-9) Further, in the above (1-7) or (1-8), the video compression device performs motion compensation based on the motion vector detection result. Thereby, the load of the compression process can be reduced.

[0256] Thus, according to the video compression device described above, the first video data 721 composed of the first frame 711 and the second video data 722 composed of the second frame 713 after completion of interpolation can be compressed separately. That is, the input video data 710 in which different frame rates are mixed can be segmented at the imaging timing of the frame rate and compressed.

[0257] Therefore, when it is desired to perform decompression or playback, selections such as the first video data 721 or both the first video data 721 and the second video data 722 can be made as the object of decompression or playback. For example, when it is desired to play back at 30 [fps] which is the imaging timing of the first frame 711, only the first video data 721 may be decompressed and played back.

[0258] As a result, the decompression process for the second video data 722 becomes unnecessary, and it is possible to speed up the decompression process for the playback target and save power. Also, for example, when it is desired to play back at 60 [fps] which is the imaging timing of the image data 712, both the first video data 721 and the second video data 722 may be decompressed and combined. As a result, the reproducibility of the subject video can be enhanced as needed, and it can be played back as a more realistic video.

[0259] (2-1) Further, the generation device includes a generation unit (second generation unit 1232) that generates a video file 800 including first compressed data obtained by compressing a plurality of first frames 711 generated based on data output from a first imaging region in which a first frame rate (for example, 30 [fps]) is set, second compressed data obtained by compressing a plurality of second frames 713 generated based on data output from a second imaging region in which a second frame rate (for example, 60 [fps]) faster than the first frame rate is set, first position information indicating the storage position of the first compressed data, and second position information indicating the storage position of the second compressed data, and a storage unit 1235 that stores the video file 800 generated by the generation unit in the storage device 1202.

[0260] Thus, by compressing the compressed video data of the first frame 711 and the second frame 713 with different imaging timings using a common compression method, they can be combined into one video file 800.

[0261] (2-2) Also, in the generation device of (2-1) above, the first frame 711 may be a frame generated based on the data output from the first imaging area and the data output from the second imaging area.

[0262] Thus, by compressing the compressed data of the first frame 711 captured at the imaging timing of the first frame rate and the compressed data of the second frame 713 captured at the imaging timing of the second frame rate using a common compression method, they can be combined into one video file 800.

[0263] (2-3) Also, in the generation device of (2-1) above, the second frame 713 may be a frame generated based on the data output from the second imaging area and the data not based on the output from the imaging device 100.

[0264] Thus, even if there is an image area (missing area 712x) that is not output at the imaging timing of the second frame rate, by treating the data output from the second imaging area as the second frame 713, it can be compressed using the same compression method as the first frame 711.

[0265] (2-4) Also, in the generation device of (2-3) above, the data not based on the output from the imaging device 100 may be predetermined data. Thus, the second frame 713 can be constituted by data independent of the output from the imaging device 100 and can be compressed using the same compression method as the first frame 711.

[0266] (2-5) Also, in the generation device of (2-4) above, the second frame 713 may be a frame generated by complementing the missing area 712x where data was not output from the first imaging area with respect to the data output from the second imaging area. As a result, since the missing area 712x that was not output at the imaging timing of the second frame rate in the second frame 713 is complemented and set as the complemented area 712y, it can be compressed by the same compression method as the first frame 711.

[0267] (2-6) Also, in the generation device of (2-1) above, the generation unit generates a video file 800 including a data unit 802 and a header unit 801 by setting the first compressed data and the second compressed data in the data unit 802 and setting the first position information and the second position information in the header unit 801. Thereby, the compressed data in the data unit 802 can be read with reference to the header unit 801.

[0268] (2-7) Also, in the generation device of (2-5) above, the generation unit sets, in the header unit 801, the first frame rate information (\"30 [fps]\" in 911) indicating the first frame rate in association with the first position information (Pa in 912), and sets the second frame rate information (\"60 [fps]\" in 911) indicating the second frame rate in association with the first position information (Pa in 912) and the second position information (Pb in 912), thereby generating a video file 800 including the header unit 801 and the data unit 802.

[0269] Thereby, it is possible to read the compressed data of the first video data 721 specified by the first position information associated with the first frame rate information, or the first compressed video data obtained by compressing the first video data 721 specified by the first position information associated with the first frame rate information, and the second compressed video data obtained by compressing the second video data 722 specified by the second position information associated with the second frame rate information.

[0270] Thus, when the first frame rate is selected, the first compressed video data obtained by surely compressing the first video data 721 from the video file 800 can be called. Also, when the second frame rate is selected, the second compressed video data obtained by surely compressing the second video data 722 from the video file 800 can be called. Further, when the first frame rate is selected, it is possible to suppress a call omission of the first compressed video data from the video file 800.

[0271] (2-8) Also, in the generation device of (2-7) above, the second generation unit 1232 generates a video file 800 including the header unit 801 and the data unit 802 by setting information (insertion position information 920) indicating the insertion destination in the first frame 711 where the second frame 713 is inserted in the header unit 801.

[0272] Thus, it is possible to improve the accuracy of the synthesis of the first video data 721 and the second video data 722, enhance the reproducibility of the subject video, and reproduce it as a more realistic video.

[0273] (2-9) Also, in the generation device of (2-3) above, the generation unit may generate a video file 800 for each of the first video data 721 and the second video data 722 and associate the two video files 800. Thus, the video file 800 of the first video data 721 alone can be distributed. Also, when it is desired to reproduce at the second frame rate, the video file 800 of the second video data 722 may be separately acquired.

[0274] In this way, by using separate video files 800 for the first video data 721 and the second video data 722, it is possible to achieve distribution (for example, download) of the video file 800 according to conditions. For example, a terminal of a free user of a video distribution service can only download the video file 800 of the first video data 721, and a terminal of a charged user can download both video files 800.

[0275] (3-1) Further, the playback device reads a video file including first compressed data obtained by compressing a plurality of first frames 711 generated based on data output from a first imaging area with a first frame rate set, and second compressed data obtained by compressing a plurality of second frames 713 generated based on data output from a second imaging area with a second frame rate faster than the first frame rate, and has a decompression unit 1234 that decompresses at least the first compressed data among the first compressed data and the second compressed data, and a playback unit 704 that plays back the plurality of frames decompressed by the decompression unit 1234.

[0276] Therefore, selections such as first video data 721 or both first video data 721 and [the second video data] can be made as the playback target. For example, if you want to play back at 30 [fps], which is the imaging timing of the first frames 711, you only need to play back the plurality of first frames 711.

[0277] This eliminates the need for unnecessary playback processing of the plurality of second frames 713, and power consumption can be reduced. Also, for example, if you want to play back at 60 [fps], which is the imaging timing of the image data 712, you only need to play back both the first video data 721 and the second video data 722. This enhances the reproducibility of the subject video as needed and enables playback as a more realistic video.

[0278] (3-2) Further, in the playback device of the above (3-1), the first frame 711 may be a frame generated based on data output from the first imaging area and data output from the second imaging area.

[0279] Note: There seems to be a missing part in the square brackets in the translation of . It should be something like "first video data 721 or both first video data 721 and second video data" for the sentence to be complete in meaning. But according to the rules, the text as provided is translated as accurately as possible.As a result, since the compressed data of the first frame 711 captured at the imaging timing of the first frame rate and the compressed data of the second frame 713 captured at the imaging timing of the second frame rate are compressed by a common compression method to generate the video file 800, by decompressing the video file 800, it is possible to select a reproduction target such as the first video data 721 or both the first video data 721 and the second video data 722.

[0280] (3-3) Also, in the playback device of (3-1) above, the second frame 713 may be a frame generated based on the data output from the second imaging area and the data not based on the output from the imaging device 100.

[0281] As a result, even if there is an image area (defective area 712x) that is not output at the imaging timing of the second frame rate, by treating the data output from the second imaging area as the second frame 713, since the video file 800 is generated by compressing it with a common compression method as the first frame 711, by decompressing the video file 800, video playback is possible at any frame rate of the first frame rate and the second frame rate.

[0282] (3-4) Also, in the playback device of (3-3) above, the data not based on the output from the imaging device 100 may be predetermined data. As a result, since the second frame 713 and the first frame 711 configured using data unrelated to the output from the imaging device 100 are compressed by a common compression method to generate the video file 800, by decompressing the video file 800, when playing back at the second frame rate, it is possible to play back a combination of both the first video data 721 and the second video data 722.

[0283] (3-5) Also, in the playback device of (3-4) above, the second frame 713 may be a frame generated by complementing the missing area 712x where data was not output from the first imaging area with respect to the data output from the second imaging area. Thereby, when playing back at the second frame rate, both the first video data 721 and the second video data 722 can be combined and played back.

[0284] (3-6) Also, the playback device of (3-1) above includes a selection unit 1233 that selects the frame rate to be played back, and the expansion unit 1234 expands the first compressed data and the second compressed data based on the frame rate selected by the selection unit 1233. Thereby, playback targets such as both the first video data 721 and the second video data 722 can be played back by selecting the frame rate at which you want to play back.

[0285] (3-7) Also, in the playback device of (3-6) above, when the first frame rate is selected by the selection unit 1233, the expansion unit 1234 expands the first compressed data, and when the second frame rate is selected by the selection unit 1233, the expansion unit 1234 expands the first compressed data and the second compressed data. Thereby, the playback target can be changed according to the selected frame rate.

[0286] In this way, selections such as the first compressed video data or both the first compressed video data and the second compressed video data can be made as the expansion target. For example, if you want to play back at 30 [fps], which is the imaging timing of the first frame 711, you may expand only the first compressed video data and play back the first video data 721.

[0287] This eliminates the need for unnecessary decompression processing of the second compressed video data, enabling power saving. Also, for example, when it is desired to play back at 60 [fps], which is the imaging timing of the image data 712, both the first compressed video data and the second compressed video data may be decompressed to play back the first video data 721 and the second video data 722. This enhances the reproducibility of the subject video as needed and enables playback as a more realistic video.

Example

[0288] Example 2 will be described. In Example 1, in the frames F2, F4,... shown in FIG. 10, since the complementary image portions Da1, Da3,... exist, the relevant range was filled with a specific color or a demosaicing process was executed. In Example 2, the composition unit 703 generates frames F2, F4,... with less discomfort without performing such image processing. In Example 2, the same reference numerals are used for the common parts as in Example 1, and the description thereof is omitted.

[0289] <Example of Frame Composition> Here, an example of composing the frame F in Example 2 will be described. In FIG. 10, an example of composing the electronic device 500 while shooting a stationary view of a landscape including a paddy field, mountains, and the sky and shooting a running train as a specific subject was described. Below, the flow of the process of this composition example will be specifically described.

[0290] FIG. 27 is an explanatory diagram showing the specific process flow of the composition example shown in FIG. 10. As also described in FIG. 10, the image sensor 100 outputs frames F1, F2-60, F3,... in time series order. The train is assumed to travel from right to left within frames F1, F2-60, F3.

[0291] In FIG. 27, the frame numbers of frames F1 to F3 indicate the frame rates of the respective frames F1 to F3. For example, the odd-numbered frame F1-30 indicates the image data of the first image area r1-30 output by imaging at a frame rate of 30 [fps] in frame F1, and the frame F1-60 indicates the image data of the second image area r1-60 output by imaging at a frame rate of 60 [fps] in frame F1.

[0292] The second image area r1-60 output by imaging at a frame rate of 60 [fps] in frame F1-60 has image data of a train, but in frame F1-30, there is no second image area r1-60. Such an area in frame F1-30 is referred to as a non-image area n1-60. Similarly, in frame F1-60, the first image area r1-30 output by imaging at a frame rate of 30 [fps] in frame F1-30 has image data of a landscape, but in frame F1-60, there is no second image area r1-60. Such an area in frame F1-60 is referred to as a non-image area n1-30.

[0293] Similarly, in frame F3, the frame F3-30 is composed of a first image area r3-30 where image data of a landscape is output and a non-image area n3-60 where nothing is output, and the frame F3-60 is composed of a second image area r3-60 where image data of a train is output and a non-image area n3-60 where nothing is output. The same applies to odd-numbered frames not shown after frames F3-30 and F3-60.

[0294] Also, the even-numbered frame F2-60 is the second frame 713 composed of the image data (train) of the second image area r2-60 output by imaging at a frame rate of 60 [fps] and a complementary area 712y filled with a specific color (for example, black). The same applies to even-numbered frames not shown hereafter.

[0295] The synthesizing unit 703 generates a frame F2, which is synthesized image data, by synthesizing the image data (train) of the second image area r2-60 of the frame F2-60 and the image data (landscape) of the first image area r1-30 of the frame F1-30. In this case, as also described in FIG. 10, the frame F2 has a complementary image portion Da1 that overlaps between the non-image area n1-60 of the frame F1-30 and the complementary area 712y of the frame F2-60 complemented from the non-image area n2-30.

[0296] In the first embodiment, the synthesizing unit 703 fills the complementary image portion Da1 with a specific color or performs a demosaicing process. However, in the second embodiment, the synthesizing unit 703 copies the image data of the complementary image portion Da1 in other image areas without performing such image processing. As a result, the synthesizing unit 703 generates a frame F2 with less sense of incongruity. The same applies to the complementary image portion Da3, but in the second embodiment, the description will focus on the complementary image portion Da1.

[0297] <Synthesis example of frame F2> Next, a synthesis example of the frame F2 by the synthesizing unit 703 will be described.

[0298] [Synthesis example 1] FIG. 28 is an explanatory diagram showing a synthesis example 1 of the frame F2 at 60 [fps] according to the second embodiment. Synthesis example 1 is an example in which, as another image area to be copied to the complementary image portion Da1, a complementary image portion Db1 at the same position as the complementary image portion Da1 in the first image area r3-30 of the frame F3 that is one frame later in time than the frame F2-60 is used. The image data of the complementary image portion Db1 is a part of the landscape.

[0299] In FIG. 28, the synthesizing unit 703 identifies a complementary image portion Da1 where the non-image region n1-60 of the frame F1-30 and the complementary region 712y of the frame F2-60 complemented from the non-image region n2-30 overlap, and identifies a complementary image portion Db1 at the same position as the identified complementary image portion Da1 from the frame F3. Then, the synthesizing unit 703 copies the image data of the complementary image portion Db1 to the complementary image portion Da1 in the frame F2. Thereby, the synthesizing unit 703 can generate a frame F2 with less discomfort.

[0300] [Synthesis Example 2] FIG. 29 is an explanatory diagram showing Synthesis Example 2 of the 60 [fps] frame F2 according to the second embodiment. In Synthesis Example 1, the image data of the first image region r1-30 of the frame F1-30 is used as the source for copying to the first image region of the frame F2, and the image data of the complementary image portion Db1 of the frame F3 is used as the source for copying to the complementary image portion Da1. In contrast, in Synthesis Example 2, the image data of the first image region r3-30 of the frame F3-30 is used as the source for copying to the first image region of the frame F2, and the image data of the complementary image portion Db2 of the frame F1 is used as the source for copying to the complementary image portion Da2.

[0301] Here, the complementary image portion Da2 is a range where the non-image region n3-60 of the frame F3-30 and the complementary region 712y of the frame F2-60 complemented from the non-image region n2-30 overlap. The complementary image portion Db2 of the frame F1 is a range at the same position as the complementary image portion Da2.

[0302] In FIG. 29, the synthesizing unit 703 identifies a complementary image portion Da2 where the non-image region n3-60 of the frame F3-30 and the complementary region 712y of the frame F2-60 complemented from the non-image region n2-30 overlap, and identifies a complementary image portion Db2 at the same position as the identified complementary image portion Da2 from the frame F1. Then, the synthesizing unit 703 copies the image data of the complementary image portion Db2 to the complementary image portion Da2 in the frame F2. Thereby, the synthesizing unit 703 can generate a frame F2 with less discomfort.

[0303] [Synthesis Example 3] Synthesis Example 3 is an example of selecting and synthesizing either one of Synthesis Example 1 and Synthesis Example 2. In Synthesis Example 3, the synthesizing unit 703 identifies the complementary image portion Da1 in Synthesis Example 1 and the complementary image portion Da2 in Synthesis Example 2. The synthesizing unit 703 selects either one of the complementary image portions Da1 and Da2 and applies the synthesis example in which the selected range is specified. When the complementary image portion Da1 is selected, the synthesizing unit 703 applies Synthesis Example 1, and when the complementary image portion Da2 is selected, the synthesizing unit 703 applies Synthesis Example 2.

[0304] As a selection criterion for selecting either one of the complementary image portions Da1 and Da2, the synthesizing unit 703 uses, for example, the narrowness of the range. In the examples of FIGS. 28 and 29, since the complementary image portion Da1 is narrower than the complementary image portion Da2, Synthesis Example 1 is applied. By selecting the narrower range, the sense of incongruity due to replication can be minimized.

[0305] [Synthesis Example 4] FIG. 30 is an explanatory diagram showing Synthesis Example 4 of the frame F2 at 60 [fps] according to the second embodiment. In Synthesis Example 4, the replication source of the complementary image portion Da1 in Synthesis Example 1 is not the image data (a part of the landscape) of the complementary image portion Db1 in the first image area r3-30 of the frame F3, but the image data (the end of the train) of the complementary image portion Db3 in the second image area r1-60 of the frame F1.

[0306] As a result, in the frame F2, the image data (train) of the second image area r2-60 has the image data of the complementary image portion Db3 added thereto. However, since it is added to the side opposite to the traveling direction of the image data (train) of the second image area r2-60, when the user views the video, the user mistakes the image data (train) of the second image area r2-60 for an afterimage of the traveling train. Therefore, also in this case, frames F2, F4,... with less sense of incongruity can be generated.

[0307] <Example of the synthesis processing procedure of the frame F2> Next, a synthesis processing procedure example of the frame F2 according to the above-described Synthesis Examples 1 to 4 will be described. In the following flowchart, the second frame 713 is a frame that is output by imaging only at the second frame rate to be synthesized (for example, 60 [[fps]]), and in which the missing area 712x is filled with a specific color (black). For example, the frames F2-60 in FIGS. 27 to 30 are the second frame 713.

[0308] Also, the first frame 711 is a frame that is one time before the second frame 713 in terms of time and includes an image area output by imaging at at least the first frame rate (for example, 30 [[fps]]) among the first frame rate and the second frame rate. For example, the frames F1 in FIGS. 27 to 30 are the first frame 711.

[0309] Also, the third frame 730 is a frame synthesized by the second frame 713 and the first frame 711 or the third frame 730. For example, the frames F2 in FIGS. 27 to 30 are the third frame 730.

[0310] Also, the fourth frame is a frame that is one time after the second frame 713 in terms of time and includes an image area output by imaging at at least the first frame rate among the first frame rate and the second frame rate. For example, the frames F3 in FIGS. 27 to 30 are the fourth frame.

[0311] [Synthesis Example 1] FIG. 31 is a flowchart showing a synthesis processing procedure example 1 according to Synthesis Example 1 of the frame F2 by the synthesis unit 703. Note that the same step numbers are assigned to the same steps as in FIG. 26, and the description thereof is omitted.

[0312] In step S2604, when the acquired frame is the second frame 713 (step S2604: Yes), the specifying unit 1240 specifies a range that is in the non-image area of the first frame 711 and becomes the complementary area 712y of the second frame 713 (step S3101). Specifically, for example, as shown in FIG. 28, the specifying unit 1240 specifies a complementary image portion Da1 in which the non-image area n1-60 of the frame F1-30 and the complementary area 712y of the frame F2-60 complemented from the non-image area n2-30 overlap.

[0313] Next, the combining unit 703 copies the image data of the first image area a1 of the first frame 711 (step S3102). Specifically, for example, as shown in FIG. 28, the combining unit 703 copies the image data (landscape) of the first image area r1-30 of the frame F1.

[0314] Then, the combining unit 703 copies the image data of the range specified in step S3101 from the fourth frame (step S3103). Specifically, for example, as shown in FIG. 28, the combining unit 703 copies the image data of the same complementary image portion Db1 as the complementary image portion Da1 specified in step S3101 from the frame F3.

[0315] Next, the combining unit 703 generates the third frame 730 by combination (step S3104). Specifically, for example, as shown in FIG. 28, the combining unit 703 updates the frame F2-60 to the frame F2 (the third frame 730) by combining the second image area r2-60 of the frame F2-60, the image data (landscape) of the copied first image area r1-30, and the image data of the copied complementary image portion Db1.

[0316] After this, the process returns to step S2602. When there is no remaining frame in the buffer (step S2602: No), the combining unit 703 ends the combining process (step S2507). As a result, the combining unit 703 can generate a frame F2 with less discomfort, as shown in FIG. 28.

[0317] [Synthesis Example 2] FIG. 32 is a flowchart showing a synthesis processing procedure example 2 according to synthesis example 2 of frame F2 by the synthesis unit 703. The same step numbers are assigned to the same steps as in FIG. 26, and the description thereof is omitted.

[0318] In step S2604, when the acquired frame is the second frame 713 (step S2604: Yes), the specifying unit 1240 specifies a range that is in the non-image area of the fourth frame and is the complementary area 712y of the second frame 713 (step S3201). Specifically, for example, as shown in FIG. 29, the specifying unit 1240 specifies a complementary image portion Da2 in which the non-image area n3-60 of the frame F3-30 and the complementary area 712y of the frame F2-60 complemented from the non-image area n2-30 overlap.

[0319] Next, the synthesis unit 703 copies the image data of the first image area a1 of the fourth frame (step S3202). Specifically, for example, as shown in FIG. 29, the synthesis unit 703 copies the image data (landscape) of the first image area r3-30 of the frame F3.

[0320] Then, the synthesis unit 703 copies the image data in the range specified in step S3201 from the first frame 711 (step S3203). Specifically, for example, as shown in FIG. 29, the synthesis unit 703 copies the image data of the same complementary image portion Db2 as the complementary image portion Da2 specified in step S3201 from the frame F1.

[0321] Next, the synthesis unit 703 generates a third frame 730 by synthesis (step S3204). Specifically, for example, as shown in FIG. 29, the synthesis unit 703 synthesizes the second image area r2-60 of the frame F2-60, the image data (landscape) of the copied first image area r3-30, and the image data of the copied complementary image portion Db2, thereby updating the frame F2-60 to the frame F2 (third frame 730).

[0322] Subsequently, the process returns to step S2602. If there is no remaining frame in the buffer (step S2602: No), the composition unit 703 ends the composition process (step S2507). As a result, the composition unit 703 can generate a frame F2 with less discomfort as shown in FIG. 29.

[0323] [Composition Example 3] FIG. 33 is a flowchart showing an example of a composition processing procedure according to Composition Example 3 of frame F2 by the composition unit 703. Note that the same step numbers are assigned to the same steps as in FIG. 26, and the description thereof is omitted.

[0324] In step S2604, if the acquired frame is the second frame 713 (step S2604: Yes), the specifying unit 1240 specifies a first range that is in the non-image area of the first frame 711 and is the complementary area 712y of the second frame 713 (step S3301). Specifically, for example, as shown in FIG. 28, the specifying unit 1240 specifies a complementary image portion Da1 in which the non-image area n1-60 of the frame F1-30 and the complementary area 712y of the frame F2-60 complemented from the non-image area n2-30 overlap.

[0325] The specifying unit 1240 specifies a second range that is in the non-image area of the fourth frame and is the complementary area 712y of the second frame 713 (step S3302). Specifically, for example, as shown in FIG. 29, the specifying unit 1240 specifies a complementary image portion Da2 in which the non-image area n3-60 of the frame F3-30 and the complementary area 712y of the frame F2-60 complemented from the non-image area n2-30 overlap.

[0326] Next, the synthesizing unit 703 selects either the specified first range or the second range (step S3303). Specifically, for example, the synthesizing unit 703 selects the narrower (smaller in area) of the first range and the second range. The selected range is referred to as the selection range. In the case of the complementary image portions Da1 and Da2, the synthesizing unit 703 selects the complementary image portion Da1. Thereby, the range used for synthesis can be minimized, and the sense of incongruity can be further suppressed.

[0327] Then, the synthesizing unit 703 duplicates the image data of the first image area a1 of the selection frame (step S3304). The selection frame is the frame that is the source of the selection range. For example, when the first range (complementary image portion Da1) is selected, the selection frame is the first frame 711 (frame F1), and when the second range (complementary image portion Da2) is selected, the selection frame is the fourth frame (frame F3).

[0328] Therefore, the image data of the first image area a1 of the selection frame is the image data (landscape) of the first image area r1-30 of frame F1 if the selection frame is frame F1, and the image data (landscape) of the first image area r3-30 of frame F3 if the selection frame is frame F3.

[0329] Then, the synthesizing unit 703 duplicates the image data of the selection range in step S3303 from the non-selection frame (step S3305). The non-selection frame is the frame that is the source of the unselected range. For example, when the first range (complementary image portion Da1) is not selected, the non-selection frame is the first frame 711 (frame F1), and when the second range (complementary image portion Da2) is not selected, the non-selection frame is the fourth frame (frame F3). Therefore, if the selection range is the complementary image portion Da1, the synthesizing unit 703 duplicates the image data of the complementary image portion Db1 at the same position as the complementary image portion Da1 from frame F3, and if the selection range is the complementary image portion Da2, the synthesizing unit 703 duplicates the image data of the complementary image portion Db2 at the same position as the complementary image portion Da2 from frame F1.

[0330] Next, the combining unit 703 generates the third frame 730 by combining (step S3306). Specifically, for example, when the selection range is the first range (complementary image portion Da1), the combining unit 703 combines the second image region r2-60 of the frame F2-60, the image data (landscape) of the duplicated first image region r1-30, and the image data of the duplicated complementary image portion Db1 to update the frame F2-60 to the frame F2 (the third frame 730).

[0331] Also, when the selection range is the second range (complementary image portion Da2), the combining unit 703 combines the second image region r2-60 of the frame F2-60, the image data (landscape) of the duplicated first image region r3-30, and the image data of the duplicated complementary image portion Db2 to update the frame F2-60 to the frame F2 (the third frame 730).

[0332] After this, the process returns to step S2602. When there is no remaining frame in the buffer (step S2602: No), the combining unit 703 ends the combining process (step S2507). Thereby, the combining unit 703 can minimize the sense of incongruity due to duplication by selecting the narrower range.

[0333] [Synthesis Example 4] FIG. 34 is a flowchart showing an example of a synthesis processing procedure according to Synthesis Example 4 of the frame F2 by the combining unit 703. Note that the same step numbers are assigned to the same steps as in FIG. 26, and the description thereof is omitted.

[0334] In step S2604, when the acquired frame is the second frame 713 (step S2604: Yes), the specifying unit 1240 specifies a range that is in the non-image area of the first frame 711 and is the complementary area 712y of the second frame 713 (step S3401). Specifically, for example, as shown in FIG. 30, the combining unit 703 specifies a complementary image portion Da1 where the non-image area n1-60 of frame F1-30 and the complementary area 712y of frame F2-60 complemented from the non-image area n2-30 overlap.

[0335] Next, the combining unit 703 duplicates the image data of the first image area a1 of the first frame 711 (step S3402). Specifically, for example, the combining unit 703 duplicates the image data (landscape) of the first image area r1-30 of frame F1.

[0336] Then, the combining unit 703 duplicates the image data in the range specified in step S3401 from the first frame 711 (step S3403). Specifically, for example, the combining unit 703 duplicates the image data of the same complementary image portion Db3 as the complementary image portion Da1 specified in step S3401 from frame F1.

[0337] Next, the combining unit 703 generates a third frame 730 by combining (step S3404). Specifically, for example, the combining unit 703 updates frame F2-60 to frame F2 (the third frame 730) by combining the second image area r2-60 of frame F2-60, the duplicated image data (landscape) of the first image area r1-30, and the duplicated image data of the complementary image portion Db3.

[0338] After that, the process returns to step S2602. When there is no remaining frame in the buffer (step S2602: No), the combining unit 703 ends the combining process (step S2507). Thereby, as shown in FIG. 30, the combining unit 703 can generate a frame F2 with less discomfort.

[0339] (3-8) As described above, the playback device of (3-6) shown in the first embodiment has a combining unit 703. When the second frame rate is selected, the combining unit 703 acquires the first video data 721 and the second video data 722 from the storage device 1202, and combines the first frame 711 and the second frame 713 obtained temporally after the first frame 711, and generates a third frame 730 by combining the second frame 713 with the image data of the first image area a1 in the first frame 711 and the image data of the second image area a2 in the second frame 713.

[0340] Thereby, it is possible to suppress the loss of image data in the second frame 713 due to the difference in frame rate. Therefore, even when there is a difference in frame rate in one frame, the reproducibility of the subject video is enhanced by the third frame 730, and it can be reproduced as a more realistic video.

[0341] (3-9) Further, in the playback device of (3-8) above, in the image data of the second image area a2 in the second frame 713, for the area overlapping with the image data of the first image area a1 in the first frame 711, the combining unit 703 applies the image data of the second image area a2 in the second frame 713 to generate the third frame 730.

[0342] Thereby, for example, in the area where the leading part of the train in frame F2-60 which is the second frame 713 overlaps with the background area of frame F1 which is the first frame 711, the combining unit 703 preferentially applies the leading part of the train in frame F2 which is the second frame 713. Therefore, an image with less discomfort (frame F2 which is the third frame 730) can be obtained, the reproducibility of the subject video is enhanced, and it can be reproduced as a more realistic video.

[0343] (3-10) Further, in the playback device of (3-8) above, for a region that does not belong to either the second image region a2 in the second frame 713 or the first image region a1 in the first frame 711, the synthesis unit 703 generates the third frame 730 by applying the image data of the second image region a2 in the first frame 711.

[0344] Thereby, for example, for the image region between the end part of the train in the second frame of frame F2-60 which is the second frame 713 and the background region of frame F1 which is the first frame 711, the image data (the end of the train) of the second image region a2 in the first frame 711 which is the first frame 711 is preferentially applied. Therefore, an image with less discomfort (frame F2 which is the third frame 730) can be obtained, the reproducibility of the subject video is enhanced, and it can be reproduced as a more realistic video.

[0345] (3-11) Further, in the playback device of (3-5) above, the specifying unit 1240 specifies, based on the first frame 711 and the second frame 713, a complementary image portion Da1 which is a non-image region n1-60 corresponding to the second imaging region in the first frame 711 and which is the complementary region 712y in the second frame 713.

[0346] The synthesis unit 703 synthesizes the image data of the second image region a2 in the second frame 713, the image data of the first image region a1 (r1-30) corresponding to the first imaging region in the first frame 711, and the specific image data of the complementary image portion Da1 specified by the specifying unit 1240 in another image region other than the image data of the first image region a1 (r1-30) in the first frame 711 and the image data of the second image region a2 in the second frame 713.

[0347] Thereby, the non-image region n2-30 that was output during the imaging of the image data 712 can be supplemented by a frame that is temporally close to the image data 712. Therefore, a synthesized frame with less discomfort than the image data 712 can be obtained.

[0348] (3-12) Also, in the playback device of (3-11) above, the first frame 711 is a frame (for example, frame F1) generated earlier in time than the second frame 713, and specific image data is also the image data of the range (Da1) in the first image area a1 (r3-30) of a frame (for example, frame F3) generated later in time than the second frame 713 by the outputs from the first imaging area and the second imaging area (that is, the image data of the complementary image portion Db1).

[0349] Thus, the non-image area n2-30 output in the imaging of the second frame 713 can be supplemented by the first frame 711 one time earlier in time than the second frame 713 and the fourth frame one time later. Therefore, a composite frame (the third frame 730) with less sense of incongruity can be obtained.

[0350] Also, in the playback device of (3-11) above, the first frame 711 is a frame (for example, frame F3) generated later in time than the second frame 713, and specific image data is also the image data of the range (Da2) in the first image area a1 (r1-30) of a frame (for example, frame F1) generated earlier in time than the second frame 713 by the outputs from the first imaging area and the second imaging area (that is, the image data of the complementary image portion Db2).

[0351] Thus, the non-image area n2-30, which is the complementary area 712y of the second frame 713, can be supplemented by the first frame 711 one time earlier in time than the second frame 713 and the fourth frame one time later. Therefore, a composite frame (the third frame 730) with less sense of incongruity can be obtained.

[0352] In the playback device of (3-5) above, the specifying unit 1240 specifies the range to be used by the synthesizing unit 703 based on the first range (Da1) and the second range (Da2). Further, the synthesizing unit 703 combines the second frame 713, the image data of the first image region a1 (r1-30 / r3-30) in one of the frames (F1 / F3) that is the source of the one range (Da1 / Da2) specified by the specifying unit 1240 among the first frame 711 and the fourth frame, and the image data (Db1 / Db2) of one of the ranges (Da1 / Da2) in the first image region a1 (r3-30 / r1-30) of the other frame (F3 / F1) that is the source of the other range (Da2 / Da1) not specified by the specifying unit 1240 among the first frame 711 and the fourth frame.

[0353] Thereby, the synthesizing unit 703 can minimize the discomfort due to duplication by selecting the narrower range.

[0354] In the playback device of (3-5) above, the first frame 711 is a frame generated earlier in time than the second frame 713, and the specific image data may be the image data of the range (Da1) in the second image region a2 of the first frame 711 (that is, the image data of the complementary image portion Db3).

[0355] Thereby, the non-image region n2-30, which is the complementary region 712y of the second frame 713, can be supplemented by the first frame 711 that is one time earlier in time than the second frame 713. Therefore, a synthesized frame (third frame 730) with less discomfort can be obtained.

Example

[0356] Example 3 will be described. In Example 1, in the frames F2, F4, … shown in FIG. 10, since the complementary image portions Da1, Da3, … exist, the synthesizing unit 703 performed painting with a specific color or executed a demosaicing process. In Example 3, similar to Example 2, the synthesizing unit 703 generates frames F2, F4, … with less discomfort without performing such image processing.

[0357] Note that in Example 3, for the common parts with Example 1 and Example 2, the same reference numerals are used and the description thereof is omitted. However, in FIGS. 35 and 36, since the reference numerals are unclear, black painting by complementation is not performed.

[0358] FIG. 35 is an explanatory diagram showing a synthesis example of the frame F2 at 60 [fps] according to Example 3. Before imaging the frame F2-60, the preprocessing unit 1210 detects a specific subject such as a train from frames F1 etc. before the frame F2-60, and detects the motion vector of the specific subject in the previous frame F1. The preprocessing unit 1210 can obtain the image area R12-60 at 60 [fps] in the next frame F2-60 based on the image area and the motion vector of the specific subject in the frame F1.

[0359] Also, in the synthesis of the frame F2 which is a synthesized frame, similar to Example 1, the synthesizing unit 703 copies the image data (landscape) of the first image area r1-30 of the previous frame F1, and combines the image data (landscape) of the first image area r1-30 with the image data (part of the train and the landscape) of the image area R12-60, thereby obtaining the frame F2.

[0360] FIG. 36 is an explanatory diagram showing the correspondence between the setting of the imaging area and the image area of the frame F2-60. (A) shows an example of detecting a motion vector, and (B) shows the correspondence between the setting of the imaging area and the image area of the frame F2-60.

[0361] The imaging area p1-60 is the imaging area of a specific subject that has already been detected after the generation of the frame F0-60, which is one frame before the frame F1 in terms of time, and before the generation of the frame F1. Therefore, in the frame F1, the image data o1 of the specific subject (train) exists in the second image area r1-60 corresponding to the imaging area p1-60.

[0362] The preprocessing unit 1210 detects the motion vector mv of the specific subject based on the image data o1 of the specific subject in the frame F0 and the image data o1 of the specific subject in the frame F1 by the detection unit 1211. Then, the preprocessing unit 1210 detects the second image area r2-60 in which the specific subject is projected in the next frame F2-60 based on the second image area r1-60 of the specific subject in the frame F1 and the motion vector mv, and detects the detection imaging area p2-60 of the imaging surface 200 of the imaging device 100 corresponding to the detected second image area r2-60.

[0363] The preprocessing unit 1210 sets the frame rate of the specific imaging area P12-60 that includes the imaging area p1-60 specified at the time of generating the frame F1 and the detection imaging area p2-60 to the second frame rate by the setting unit 1212, and outputs the setting instruction to the imaging device 100. As a result, the imaging device 100 sets the specific imaging area P12-60 to the second frame rate and performs imaging to generate the image data 712.

[0364] The first generation unit 701 complements the image data 712 generated by imaging at the second frame rate set by the setting unit 1212, and outputs the second frame 713 (F2-60). In this case, the image data output from the specific imaging area P12-60 becomes the image data of the image area R12-60.

[0365] The composition unit 703 composes the image data of the first image area r1-30 included in the frame F1 and the image data (image area R12-60) from the specific imaging area P12-60 included in the second frame 713 (F2-60). As a result, the frame F2-60 is updated to the frame F2 (the third frame 730).

[0366] Note that, after the generation of frame F2-60 and before the generation of the next frame F3, the preprocessing unit 1210 may set the frame rate of the detection imaging area p2-60 to the second frame rate and set the frame rates of the other imaging areas in the imaging surface 200 other than the detection imaging area p2-60 to the first frame rate.

[0367] Thereby, in the generation of frame F3 obtained by imaging including the imaging area with the first frame rate, similar to frame F1, the second imaging area where the second frame rate is set is only the detection imaging area p2-60. In this way, since specific detection imaging areas are set for the frames F2-60, F4-60,... to be synthesized, wasteful processing in frames F1, F3,... is suppressed.

[0368] Frame F2-60 includes the image data o1 of a specific subject (train) and the image data o2 of a part of the scenery in the image area R12-60. In this way, the image area R12-60 is extended on the opposite side of the moving direction of the specific subject compared to the second image area r2-60. Therefore, it is not necessary to identify the complementary image portions Da1, Da2 as in the second embodiment and replicate and synthesize the image data of the complementary image portions Db1, Db2 of other frames. Note that the synthesis process of the third embodiment is executed, for example, in step S2507 of FIG. 25. Also, this synthesis process is applied to the synthesis of frames F2-60, F4-60,... and is not executed in frames F1, F3,... including the image area with the first frame rate.

[0369] Thus, in the third embodiment, since the source image data consists of the image region R12-60 within the second frame 713 and the first image region r1-30 of the first frame F1, a frame F2 with less discomfort can be generated. That is, since the image data o1 and o2 are the image data output by imaging at the same timing, the boundary between the image data o1 and o2 is not unnatural and causes no discomfort. Also, as in the second embodiment, since there is no need for processes such as specifying the complementary image portions Da1 and Da2 or selecting an optimal range from the complementary image portions Da1 and Da2, the load of the synthesis process for frame F2 can be reduced.

[0370] (4-1) As described above, the imaging device according to the third embodiment includes an imaging element 100, a detection unit 1211, and a setting unit 1212. The imaging element 100 has a first imaging region for imaging a subject and a second imaging region for imaging a subject, and it is possible to set a first frame rate (e.g., 30 [fps]) for the first imaging region and a second frame rate (e.g., 60 [fps]), which is faster than the first frame rate, for the second imaging region.

[0371] Based on the second image region r1-60 of a specific subject included in the frame F1 generated by the output from the imaging element 100, the detection unit 1211 detects the detection imaging region p2-60 of the specific subject in the imaging element 100. The setting unit 1212 sets the frame rate of a specific imaging region P12-60 that includes the imaging region p1-60 of the specific subject used for generating the frame F1 and the imaging region (hereinafter referred to as the detection imaging region) p2-60 detected by the detection unit 1211 to the second frame rate.

[0372] Thereby, the imaging region at the second frame rate can be extended and set, and the specific subject can be imaged at the second frame rate so that the complementary image portion Da1 where the non-image regions overlap in the frames F1 and F2 does not occur, and the image loss of the frame F2-60 output by imaging at the second frame rate can be suppressed.

[0373] (4-2) Further, in the imaging device of (4-1) above, the detection unit 1211 detects the detection imaging region p2-60 of the specific subject based on the second image region r1-60 of the specific subject included in the frame F1 and the motion vector mv of the specific subject between the frame F1 and the frame F0-60 that is temporally earlier than the frame F1.

[0374] Thereby, prediction of the detection imaging region p2-60 of the specific subject can be easily realized.

[0375] (4-3) Further, in the imaging device of (4-1) above, when the frame is the first frame F1 generated by the output from the first imaging region, the setting unit 1212 sets the frame rate of the specific imaging region to the second frame rate, and when the frame is the second frame F2-60 generated by the output from the specific imaging region after the first frame F1, the frame rate of the detection imaging region p2-60 is set to the second frame rate, and the frame rate of the other imaging regions (the portion of the imaging surface 200 excluding the detection imaging region p2-60) other than the detection imaging region p2-60 is set to the first frame rate.

[0376] Thereby, since a specific detection imaging region is set only for the frames F2-60, F4-60,... to be synthesized, it is possible to suppress unnecessary processing in the frames F1, F3,....

[0377] (4-4) Further, the image processing device according to Embodiment 3 has a first imaging region for imaging a subject and a second imaging region for imaging a subject, and can set a first frame rate (for example, 30 [fps]) for the first imaging region, and can set a second frame rate (for example, 60 [fps]) higher than the first frame rate for the second imaging region, and processes an image of a frame generated by the output from the imaging element 100.

[0378] This image processing apparatus includes a detection unit 1211, a setting unit 1212, a first generation unit 701, and a synthesis unit 703. The detection unit 1211 detects an imaging region p2-60 of a specific subject in the imaging device 100 based on a second image region r1-60 of the specific subject included in a frame F1 generated by the output from the imaging device 100. The setting unit 1212 sets the frame rate of a specific imaging region P12-60 that includes the imaging region p1-60 of the specific subject used for generating the frame F1 and the detected imaging region p2-60 detected by the detection unit 1211 to a second frame rate.

[0379] The first generation unit 701 complements the image data 712 generated by imaging at the second frame rate set by the setting unit 1212 and outputs a second frame 713 (F2-60).

[0380] The synthesis unit 703 synthesizes the image data of the first image region r1-30 included in the first frame F1 and the image data (image region R12-60) from the specific imaging region P12-60 included in the second frame 713 (F2-60).

[0381] Thereby, the imaging region at the second frame rate can be extended and set, and the specific subject can be imaged at the second frame rate so that a complementary image portion Da1 where non-image regions overlap in the frames F1 and F2 does not occur, and image loss of the frame F2-60 output by imaging at the second frame rate can be suppressed. Also, at the time of synthesis, since it is not necessary to supplement the overlapping complementary image portion Da1, an image with less discomfort can be obtained, and the load of the synthesis process can be reduced.

[0382] Note that the present invention is not limited to the above content, and any combination thereof may be used. Also, other aspects conceivable within the scope of the technical idea of the present invention are included in the scope of the present invention.

Explanation of Reference Numerals

[0383] 100 imaging element, 701 completion unit, 702 compression / expansion unit, 703 synthesis unit, 704 playback unit, 800 video file, 801 header unit, 802 data unit, 835 additional information, 910 imaging condition information, 911 frame rate information, 912 position information, 920 insertion position information, 921 insertion frame number, 922 insertion destination, 1201 processor, 1202 memory device, 1210 preprocessing unit, 1211 detection unit, 1212 setting unit, 1220 acquisition unit, 1231 compression unit, 1232 generation unit, 1233 selection unit, 1234 expansion unit, 1240 identification unit

Claims

【Claim 1】 A video compression device that has a plurality of imaging regions for imaging a subject and compresses a plurality of frames output from an image sensor in which imaging conditions can be set for each imaging region, an acquisition unit that acquires data output from a first imaging region for which a first frame rate is set and data output from a second imaging region for which a second frame rate is set; a generation unit that generates a plurality of first frames based on the data output from the first imaging region acquired by the acquisition unit and generates a plurality of second frames based on the data output from the second imaging region; a compression unit that compresses the plurality of first frames generated by the generation unit and compresses the plurality of second frames; A video compression device comprising:

Images