Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device, an image capturing device, an image processing method, and an image processing program that perform suitable compression processing on moving images captured in a first moving image capturing mode and a second moving image capturing mode, the shooting conditions of which are different.SOLUTION: In an image processing method, the possible range of a quantization parameter applied when compressing a moving image is made different between a first moving image shooting mode and a second moving image shooting mode which have different shooting conditions. In the first moving image shooting mode, a quantization parameter is determined within a first range, and in the second moving image shooting mode, the quantization parameter is determined within a second range which is narrower than the first range. In particular, the second upper limit value of the second range is made smaller than the first upper limit value of the first range, and the second lower limit value of the second range is made larger than the first lower limit value of the first range.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image capturing device, an image processing method, and an image processing program, and more particularly to a technique for compressing moving images. [Background technology]

[0002] In recent years, in order to capture a momentary scene of a subject, an imaging device has been proposed that has a video imaging mode for capturing a video for extracting a still image (Patent Document 1).

[0003] The moving image from which a still image is extracted, as described in Patent Document 1, is set to have a shorter exposure time per frame than a normal moving image, for example.

[0004] Incidentally, since the amount of moving image data is enormous, when recording moving images, the moving images are compressed and recorded using, for example, the MPEG (Moving Picture Experts Group) coding method.

[0005] When compressing moving images, increasing the compression rate of the moving image results in a decrease in image quality, while decreasing the compression rate increases the bit rate (the number of bits transferred or processed per unit time), which can exceed the processing capacity of the device.

[0006] Therefore, the quantization parameter (QP) value is controlled according to the amount of generated code after quantization of image data of past frames of video. When the amount of generated code increases, the QP The P value is increased (compression rate is increased) to limit the bit rate so that it does not exceed the processing capacity, and when the amount of generated code becomes small, the QP value is decreased (compression rate is decreased) to achieve high image quality.

[0007] Patent Document 2 proposes an image processing device that can avoid image quality degradation immediately after the image being processed is switched from a still image to a moving image. This image processing device limits the amount of generated code to a preset lower limit if the amount of generated code falls below that lower limit, which may occur immediately after the image being processed is switched from a still image to a moving image.

[0008] That is, even if the image to be compressed is a still image and the actual amount of generated code is zero or close to it, it is assumed that a certain amount of code (lower limit value) has been generated, and this prevents the QP value from being set too small when the image to be compressed switches from a still image to a video. As a result, immediately after the image to be compressed switches from a still image to a video, a situation where the QP value is set to an excessively small value and an excessively large value is repeated does not occur, and deterioration of the video image quality at the time of switching from a still image to a video is avoided. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-32303 [Patent Document 2] Japanese Patent Application Laid-Open No. 2013-187634 Summary of the Invention [Problem to be solved by the invention]

[0010] The image processing device described in Patent Document 2 compresses an input image that contains a mixture of still image frames and moving image frames, but does not compress moving images for extracting still images.

[0011] Furthermore, when a video for extracting still images is subjected to compression processing, any frame of the video may be extracted as a still image, so it is necessary to maintain a constant image quality (the image quality required for a still image) for all frames of the video. However, Patent Document 2 does not describe compressing a video for extracting still images, and does not perform compression processing to maintain a constant image quality for all frames of the video.

[0012] In the case of normal videos that are viewed as videos, the compression process is optimized taking into account the image quality of the entire video, so there is a possibility that the QP value may be adjusted to be too large for the purpose of extracting a single frame of the video as a still image, resulting in the problem that the image quality required for a still image cannot be obtained.

[0013] The present invention has been made in view of the above circumstances, and provides a first moving image shooting mode in which shooting conditions are different. and the second video shooting mode, a preferred setting is made for the video shot in each video shooting mode. The present invention aims to provide an image processing device, a photographing device, an image processing method, and an image processing program that are capable of performing appropriate compression processing. [Means for solving the problem]

[0014] In order to achieve the above-mentioned object, one aspect of the invention is an image processing device comprising a video acquisition unit that acquires a video captured based on a first video shooting mode or a second video shooting mode whose shooting conditions are different from those of the first video shooting mode, and a compression processing unit that determines a quantization parameter for image data of frames constituting the video acquired by the video acquisition unit and compresses the image data, wherein the compression processing unit determines the quantization parameter within a first range in the first video shooting mode, and determines the quantization parameter within a second range in the second video shooting mode, and the second upper limit value of the second range is smaller than the first upper limit value of the first range, and the second lower limit value of the second range is larger than the first lower limit value of the first range.

[0015] According to one aspect of the present invention, a moving image is captured in a first moving image capturing mode and a second moving image capturing mode. The range of possible quantization parameters applied when compressing is varied, and in the first moving image capture mode, the quantization parameters are determined within a first range, and in the second moving image capture mode, the quantization parameters are determined within a second range narrower than the first range. In particular, by making the second upper limit value of the second range smaller than the first upper limit value of the first range and the second lower limit value of the second range larger than the first lower limit value of the first range, the compression process for moving images captured in the second moving image capture mode can maintain a constant image quality for all frames of the moving image and prevent the bit rate from becoming excessive, compared to the compression process for moving images captured in the first moving image capture mode.

[0016] In the image processing device according to another aspect of the present invention, the second range is preferably set in accordance with a set value of the frame rate.

[0017] In the image processing device according to still another aspect of the present invention, it is preferable that the second range be set in response to an input from an instruction input unit that inputs an instruction from outside.

[0018] In an image processing device according to yet another aspect of the present invention, it is preferable to include a scene determination unit that determines the scene of the moving image acquired by the moving image acquisition unit, and the second range is set according to the scene determined by the scene determination unit. In a scene with a lot of subject movement, such as a sports scene, the second lower limit value may be set to be larger than in other modes with less movement, so that the bit rate does not become excessively large.

[0019] In the image processing device according to still another aspect of the present invention, the compression processing unit performs the MPEG encoding method. It is preferable to compress moving images using the MPEG encoding method, which is a typical encoding method for compressing moving images, and includes MPEG-2, MPEG-4, H.264 / AVC, and the like.

[0020] In the image processing device according to still another aspect of the present invention, it is preferable that the compression processing unit determines the quantization parameter in accordance with the amount of code generated after quantization of image data of past frames of the moving image.

[0021] In an image processing device according to yet another aspect of the present invention, it is preferable that in the second video shooting mode, at least one of the shutter speed, autofocus speed, autoexposure tracking speed, white balance tracking speed, and frame rate is set to be faster than in the first video shooting mode.

[0022] A photographing device according to yet another aspect of the present invention includes the above-described image processing device and a video shooting unit that shoots a video based on a first video shooting mode or a second video shooting mode, and a video acquisition unit that acquires the video shot by the video shooting unit.

[0023] Yet another aspect of the invention is an image processing method including a video acquisition step of acquiring a video captured based on a first video shooting mode or a second video shooting mode whose shooting conditions are different from those of the first video shooting mode, and a compression processing step of determining a quantization parameter for image data of frames constituting the video acquired in the video acquisition step and compressing the image data, wherein the compression processing step determines a quantization parameter within a first range in the first video shooting mode, and determines a quantization parameter within a second range in the second video shooting mode, and the second upper limit value of the second range is smaller than the first upper limit value of the first range, and the second lower limit value of the second range is larger than the first lower limit value of the first range.

[0024] In the image processing method according to yet another aspect of the present invention, the second range is preferably set in accordance with a set value of the frame rate.

[0025] In the image processing method according to still another aspect of the present invention, the second range is preferably set in response to an input from an instruction input unit that inputs an instruction from outside.

[0026] In an image processing method according to yet another aspect of the present invention, it is preferable to include a step of determining a scene of the video acquired in the video acquisition step, and the second range is set according to the determined scene.

[0027] In the image processing method according to still another aspect of the present invention, the compression processing step preferably compresses the moving image using an MPEG encoding method.

[0028] In the image processing method according to still another aspect of the present invention, the compression step preferably determines a quantization parameter in accordance with a generated code amount after quantization of image data of a past frame of the moving image.

[0029] In an image processing method according to yet another aspect of the present invention, it is preferable that the second video shooting mode has at least one of the shutter speed, autofocus speed, autoexposure tracking speed, white balance tracking speed, and frame rate set to be faster than the first video shooting mode.

[0030] The invention according to still another aspect includes a video acquisition function for acquiring a video captured in a first video capture mode or a second video capture mode having different capture conditions from the first video capture mode, and a video capture function for determining a quantization parameter for image data of frames that constitute the video captured by the video acquisition function. and a compression processing function for compressing image data, wherein the compression processing function determines a quantization parameter within a first range in a first moving image shooting mode, and determines a quantization parameter within a second range in a second moving image shooting mode, and the second upper limit value of the second range is smaller than the first upper limit value of the first range, and the second lower limit value of the second range is larger than the first lower limit value of the first range. [Effects of the Invention]

[0031] According to the present invention, suitable compression processing can be performed on videos captured in each video capture mode depending on the first video capture mode and the second video capture mode, which have different shooting conditions.In particular, the compression processing on videos captured in the second video capture mode can be performed in a manner that maintains a constant image quality for all frames of the video while preventing the bit rate from becoming excessive, compared to the compression processing on videos captured in the first video capture mode. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a perspective view of the imaging device according to the present invention, seen obliquely from the front. [Figure 2] FIG. 2 is a rear view of the imaging device. [Figure 3] FIG. 3 is a block diagram showing an embodiment of the internal configuration of the imaging device. [Figure 4] FIG. 4 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention. [Figure 5] FIG. 5 is a schematic diagram showing the relationship between the QP value and the change in the amount of generated code after quantization of image data of past frames when compressing a normal moving image. [Figure 6] FIG. 6 is a schematic diagram showing the relationship between the QP value and the change in the amount of generated code after quantization of image data of past frames when compressing a moving image for extracting still images. [Figure 7] FIG. 7 is a block diagram showing a second embodiment of an image processing device according to the present invention. [Figure 8] FIG. 8 is a flowchart showing an embodiment of an image processing method according to the present invention. [Figure 9] FIG. 9 is an external view of a smartphone, which is an embodiment of the imaging device according to the present invention. [Figure 10] FIG. 10 is a block diagram showing the configuration of a smartphone. DETAILED DESCRIPTION OF THE INVENTION

[0033] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an image processing device, a photographing device, an image processing method, and an image processing program according to the present invention will now be described with reference to the accompanying drawings.

[0034] <Appearance of the imaging device> FIG. 1 is a perspective view of an imaging device according to the present invention as seen obliquely from the front, and FIG. 2 is a rear view of the imaging device.

[0035] As shown in FIG. 1, the photographing device 10 is a mirrorless digital single-lens camera that is made up of an interchangeable lens 100 and a camera body 200 to which the interchangeable lens 100 can be attached or detached.

[0036] In FIG. 1, the front of the camera body 200 is provided with a body mount 260 to which the interchangeable lens 100 is attached, a viewfinder window 20 of the optical viewfinder, etc., and the top of the camera body 200 is mainly provided with a shutter release switch 22, a shutter speed dial 23, an exposure compensation dial 24, a power lever 25, and a built-in flash 30.

[0037] As shown in FIG. 2, the rear surface of the camera body 200 is provided with a liquid crystal monitor 216, an optical finder eyepiece 26, a MENU / OK key 27, a cross key 28, a playback button 29, and the like.

[0038] The LCD monitor 216 displays live view images in the shooting mode, plays back and displays captured images in the playback mode, and also functions as a display unit that displays various menu screens. The cross key 28 functions as a notification unit that notifies the user of various information. The MENU / OK key 27 is an operation key that functions both as a menu button for issuing a command to display a menu on the screen of the LCD monitor 216 and as an OK button for issuing a command to confirm and execute a selection. The cross key 28 is an operation unit that inputs instructions in four directions (up, down, left, and right), and functions as a multi-function key that selects an item from the menu screen and instructs the selection of various setting items from each menu. The up and down keys of the cross key 28 also function as a zoom switch during shooting or a playback zoom switch in playback mode, and the left and right keys function as frame-by-frame (forward and backward) buttons in playback mode. The cross key 28 also functions as an operation unit that specifies a desired subject for focus adjustment from multiple subjects displayed on the LCD monitor 216.

[0039] The MENU / OK key 27, the cross key 28, and the liquid crystal monitor 216 function as a shooting mode selection section for selecting various shooting modes, and also function as a scene selection section (scene determination section).

[0040] That is, by operating the MENU / OK key 27 to display a menu screen on the LCD monitor 216, and using that menu screen, it is possible to set a still image shooting mode for shooting a single still image, as well as a moving image shooting mode for shooting a moving image. The moving image shooting mode includes a first moving image shooting mode and a second moving image shooting mode which has different shooting conditions from the first moving image shooting mode.

[0041] In the second video capture mode, a video is captured under different capture conditions than in the first video capture mode (a video under capture conditions that prioritize extracting still images over viewing the video itself). Specifically, in the second video capture mode, at least one of the shutter speed, autofocus speed, autoexposure tracking speed, and white balance tracking speed is set faster than in the first video capture mode, and / or the frame rate is set higher than in the first video capture mode. In addition, the resolution and frame rate are set to the highest values ​​that can be set in the image capture device 10 (for example, 4,000 x 2,000 pixels, 30 fps (frames per second)), and the color tone is also set with the assumption that still images will be extracted. The upper limit of the ISO sensitivity is also set higher than in the first video capture mode.

[0042] For example, in the first video capture mode, the shutter speed is set to a value corresponding to the frame rate of the video being recorded (1 / 30 seconds if the frame rate is 30 fps), but in the second video mode, it is set to a value faster than the frame interval (e.g., less than 1 / 30 seconds). In the first video capture mode, the shutter speed is set to a value corresponding to the frame rate of the video to ensure smooth video playback, but in this case, blurring of moving subjects may occur. For this reason, in the second video capture mode, the shutter speed is set faster than in the first video capture mode (faster than the frame interval), which makes it possible to extract high-quality still images with less subject blurring. Similarly, by increasing the upper limit of the ISO sensitivity, the shutter speed can be increased, thereby enabling the extraction of still images with less blurring. Furthermore, by setting the autofocus speed, autoexposure tracking speed, autowhite balance tracking speed, etc. faster than in the first video capture mode, it is possible to capture more frames in which the subject is in focus, frames with appropriate exposure, etc. Regarding the frame rate, by setting it to a high rate, the frame interval of the video becomes shorter and the number of frames that can be extracted as still images increases. The frame rate setting value (30 fps, 60 fps, etc.) can be set using the MENU / OK key 27, the cross key 28, etc.

[0043] According to the second video capture mode described above, a video can be stored and frames constituting the video can be extracted as still images, allowing the user to easily capture photos of events that may occur at any time (natural phenomena, accidents, unexpected happenings, etc.), or photos of the instantaneous state of a subject whose state changes over time or a moving subject. In this case, still images can be extracted not only at the time when a still image recording command is issued but also at other times, allowing the user to obtain still images at the desired timing. Furthermore, by setting the shooting conditions suitable for still image extraction (such as the shutter speed, resolution, and frame rate described above), high-quality still images can be extracted.

[0044] Furthermore, by operating the MENU / OK key 27 to display a menu screen on the LCD monitor 216, a shooting scene can be selected using the menu screen. Shooting scenes include landscapes, night scenes, sunsets, macro scenes, portraits, moving subjects, and sports. Any of these shooting scenes can be manually or automatically selected or determined and set in the camera. Depending on the shooting scene set in the camera, optimal shooting conditions for that shooting scene are set.

[0045] The playback button 29 is a button for switching to a playback mode in which a recorded still image or video is displayed on the liquid crystal monitor 216 .

[0046] <Internal structure of the imaging device> [Interchangeable Lens] FIG. 3 is a block diagram showing an embodiment of the internal configuration of the image capturing device 10. As shown in FIG.

[0047] The interchangeable lens 100 that functions as the photographic optical system that constitutes the photographing device 10 is manufactured in accordance with the communication standard of the camera body 200, and is an interchangeable lens that can communicate with the camera body 200, as will be described later. The interchangeable lens 100 includes a photographic optical system 102, a focus lens control unit 116, an aperture control unit 118, a lens-side CPU (Central Processing Unit) 120, a flash ROM (Read Only Memory) 126, a lens-side communication unit 150, and a lens mount 160.

[0048] The imaging optical system 102 of the interchangeable lens 100 includes a lens group 104 including a focus lens, and an aperture 108 .

[0049] The focus lens control unit 116 moves the focus lens and controls the position (focus position) of the focus lens in accordance with a command from the lens side CPU 120. The iris control unit 118 controls the iris 108 in accordance with a command from the lens side CPU 120.

[0050] The lens-side CPU 120 controls the interchangeable lens 100 and includes a ROM 124 and a RAM (Random Access Memory) 122 built in.

[0051] The flash ROM 126 is a non-volatile memory that stores programs downloaded from the camera body 200 and the like.

[0052] The lens-side CPU 120 performs overall control of each section of the interchangeable lens 100 in accordance with a control program stored in a ROM 124 or a flash ROM 126, using a RAM 122 as a work area.

[0053] With the lens mount 160 attached to a body mount 260 of the camera body 200, the lens side communication unit 150 communicates with the camera body 200 via multiple signal terminals (lens side signal terminals) provided on the lens mount 160. That is, in accordance with commands from the lens side CPU 120, the lens side communication unit 150 transmits and receives (bidirectional communication) request signals and response signals to and from a body side communication unit 250 of the camera body 200 connected via the lens mount 160 and the body mount 260, and notifies the camera body 200 of lens information of each optical member of the photographic optical system 102 (position information of the focus lens, focal length information, aperture information, etc.).

[0054] The interchangeable lens 100 also includes a detection unit (not shown) that detects position information of the focus lens and aperture information. Here, the aperture information is information indicating the aperture value (F-number) of the aperture 108, the opening diameter of the aperture 108, etc.

[0055] In order to respond to a request for lens information from the camera body 200, the lens-side CPU 120 preferably stores various types of lens information, including detected focus lens position information and aperture information, in the RAM 122. Furthermore, the lens information can be detected when there is a request for lens information from the camera body 200, or detected when the optical member is driven, or detected at a fixed cycle (a cycle sufficiently shorter than the frame cycle of a video), and the detection results can be stored.

[0056] [Camera body] The camera body 200 constituting the photographing device 10 shown in FIG. 3 includes an image sensor 201, an image sensor control unit 202, an analog signal processing unit 203, an A / D (Analog / Digital) converter 204, an image input controller 205, a digital signal processing unit 206, a RAM 207, a compression / decompression processing unit 208, a media control unit 210, a memory card 212, a display control unit 214, a liquid crystal monitor 216, a main body CPU 220, an operation unit 222, a clock unit 224, a flash ROM 226, a ROM 228, an AF (Autofocus) control unit 230, an AE (Auto Exposure) control unit 232, a white balance correction unit 234, a wireless communication unit 236, a GPS (Global Positioning System) receiving unit 238, a power control unit 240, a battery 242, a main body communication unit 250, a main body mount 260, a flash light emission unit 270 and a flash control unit 272 constituting the built-in flash 30 (FIG. 1), a focal-plane shutter (FPS), a The controller 290 includes a shutter 280 and an FPS control unit 296.

[0057] The image sensor 201 is a CMOS (Complementary Metal-Oxide Semiconductor) type The image sensor 201 is configured by a color image sensor. The image sensor 201 is not limited to a CMOS type, but may be an XY address type or a CCD (Charge Coupled Device) type image sensor. That's fine too.

[0058] Each pixel of the image sensor 201 is provided with a color filter of one of the three primary colors, red (R), green (G), and blue (B) (R filter, G filter, B filter), arranged in a predetermined color filter array. The color filter array may be a typical Bayer array, but is not limited to this, and may be, for example, a Trans (registered trademark) array or other color filter array.

[0059] An optical image of a subject formed on the light receiving surface of the image sensor 201 by the photographing optical system 102 of the interchangeable lens 100 is converted into an electrical signal by the image sensor 201. Charges corresponding to the amount of incident light are accumulated in each pixel of the image sensor 201, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 201 as an image signal.

[0060] The image sensor control unit 202 controls the reading of image signals from the image sensor 201 in accordance with commands from the main body CPU 220. When capturing a still image, the image sensor control unit 202 controls the exposure time by opening and closing the FPS 280, and then reads out all lines of the image sensor 201 with the FPS 280 closed. The image sensor 201 and image sensor control unit 202 in this example perform an exposure operation sequentially for at least one line or pixel (i.e., a method in which reset is performed sequentially for each line or pixel, charge accumulation is started, and the accumulated charge is read out), a so-called rolling shutter method. In particular, it has the function of shooting video or live view images using the rolling shutter method with the FPS280 open.

[0061] The analog signal processing unit 203 performs various analog signal processing operations on the analog image signal obtained by capturing an image of a subject with the image sensor 201. The analog signal processing unit 203 includes a sample-and-hold circuit, a color separation circuit, an AGC (Automatic Gain Control) circuit, etc. The AGC circuit functions as a sensitivity adjustment unit that adjusts the sensitivity (ISO sensitivity (ISO: International Organization for Standardization)) during shooting, and adjusts the gain of an amplifier that amplifies the input image signal so that the signal level of the image signal falls within an appropriate range. The A / D converter 204 converts the analog image signal output from the analog signal processing unit 203 into a digital image signal.

[0062] When capturing a still image or a moving image, RGB image data for each pixel (mosaic image data) output via the image sensor 201, analog signal processing unit 203, and A / D converter 204 is input from the image input controller 205 to the RAM 207 and temporarily stored therein. Note that when the image sensor 201 is a CMOS image sensor, the analog signal processing unit 203 and the A / D converter 204 are often built into the image sensor 201.

[0063] The digital signal processing unit 206 performs various types of digital signal processing on image data stored in the RAM 207. The digital signal processing unit 206 appropriately reads image data stored in the RAM 207, performs digital signal processing on the read image data such as offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaicing processing (also called demosaicing processing or synchronization processing), and RGB / YCrCb conversion processing, and stores the image data after digital signal processing back in the RAM 207. Note that demosaicing is, for example, in the case of an image sensor consisting of three RGB color filters, a process of calculating all RGB color information for each pixel from a mosaic image made up of RGB, and generates synchronized image data of three RGB planes from the mosaic data (dot-sequential RGB data).

[0064] The RGB / YCrCb conversion process converts the synchronized RGB data into luminance data (Y) and color difference data (Cr, Cb).

[0065] When recording a still image or a moving image, a compression / decompression processing unit 208 performs compression processing on the uncompressed luminance data Y and color difference data Cb, Cr temporarily stored in the RAM 207. In the case of a still image, compression is performed in, for example, JPEG (Joint Photographic Coding Experts Group) format, and in the case of a moving image, compression is performed in JPEG (Joint Photographic Coding Experts Group) format. In this case, the data is compressed using, for example, the H.264 / AVC (Advanced Video Coding) format, which is one of the MPEG encoding formats. The image data compressed by the compression / decompression processing unit 208 is recorded on the memory card 212 via the media control unit 210. In addition, the compression / decompression processing unit 208 performs decompression processing on the compressed image data obtained from the memory card 212 via the media control unit 210 in the playback mode, and generates uncompressed image data.

[0066] The compression / decompression processing unit 208 (particularly the compression processing unit) according to the present invention will be described in detail later.

[0067] The media control unit 210 controls the recording of image data compressed by the compression / decompression processing unit 208 onto a memory card 212. The media control unit 210 also controls the reading of compressed image data from the memory card 212.

[0068] A display control unit 214 controls the display of uncompressed image data stored in the RAM 207 on a liquid crystal monitor 216. The liquid crystal monitor 216 is configured by a liquid crystal display device. However, the liquid crystal monitor 216 may be replaced by a display device such as an organic electroluminescence device.

[0069] When a live view image is to be displayed on the liquid crystal monitor 216, digital image signals continuously generated by the digital signal processing unit 206 are temporarily stored in the RAM 207. The display control unit 214 converts the digital image signals temporarily stored in the RAM 207 into a signal format for display and sequentially outputs them to the liquid crystal monitor 216. This allows the captured image to be displayed in real time on the liquid crystal monitor 216, allowing the liquid crystal monitor 216 to be used as an electronic viewfinder.

[0070] The shutter release switch 22 is a shooting instruction unit for inputting instructions to shoot still images or moving images, and is configured as a two-stroke switch consisting of a so-called "half-press" and "full press."

[0071] In the still image shooting mode, an S1 on signal is output when the shutter release switch 22 is pressed halfway, and an S2 on signal is output when the switch is pressed further from halfway to the full press. When the S1 on signal is output, the main body CPU 220 executes shooting preparation processes such as AF control (automatic focus adjustment) and AE control (automatic exposure control), and when the S2 on signal is output, it executes still image shooting and recording processes.

[0072] It goes without saying that AF and AE are performed automatically when the auto mode is set by the operation unit 222, and that AF and AE are not performed when the manual mode is set.

[0073] Also, in the case of a video shooting mode (first video shooting mode for normal video or second video shooting mode for extracting still images), when the shutter release switch 22 is fully pressed and an S2 on signal is output, the camera body 200 enters a video recording mode in which video recording begins, and performs image processing and recording processing of the video. After that, when the shutter release switch 22 is fully pressed again and an S2 on signal is output, the camera body 200 enters a standby state and temporarily suspends the video recording processing.

[0074] The shutter release switch 22 is not limited to a two-stroke switch consisting of a half-press and a full-press, and may output an S1 ON signal and an S2 ON signal with a single operation. Alternatively, separate switches may be provided to output an S1 ON signal and an S2 ON signal.

[0075] In addition, in a form in which operation instructions are given using a touch panel or the like, the operation instructions may be output by touching an area corresponding to the operation instructions displayed on the screen of the touch panel as the operation means, and the form of the operation means is not limited to these as long as it instructs the shooting preparation process or the shooting process.

[0076] The still or video images obtained by shooting are compressed by the compression / decompression processing unit 208, and the compressed image data is converted into an image file with the necessary additional information such as the shooting date and time, GPS information, and shooting conditions (F-number, shutter speed, ISO sensitivity, etc.) added to the header, and then stored on the memory card 212 via the media control unit 210.

[0077] The body-side CPU 220 controls the overall operation of the camera body 200 and the driving of the optical components of the interchangeable lens 100, and controls each part of the camera body 200 and the interchangeable lens 100 based on input from an operating unit 222 including the shutter release switch 22.

[0078] The clock unit 224 functions as a timer and measures time based on commands from the main body CPU 220. The clock unit 224 also functions as a calendar and measures the current date and time.

[0079] The flash ROM 226 is a readable and writable non-volatile memory that stores setting information.

[0080] The ROM 228 stores a camera control program executed by the body CPU 220, an image processing program according to the present invention, defect information on the image sensor 201, and various parameters and tables used for image processing, etc. The body CPU 220 controls each part of the camera body 200 and the interchangeable lens 100 in accordance with the camera control program or image processing program stored in ROM 228, using RAM 207 as a working area.

[0081] The AF control unit 230, which functions as an automatic focus adjustment unit, calculates the defocus amount required to control the phase difference AF when the image sensor 201 includes phase difference pixels, and based on the calculated defocus amount, notifies the interchangeable lens 100 of a command for the position (focus position) to which the focus lens should move via the body-side CPU 220 and the body-side communication unit 250.

[0082] A focus lens position command corresponding to the defocus amount calculated by the AF control unit 230 is notified to the interchangeable lens 100, and the lens-side CPU 120 of the interchangeable lens 100 that has received the focus lens position command moves the focus lens via the focus lens control unit 116, and controls the position of the focus lens (focus position). Note that the AF control unit 230 is not limited to one that performs phase difference AF, and may also be one that performs contrast AF, which moves the focus lens so that the contrast in the AF area is maximized.

[0083] The AE control unit 232 detects the brightness of the subject (subject luminance) and calculates a numerical value (exposure value (EV)) required for AE control and AWB (Auto White Balance) control corresponding to the subject luminance. The AE control unit 232 calculates the EV based on the luminance of the image acquired via the image sensor 201, and the shutter speed and F-number when the luminance of the image was acquired.

[0084] The main body CPU 220 can determine the F-number, shutter speed, and ISO sensitivity from a predetermined program diagram based on the EV value obtained from the AE control unit 232, and perform AE control.

[0085] The white balance correction unit 234 calculates white balance gains (WB (White Balance) gains) Gr, Gg, and Gb for each color data of RGB data (R data, G data, and B data). The WB gains Gr, Gg, and Gb are calculated for the R data, G data, and B data, respectively. Here, the white balance correction is performed by multiplying the WB gains Gr, Gg, and Gb. One possible calculation method is to identify the type of light source illuminating the subject based on scene recognition (e.g., determination of indoors or outdoors) using the brightness (EV value) of the subject and the color temperature of the ambient light, and then read out the WB gain corresponding to the identified light source type from a memory unit in which appropriate WB gains for each light source type are stored in advance. However, it is necessary to determine the WB gains Gr, Gg, and Gb using at least the EV value. Other known methods are contemplated.

[0086] The wireless communication unit 236 is compatible with Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (registered trademark), This is the part that performs short-range wireless communication according to standards such as the IEEE 802.11b standard, and sends and receives necessary information between surrounding digital devices (smartphones, etc.).

[0087] The GPS receiving unit 238 receives GPS signals transmitted from a plurality of GPS satellites in accordance with instructions from the main body side CPU 220, and performs positioning calculation processing based on the received plurality of GPS signals. This executes the GPS command to acquire GPS information consisting of the latitude, longitude, and altitude of the camera body 200. The acquired GPS information can be recorded in the header of the image file as additional information indicating the shooting location of the captured image.

[0088] In accordance with commands from the body-side CPU 220, the power supply control unit 240 supplies the power supply voltage supplied from the battery 242 to each unit of the camera body 200. In addition, in accordance with commands from the body-side CPU 220, the power supply control unit 240 supplies the power supply voltage supplied from the battery 242 to each unit of the interchangeable lens 100 via the body mount 260 and the lens mount 160.

[0089] The lens power switch 244 switches on and off the power supply voltage and changes the level of the power supply voltage applied to the interchangeable lens 100 via the body mount 260 and the lens mount 160 in accordance with commands from the body-side CPU 220 .

[0090] In accordance with commands from the body-side CPU 220, the body-side communication unit 250 transmits and receives (bidirectional communication) request signals and response signals to and from the lens-side communication unit 150 of the interchangeable lens 100 connected via the body mount 260 and the lens mount 160. Note that the body mount 260 is provided with a plurality of terminals 260A as shown in Fig. 1, and when the interchangeable lens 100 is attached to the camera body 200 (the lens mount 160 and the body mount 260 are connected), the plurality of terminals 260A (Fig. 1) provided on the body mount 260 are electrically connected to a plurality of terminals (not shown) provided on the lens mount 160, enabling bidirectional communication between the body-side communication unit 250 and the lens-side communication unit 150.

[0091] The built-in flash 30 (Fig. 1) is, for example, a TTL (Through The Lens) automatic light control type. The flash is composed of a flash light emitting unit 270 and a flash control unit 272 .

[0092] The flash control unit 272 has a function of adjusting the light emission amount (guide number) of the flash light emitted from the flash emission unit 270. That is, the flash control unit 272 causes the flash emission unit 270 to pre-emit (control light emission) a flash light with a small light emission amount in synchronization with a flash photography instruction from the body-side CPU 220, determines the light emission amount of the flash light to be emitted in the main emission based on reflected light (including ambient light) incident via the photographing optical system 102 of the interchangeable lens 100, and causes the flash emission unit 270 to emit the flash light of the determined light emission amount (main emission).

[0093] The FPS 280 constitutes a mechanical shutter of the photographing device 10, and is disposed immediately before the image sensor 201. The FPS control unit 296 controls the opening and closing of the front and rear curtains of the FPS 280 based on input information (S2 on signal, shutter speed, etc.) from the main body side CPU 220, and controls the exposure time (shutter speed) of the image sensor 201.

[0094] Next, the compression / decompression processing unit 208 that compresses the moving image captured in the first moving image capture mode or the second moving image capture mode when the first moving image capture mode or the second moving image capture mode is set will be described.

[0095] [First embodiment] FIG. 4 is a block diagram showing a first embodiment of an image processing device according to the present invention, and particularly shows the compression processing section of the compression / decompression processing section 208 in the camera body 200.

[0096] 4 is mainly composed of a video acquisition unit 302, an orthogonal transformer 310, a quantization unit 320, an encoding unit 330, and a bit rate control unit 340. The compression processing unit 208-1 in this example complies with H.264 / AV, which is one of the MPEG encoding methods. Compression shall be performed using method C.

[0097] In the MPEG compression method, compression and editing are performed in units of 1 GOP (Group Of Pictures), which is a collection of several frames (for example, 15 frames) of moving images. It includes I (Intra) frames, which compress only the information of a frame and do not use correlation information with other frames that precede and follow it in time, P (Predictive) frames, which are represented by correlation information from past frames in time, and B (Bidirectionally) frames, which are represented by correlation information from previous and subsequent frames in time, and the first frame of one GOP is at least an I frame.

[0098] The compression processing unit 208-1 shown in FIG. 4 is a part that compresses normal moving images captured by the moving image capturing unit of the image capturing device 10 (interchangeable lens 100 and image sensor 201 of camera body 200, etc.) or moving images 300 for extracting still images.

[0099] The video acquisition unit 302 of the compression processing unit 208-1 is a part that acquires image data of frames that make up the video 300 captured by the video shooting unit, and in this example, for simplicity of explanation, it is assumed that the I frame, P frame, and B frame that make up one GOP are acquired sequentially.

[0100] Each frame constituting one GOP is coded in units of a macroblock of 16 x 16 pixels. The luminance data Y and chrominance data Cb, Cr of one macroblock are converted into four luminance data Y blocks of 8 x 8 pixels in a Y:Cr:Cb=4:1:1 format and one chrominance data Cr, Cb block each thinned to 8 x 8 pixels, and then quantized for each block (unit block).

[0101] The orthogonal transformer 310 performs an orthogonal transform on the data of the 8×8 pixel unit block in accordance with a method called a Discrete Cosine Transform (DCT) to decompose the data into frequency components. and calculates the orthogonal transform coefficients.

[0102] The quantization unit 320 quantizes the orthogonal transform coefficients obtained by the orthogonal transformer 310 based on the quantization parameter (QP value) determined (set) by the bit rate control unit 340.

[0103] In H.264 / AVC, the QP value is specified in the range of 1 to 51. When a QP value is determined within this range, the quantization step size (Q step ) is decided. Q step is a value by which the orthogonal transform coefficients used in the quantization process are divided. In H.264 / AVC, this value doubles when the QP value increases by 6, and can be derived using a lookup table based on the determined QP value, or by calculation.

[0104] The quality and bitrate of the compressed bitstream are primarily determined by the QP value chosen to quantize each macroblock. stepis a value that controls how much spatial detail is preserved in the compressed macroblock.

[0105] Q step The smaller the Q, the more detail is preserved and the better the image quality, but the higher the bitrate. step As QP increases, less detail is preserved, and the bit rate is reduced, but the image quality deteriorates. Therefore, the bit rate control unit 340 determines the QP value (Q step ) must be determined.

[0106] The method by which the bit rate control unit 340 determines the QP value will be described later.

[0107] The encoding unit 330 is a part that entropy-encodes the quantized values ​​supplied from the quantization unit 320, and in H.264 / AVC, it uses variable length coding (VLC; You can choose between Variable Length Coding (VLC) and arithmetic coding. The compressed data 350 (encoded data) further compressed by the unit 330 is sent as a bit stream to the media control unit 210 (FIG. 3). The media control unit 210 also functions as a moving image recording unit that records the compressed data of the normal moving image and the moving image for extracting still images onto the memory card 212.

[0108] The bitrate control unit 340 functions as a VBV (Video Buffering Verifier) ​​buffer, and acquires, for example, in units of macroblocks, coded data (amount of generated code) after quantization of image data of past frames of moving images output from the coding unit 330. The bitrate control unit 340 calculates the VBV buffer occupancy from the acquired amount of generated code and a preset bitrate of the bitstream, and determines a QP value that will not cause the VBV buffer to fail. The bitrate control unit 340 outputs the determined QP value to the quantization unit 320.

[0109] Instead of the determined QP value, the bit rate control unit 340 determines the quantization step size (Q step ) to the quantization unit 320. The bit rate control unit 340 may also determine the QP value on a frame-by-frame or GOP-by-GOP basis.

[0110] The quantization unit 320 calculates the QP value corresponding to the QP value input from the bit rate control unit 340. step or directly from the bit rate control unit 340 step and the orthogonal transform coefficients are obtained as Q step The quantized value is calculated by dividing by 1 and rounding to an integer.

[0111] Next, the method by which the bit rate control unit 340 determines the QP value will be described in more detail.

[0112] As shown in FIG. 4, a shooting mode command indicating a first moving image shooting mode for normal moving images or a second moving image shooting mode for extracting still images is sent from a shooting mode selection unit 360 to the bit rate control unit 340.

[0113] In this example, the shooting mode selection unit 360 is an on-screen interactive operation unit using the MENU / OK key 27, the cross key 28, the LCD monitor 216, etc., but it may also be a mode dial for selecting various shooting modes.

[0114] When the first video shooting mode is selected by the shooting mode selection unit 360 and an instruction to shoot a video is input from the shutter release switch 22, the photographing device 10 shoots a video (normal video) under shooting conditions suitable for viewing the video, while when the second video shooting mode is selected by the shooting mode selection unit 360 and an instruction to shoot a video is input from the shutter release switch 22, the photographing device 10 shoots a video under shooting conditions suitable for extracting still images (video for extracting still images).

[0115] Here, the shooting conditions for the moving image for still image extraction and the shooting conditions for the normal moving image differ, for example, in shutter speed, and the shutter speed for each frame of the moving image for still image extraction is set higher than that of the normal moving image.

[0116] This is because, in the case of a video for extracting still images, it is preferable to set the shutter speed so that camera shake or image blur does not occur in each frame. On the other hand, in the case of a normal video, it is preferable to set the shutter speed to a value corresponding to the frame interval determined by the frame rate in order to ensure the continuity of each frame. Also, in the case of a video for extracting still images, it is preferable to make the tracking speeds of AF, AE, and AWB as fast as possible, while in the case of a normal video, it is preferable to make the tracking speeds of AF, AE, and AWB slow. Also, in the case of a video for extracting still images, it is preferable to make the tracking speeds of AF, AE, and AWB as slow as possible in order to capture a momentary scene. To capture the image, it is preferable to increase the frame rate (the number of frames per unit time) compared to normal moving images.

[0117] Now, the bit rate control unit 340 sets different ranges of QP values ​​(lower and upper limits) when a shooting mode command indicating the first moving image shooting mode is issued from the shooting mode selection unit 360 and when a shooting mode command indicating the second moving image shooting mode is issued.

[0118] FIG. 5 is a schematic diagram showing the relationship between the QP value and the change in the amount of generated code after quantization of image data of past frames when performing normal compression processing of a moving image.

[0119] As shown in Fig. 5, the bitrate control unit 340 determines the QP value to be used for quantization of normal video within a range (first range) between a first lower limit value (Min. 1) and a first upper limit value (Max. 1). In the case of H.264 / AVC, the maximum possible range of the QP value is 1 to 51, so Min. 1 and Max.1 are set within the range of 1 to 51. Min.1 and Max.1 are set taking into consideration the image quality of the entire video. It is preferable to set it with consideration.

[0120] As shown in Figure 5, when the QP value is set to Min.1, the scene changes. When a frame or GOP of a scene with a high bit rate is quantized, the amount of generated code after quantization increases rapidly. In this case, the bit rate control unit 340 increases the QP value so that the VBV buffer does not collapse (the VBV buffer occupancy does not overflow).

[0121] In the example shown in FIG. 5, when the amount of generated code increases suddenly, the QP value is changed from Min. 1 to Max. 1.

[0122] Then, when the scene changes from a moving one to a still one, the QP value is kept at Max. The bit rate control unit 340 reduces the QP value so that the VBV buffer does not collapse (the VBV buffer occupancy does not underflow) due to the decrease in the amount of generated code.

[0123] Min.1 and Max.1 for normal videos are set taking into account the image quality of the entire video, so they are suitable for viewing as a video, but when extracting a single frame as a still image from a normal video, the QP value for the extracted still image may be too high.

[0124] FIG. 6 is a schematic diagram showing the relationship between the QP value and the change in the amount of generated code after quantization of image data of past frames when compressing a moving image for extracting still images.

[0125] As shown in FIG. 6, the bit rate control unit 340 determines the QP value to be used for quantizing the moving image for extracting a still image within the range (second range) between a second lower limit value (Min. 2) and a second upper limit value (Max. 2).

[0126] Min.2 and Max.2 are set to ensure a certain image quality (the image quality required for still images) for all frames of the video used to extract still images, and to have the same bit rate as for normal video.

[0127] Specifically, Min.2 should be set to a value greater than Min.1. Min.2 is used for scene changes. When quantizing frames or GOPs of scenes with a lot of movement, the amount of generated code after quantization is set so as not to increase significantly.

[0128] Also, by setting Min.2 larger than Min.1, the time required for the calculation is shorter than when Min.1 is set. It is possible to suppress an increase in the amount of generated code after quantization.

[0129] On the other hand, Max.2 is set to a value smaller than Max.1. Max.2 is the QP value used for quantization. Even if Max.2 is used for a frame, the frame can still retain the desired image quality as a still image. Set it to the value you want.

[0130] As shown in Figure 6, when the QP value is set to Min.2, which is larger than Min.1, the amount of code generated after quantization is reduced in still scenes, etc., compared to normal video. As a result, even when quantizing frames or GOPs in moving scenes such as scene changes, the amount of code generated can be reduced so as not to increase suddenly.

[0131] In addition, in the example shown in Figure 6, when the amount of generated code increases, the QP value is changed from Min. 2 to Max. 2. However, since the increase in the amount of generated code is suppressed, it is possible to prevent the VBV buffer from collapsing (the VBV buffer occupancy from overflowing) even when Max. 2 is smaller than Max. 1.

[0132] In particular, since the QP value can be limited to a maximum of Max. 2, which is smaller than Max. 1, the image quality required for still images can be guaranteed for all frames of the video used to extract still images.

[0133] [Second embodiment] Fig. 7 is a block diagram showing a second embodiment of an image processing device according to the present invention, and particularly shows the compression processing section of the compression / decompression processing section 208 in the camera body 200. In the second embodiment shown in Fig. 7, parts that are common to the first embodiment shown in Fig. 4 are given the same reference numerals, and detailed description thereof will be omitted.

[0134] The compression processing unit 208-2 shown in FIG. 7 differs from the compression processing unit 208-1 of the first embodiment shown in FIG. 4 mainly in that it has a scene determination unit that determines the scene of the moving image for extracting a still image.

[0135] The scene determination unit is a part that determines the shooting scene based on scene information from scene selection unit 370, which selects shooting scenes such as landscape, night view, sunset, macro, person, moving object, sports, etc., and bit rate control unit 340 may function as the scene determination unit, or a scene determination unit may be provided that is independent of bit rate control unit 340. Furthermore, the scene determination unit may analyze a live view image and automatically determine the shooting scene based on the analysis results.

[0136] When the second video shooting mode for extracting still images is set, the bit rate control unit 340 sets a range of selectable QP values ​​(second range) according to scene information indicating the video shooting scene determined by the scene determination unit. For example, the second range of QP values ​​defined by Min.2 and Max.2 can be set by reading Min.2 and Max.2 recorded corresponding to the scene information from a table in which the relationship between the scene information and Min.2 and Max.2 is recorded according to the scene information.

[0137] It goes without saying that Min. 2, which is set according to the shooting scene of the video, is larger than Min. 1 of the QP value for normal video, and Max. 2 is smaller than Max. 1 of the QP value for normal video. Furthermore, when set to the first video shooting mode for normal video, the bit rate control unit 340 may set a range of selectable QP values ​​(first range) according to scene information indicating the shooting scene of the video determined by the scene determination unit.

[0138] The range of the QP value set by the scene determination and the scene determination result is, for example, The QP value range may be determined by determining whether the scene is an "instantaneous change scene" based on the results of the determination from the video, such as whether there is a moving object or not, and then determining whether the scene is an "instantaneous change scene." Furthermore, a "instant change cut-out mode" and a "normal cut-out mode (a mode other than the instantaneous change cut-out mode)" may be provided as second video shooting modes for extracting still images, and when the user selects the "instant change cut-out mode," the QP value range (second range) may be narrower than that of the "normal cut-out mode."

[0139] In addition, in the second moving image shooting mode, the second range of the QP value may be shifted in accordance with the frame rate (the QP value may be shifted higher as the frame rate increases).

[0140] The following Table 1 shows an example of the lower and upper limits of the QP value.

[0141] [Table 1]

[0142] [Scene detection] An example of a scene where the scene changes suddenly is when shooting lightning.

[0143] When a video continues with frames of the same composition, the QP value gradually decreases for frames of the same composition, and the image compression rate decreases. In this case, if the screen changes suddenly, there is a concern that the compression rate will not be achieved and the QP value will increase.

[0144] Therefore, it is preferable to provide an "instant change cut-out mode" as a second moving image capture mode for extracting still images, and to further narrow the range of QP values ​​compared to the "normal cut-out mode."

[0145] [About frame rate] When the frame rate changes from 30 fps to 60 fps, the amount of data simply doubles.

[0146] If the frame rate increases, the QP value range is shifted accordingly (both the lower and upper limits). This makes it possible to achieve the same bit rate after compression for 30fps and 60fps, and suppresses degradation of image quality even when the frame rate changes from 30fps to 60fps.

[0147] Furthermore, the second range of the QP value in the second video shooting mode may be set appropriately in accordance with input from an instruction input unit that inputs instructions from outside, rather than being limited to the upper and lower limit values ​​of the QP value shown in Table 1.

[0148] [Image processing method] 8 is a flowchart showing an embodiment of an image processing method according to the present invention. The processing operations of each unit of compression processing unit 208-1 shown in FIG.

[0149] In FIG. 8, the bit rate control unit 340 receives the shooting mode input from the shooting mode selection unit 360. It is determined whether the first moving image shooting mode for normal moving images or the second moving image shooting mode for extracting still images is selected in response to the shadow mode command (step S10).

[0150] If the first moving image shooting mode is selected, a first range (first lower limit Min.1, first upper limit Max.1) of quantization parameters (QP values) for normal moving images is set (step S12), and if the second moving image shooting mode is selected, a second range (second lower limit Min.2, second upper limit Max.2) of QP values ​​for still image extraction is set (step S13). Note that Min.2 is a value larger than Min.1, and Max.2 is a value smaller than Max.1. Step S14 may also include a step of determining a scene of the moving image for still image extraction by a scene determination unit, and the second range may be set according to the determined scene.

[0151] Next, the main body CPU 220 of the photographing device 10 determines whether or not moving image photographing has started based on an input signal from the operation unit 222 (shutter release switch 22) (step S14).

[0152] When it is determined in step S14 that moving image shooting has started, the main body CPU 220 causes the moving image shooting section to shoot moving images in the first moving image shooting mode or the second moving image shooting mode (step S15, moving image acquisition step).

[0153] Next, the moving image captured by the moving image capturing unit is compressed in units of 1 GOP using the H.264 / AVC method, which is one of the MPEG encoding methods (compression processing step).

[0154] That is, when video capture begins, the video acquisition unit 302 of the compression processing unit 208-1 sequentially acquires each frame (I frames, P frames, and B frames that make up 1 GOP) of a normal video or a video for still image extraction (step S16).

[0155] Each frame is compressed for each unit block of 8 x 8 pixels. The orthogonal transformer 310 performs a discrete cosine transform (DCT) on the data of the unit block to calculate orthogonal transform coefficients (step S18).

[0156] The bit rate control unit 340, which functions as a VBV buffer, acquires, for example, in units of macroblocks, the coded data (the amount of code generated after quantization of image data of past frames of a moving image) output from the coding unit 330 (step S20). The bit rate control unit 340 calculates the VBV buffer occupancy from the acquired amount of code generated and a preset bit rate of the bit stream, and determines a quantization parameter (QP value) that will prevent the VBV buffer from failing (step S22).

[0157] Here, in the determination of the QP value in step S22, if the video currently being compressed is a normal video, the QP value is determined within the first range (first lower limit Min.1, first upper limit Min.2) set in step S12. In the case of a video for still image extraction, the QP value is determined within the range of 1 (Max. 1). The QP value is determined within the second range (second lower limit Min.2, second upper limit Max.2).

[0158] The quantization unit 320 converts the orthogonal transform coefficients input from the orthogonal transformer 310 into a quantization step size (QP) corresponding to the QP value determined by the bit rate control unit 340. step ) and calculates a quantized value rounded to an integer (step S24).

[0159] Steps S18 to S24, in which a QP value is determined within the first range set in step S12 and the video frames are quantized and compressed for each unit block, correspond to a first compression step for compressing normal video, and steps S18 to S24, in which a QP value is determined within the second range set in step S13 and the video frames are quantized and compressed for each unit block, correspond to a second compression step for compressing video for still image extraction.

[0160] The quantized values ​​calculated by the quantization unit 320 are entropy coded by the coding unit 330 and output as a bitstream to the media control unit 210 (Figure 3), which then records the bitstream on the memory card 212 as a normal video or a video for still image extraction by the media control unit 210 (step S26).

[0161] Next, the main body CPU 220 of the photographing device 10 determines whether or not video shooting has ended based on an input signal from the operation unit 222 (step S28). If it is determined that video shooting has not ended (if "No"), the process proceeds to step S15. This allows video shooting, compression processing, recording processing, etc. to continue. If it is determined that video shooting has ended (if "Yes"), the image processing ends.

[0162] The photographing device 10 of this embodiment is a mirrorless digital single-lens camera, but is not limited to this and may be a single-lens reflex camera, a lens-integrated photographing device, a digital video camera, etc. It can also be applied to mobile devices that have functions other than photographing (call functions, communication functions, other computer functions) in addition to a photographing function. Other embodiments to which the present invention can be applied include, for example, mobile phones and smartphones with camera functions, PDAs (Personal Digital Assistants), and portable game consoles. An example of a smartphone to which the present invention can be applied is described below.

[0163] <Smartphone configuration> Fig. 9 shows the appearance of a smartphone 500, which is one embodiment of the imaging device of the present invention. The smartphone 500 shown in Fig. 9 has a flat housing 502, and is provided with a display input unit 520, which is an integrated unit of a display panel 521 as a display unit and an operation panel 522 as an input unit, on one side of the housing 502. The housing 502 also includes a speaker 531, a microphone 532, an operation unit 540, and a camera unit 541. Note that the configuration of the housing 502 is not limited to this, and for example, a configuration in which the display unit and the input unit are independent, or a configuration having a foldable structure or a sliding mechanism can also be used.

[0164] Fig. 10 is a block diagram showing the configuration of the smartphone 500 shown in Fig. 9. As shown in Fig. 10, the main components of the smartphone include a wireless communication unit 510 that performs mobile wireless communication via a base station and a mobile communication network, a display input unit 520, a call unit 530, an operation unit 540, a camera unit 541, a recording unit 550, an external input / output unit 560, a GPS (Global Positioning System) receiving unit 570, a motion sensor unit 580, a power supply unit 590, and It includes a main control unit 501.

[0165] The wireless communication unit 510 performs wireless communication with a base station accommodated in the mobile communication network in accordance with instructions from the main control unit 501. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0166] The display input unit 520 is a so-called touch panel that, under the control of the main control unit 501, displays images (still images and moving images), text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 521 and an operation panel 522.

[0167] The display panel 521 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 522 is placed so that an image displayed on the display surface of the display panel 521 can be seen, and is a device that detects one or more coordinates operated by a user's finger or a stylus. When such a device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 501. Next, the main control unit 501 detects the operation position (coordinates) on the display panel 521 based on the received detection signal.

[0168] 9, display panel 521 and operation panel 522 of smartphone 500, exemplified as an embodiment of the imaging device of the present invention, are integrated to form display input unit 520, with operation panel 522 being arranged so as to completely cover display panel 521. When such an arrangement is adopted, operation panel 522 may also have a function for detecting user operations in an area outside display panel 521. In other words, operation panel 522 may have a detection area for the overlapping portion that overlaps display panel 521 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 521 (hereinafter referred to as a non-display area).

[0169] The size of the display area and the size of the display panel 521 may be completely the same, but they do not necessarily have to be the same. Also, the operation panel 522 may have two sensitive areas, one on the outer edge and the other on the inner side. Furthermore, the width of the outer edge is designed appropriately depending on the size of the housing 502, etc. Also, the position detection method used in the operation panel 522 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, etc., and any method can be used.

[0170] The call unit 530 includes a speaker 531 and a microphone 532, and converts the user's voice input through the microphone 532 into voice data that can be processed by the main control unit 501 and outputs the voice data to the main control unit 501, or decodes voice data received by the wireless communication unit 510 or the external input / output unit 560 and outputs the decoded voice data from the speaker 531. Also, as shown in Fig. 9, for example, the speaker 531 and the microphone 532 can be mounted on the same surface on which the display input unit 520 is provided.

[0171] The operation unit 540 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 9, the operation unit 540 is a push-button switch mounted on the side of the housing 502 of the smartphone 500, which turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0172] The recording unit 550 stores the control program of the main control unit 501, control data, application software (including the image processing program according to the present invention), address data associating names and telephone numbers of communication partners, data of emails sent and received, web data downloaded by web browsing, and downloaded content data, and also temporarily stores streaming data. The recording unit 550 is composed of an internal storage unit 551 built into the smartphone and an external storage unit 562 having a removable external memory slot. The internal storage unit 551 and the external storage unit 552 constituting the recording unit 550 may be flash memory type, hard disk drive type, or the like. It is realized using recording media such as a hard disk type, a multimedia card micro type, a card-type memory (e.g., Micro SD (registered trademark) memory, etc.), RAM (Random Access Memory), and ROM (Read Only Memory).

[0173] The external input / output unit 560 serves as an interface with all external devices connected to the smartphone 500, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE1394, etc.) or a network (e.g., the Internet, a wireless LAN (Local Area Network), Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.).

[0174] Examples of external devices connected to the smartphone 500 include a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) card, or an audio / video I / O (Input / Output) terminal. The external input / output unit may transmit data received from such external devices to various components within the smartphone 500, or transmit data within the smartphone 500 to the external devices.

[0175] The GPS receiver 570 receives GPS signals transmitted from GPS satellites ST1 to STn in accordance with instructions from the main controller 501, executes positioning calculation processing based on the received GPS signals, and detects the position consisting of the latitude, longitude, and altitude of the smartphone 500. When the GPS receiver 570 can acquire position information from the wireless communication unit 510 or the external input / output unit 560 (for example, a wireless LAN), it can also detect the position using the position information.

[0176] The motion sensor unit 580 includes, for example, a three-axis acceleration sensor and a gyro sensor, and detects the physical movement of the smartphone 500 in accordance with instructions from the main control unit 501. By detecting the physical movement of the smartphone 500, the direction of movement and acceleration of the smartphone 500 are detected. The detection results are output to the main control unit 501.

[0177] The power supply unit 590 supplies power stored in a battery (not shown) to each unit of the smartphone 500 in accordance with instructions from the main control unit 501.

[0178] The main control unit 501 includes a microprocessor, operates according to the control program and control data stored in the recording unit 550, and controls all the units of the smartphone 500. The main control unit 501 also includes a mobile communication control function and an application processing function that control all the units of the communication system in order to perform voice communication and data communication via the wireless communication unit 510.

[0179] The application processing function is realized by the main control unit 501 operating in accordance with the application software stored in the recording unit 550. Examples of the application processing function include an infrared communication function that controls the external input / output unit 560 to perform data communication with a connected device, an email function that sends and receives emails, a web browsing function that views web pages, and an image processing function that performs compression processing according to the present invention.

[0180] The main control unit 501 also has an image processing function for displaying video on the display input unit 520 based on image data (still image or video data) such as received data or downloaded streaming data. The image processing function refers to a function in which the main control unit 501 decodes the image data, performs image processing on the decoding result, and displays the image on the display input unit 520.

[0181] Furthermore, the main control unit 501 controls the display of the display panel 521, and controls the operation unit 540 and the operation An operation detection control for detecting a user operation through the panel 522 is executed.

[0182] By executing display control, main control unit 501 displays software keys such as icons and scroll bars for launching application software, or displays a window for creating an e-mail. Note that a scroll bar is a software key that accepts an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​display panel 521.

[0183] In addition, by executing operation detection control, the main control unit 501 detects user operations through the operation unit 540, accepts operations on icons through the operation panel 522 and input of character strings into window input fields, or accepts requests to scroll the displayed image through a scroll bar.

[0184] Furthermore, by executing operation detection control, the main control unit 501 determines whether the operation position on the operation panel 522 is an overlapping portion (display area) that overlaps the display panel 521 or an outer edge portion (non-display area) that does not overlap the display panel 521, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 522 and the display position of the software keys.

[0185] The main control unit 501 can also detect gesture operations on the operation panel 522 and execute a preset function in accordance with the detected gesture operation. A gesture operation is not a conventional simple touch operation, but refers to an operation of drawing a path with a finger or the like, specifying multiple positions simultaneously, or combining these to draw a path for at least one of multiple positions.

[0186] The camera unit 541 is a CMOS (Complementary Metal Oxide Semiconductor) or CCD It is a digital camera that takes pictures electronically using an image sensor such as a Charge-Coupled Device (CCD). , which corresponds to the photographing device 10 shown in FIG. 1 . Under the control of the main control unit 501, the camera unit 541 can compress image data of a still image obtained by photographing, for example, using JPEG (Joint Photographic coding Experts Group), or compress image data of a moving image, for example, using H.264 / AVC, and record the compressed data in the recording unit 550, or output the compressed data via the external input / output unit 560 or the wireless communication unit 510. As shown in FIG. 9 , in the smartphone 500, the camera unit 541 is mounted on the same surface as the display input unit 520, but the mounting position of the camera unit 541 is not limited thereto, and the camera unit 541 may be mounted on the back surface of the display input unit 520, or multiple camera units 541 may be mounted. Note that when multiple camera units 541 are mounted, the camera unit 541 used for photographing can be switched to photograph a single camera unit, or multiple camera units 541 can be used simultaneously to photograph.

[0187] Furthermore, the camera unit 541 can be used for various functions of the smartphone 500. For example, an image acquired by the camera unit 541 can be displayed on the display panel 521, or an image from the camera unit 541 can be used as one of the operation inputs for the operation panel 522. Furthermore, when the GPS receiver 570 detects a position, the position can be detected by referring to an image from the camera unit 541. Furthermore, by referring to the image from the camera unit 541, it is possible to determine the direction of the optical axis of the camera unit 541 of the smartphone 500 or to determine the current usage environment without using a three-axis acceleration sensor or by using a three-axis acceleration sensor (gyro sensor) in combination. Of course, the image from the camera unit 541 can also be used in application software.

[0188] In addition, the image data of a still image or video can be added with location information acquired by the GPS receiving unit 570, audio information acquired by the microphone 532 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 580, etc., and recorded in the recording unit 550, or output via the external input / output unit 560 or wireless communication unit 510.

[0189] [others] In this embodiment, the H.264 / AVC encoding method has been described as an example, but the present invention is not limited to this and can also be applied to compression using other encoding methods such as MPEG-2 and MPEG-4. For example, in MPEG-2, the QP value is defined as a range of 1 to 31. Therefore, when setting different QP value ranges (lower and upper limits) for normal video and video for still image extraction, the setting must be made according to the encoding method.

[0190] The hardware structure of the processing unit that executes various processes in the compression / decompression processing unit 208 of the imaging device according to the present invention is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0191] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0192] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0193] Furthermore, the present invention includes an image processing program that, when installed in an image capture device, causes the image capture device to function as the image capture device of the present invention, and a recording medium on which this image processing program is recorded.

[0194] Furthermore, it goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0195] 10 Imaging equipment 20 Viewfinder window 22 Shutter release switch 23 Shutter speed dial 24 Exposure compensation dial 25 Power Lever 26 Eyepiece 27 MENU / OK key 28 D-pad 29 Play button 30 Built-in flash 100 interchangeable lenses 102 Photographing optical system 104 lens group 108 aperture 116 Focus lens control unit 118 Aperture control section 120 Lens side CPU 122 RAM 124 ROM 126 Flash ROM 150 Lens communication unit 160 lens mount 200 camera body 201 Image Sensor 202 Image sensor control unit 203 Analog signal processing section 204 A / D converter 205 Image Input Controller 206 Digital Signal Processing Unit 207 RAM 208 Compression / Expansion Processing Unit 208-1, 208-2 Compression processing unit 210 Media Control Unit 212 Memory Card 214 Display control unit 216 LCD monitor 220 Main unit CPU 222 Operation section 224 Clock Section 226 Flash ROM 228 ROM 230 AF control unit 232 AE control section 234 White balance correction section 236 Radio Communication Department 238 GPS receiver 240 Power supply control unit 242 Battery 244 Lens power switch 250 Main unit communication unit 260 Body Mount 260A terminal 270 Flash unit 272 Flash control unit 296 FPS control unit 300 videos 302 Video Acquisition Unit 310 Quadrature Transformer 320 Quantization section 330 Encoding section 340 Bitrate control section 350 compressed data 360 shooting mode selection section 370 Scene Selection 500 smartphones 501 Main control unit 502 Case 510 Wireless Communication Department 520 Display and input section 521 Display Panel 522 Operation Panel 530 Telephone section 531 Speaker 532 Microphone 540 Operation section 541 Camera Club 550 Recording Department 551 Internal storage 552 External Memory Unit 560 External input / output section 562 External Memory Unit 570 GPS receiver 570 Receiving Unit 580 Motion Sensor Unit 590 Power supply section Max.1 First upper limit Max.2 Second upper limit Min.1 First lower limit Min.2 Second lower limit S10~S28 steps

Claims

1. a processor; The processor: acquiring a video captured in a first video capture mode or a second video capture mode having different capture conditions from the first video capture mode; executes image processing to compress image data of frames constituting the moving image based on a quantization parameter; an upper limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is smaller than an upper limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode, at the same frame rate; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode.

1. An image processing device comprising:

2. The same frame rate is a predetermined frame rate setting value.

2. The image processing device according to claim 1, wherein:

3. an upper limit of the second quantization parameter varies depending on the setting value; 3. The image processing device according to claim 2.

4. the upper limit of the second quantization parameter is changed in response to an input from an instruction input unit that inputs an instruction from outside.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. the second moving image shooting mode is a moving image shooting mode in which a moving image is shot under shooting conditions suitable for extracting a still image; 5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. The processor: varying the upper limit of the second quantization parameter depending on the scene of the moving image; 6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. a processor; The processor: acquiring a video captured in a first video capture mode or a second video capture mode having different capture conditions from the first video capture mode; executes image processing to compress image data of frames constituting the moving image based on a quantization parameter; at the same frame rate, a lower limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is greater than a lower limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode.

1. An image processing device comprising:

8. The same frame rate is a predetermined frame rate setting value.

8. The image processing device according to claim 7,

9. the lower limit of the second quantization parameter varies depending on the setting value.

9. The image processing device according to claim 8,

10. the lower limit of the second quantization parameter is changed in response to an input from an instruction input unit that inputs an instruction from outside.

9. The image processing device according to claim 7, wherein:

11. the second moving image shooting mode is a moving image shooting mode in which a moving image is shot under shooting conditions suitable for extracting a still image; 11. The image processing device according to claim 7, wherein the image processing device is a computer.

12. The processor: varying the lower limit of the second quantization parameter depending on the scene of the moving image; 12. The image processing device according to claim 7, wherein the image processing device is a computer.

13. a function of acquiring a video captured in a first video capture mode or a second video capture mode under different capture conditions from the first video capture mode; a function of performing image processing to compress image data of frames that make up the moving image based on a quantization parameter; An image processing program that causes a computer to realize the above, an upper limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is smaller than an upper limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode, at the same frame rate; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode.

1. An image processing program comprising:

14. a function of acquiring a video captured in a first video capture mode or a second video capture mode under different capture conditions from the first video capture mode; a function of performing image processing to compress image data of frames that make up the moving image based on a quantization parameter; An image processing program that causes a computer to realize the above, at the same frame rate, a lower limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is greater than a lower limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode.

1. An image processing program comprising:

15. a step in which a processor acquires a video captured in a first video capture mode or a second video capture mode having different capture conditions from the first video capture mode; a step in which the processor executes image processing to compress image data of frames that constitute the moving image based on a quantization parameter; An image processing method comprising: an upper limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is smaller than an upper limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode, at the same frame rate; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode. An image processing method comprising:

16. a step in which a processor acquires a video captured in a first video capture mode or a second video capture mode having different capture conditions from the first video capture mode; a step in which the processor executes image processing to compress image data of frames that constitute the moving image based on a quantization parameter; An image processing method comprising: at the same frame rate, a lower limit of the second quantization parameter, which is the quantization parameter in the second moving image shooting mode used in the image processing, is greater than a lower limit of the first quantization parameter, which is the quantization parameter in the first moving image shooting mode; In the second moving image shooting mode, at least one of a shutter speed, an autofocus speed, an autoexposure tracking speed, a white balance tracking speed, and a frame rate is set to be faster than in the first moving image shooting mode. An image processing method comprising:

Images