Image processing device, image processing method, and program

Abstract

The image processing device is configured to perform: jitter information processing of outputting output jitter information for input image data constituting an animation; jitter information normalization processing of obtaining normalized jitter information by normalizing the output jitter information based on perspective information; and jitter change processing of changing the jitter state of the input image data using the normalized jitter information.

Description

Technical Field

[0001] The present technology relates to an image processing device, an image processing method, and a program, and particularly to image processing for image dithering. Background Art

[0002] There is known a technique for performing image processing such as various corrections on moving pictures captured by an image capture device.

[0003] The following Patent Document 1 discloses that anti-vibration processing is performed on image data related to a captured image, and the influence of the anti-vibration processing is removed from the anti-vibration processed image data.

[0004] Citation List

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-216510 Summary of the Invention

[0007] Problems to be solved by the present invention

[0008] Incidentally, in recent years, users can easily perform image capturing, image adjustment, and the like by using a mobile terminal such as a smartphone or a tablet, a camera itself, or a personal computer, and movie distribution is also prevalent.

[0009] Under such circumstances, it is desired to generate images with higher quality or various images instead of outputting the images captured by the user as they are. Furthermore, it is also desired that broadcasters and the like can perform the generation of various images.

[0010] For example, applying jitter to images based on the content of an animation is a production technique that expands the expression of an image. Even if one wishes to create an image with intentional jitter by adding jitter or by not completely removing it but leaving jitter, the degree of jitter may appear different depending on the viewing angle of the target image.

[0011] Therefore, the present disclosure proposes a technology capable of generating an image to which shaking of a target degree of shaking is applied, when shaking is added or adjusted in animation.

[0012] Solution to the problem

[0013] An image processing device according to the present technology includes: a shake information processing unit configured to output output shake information for input image data constituting an animation; a shake information normalization unit configured to obtain normalized shake information by normalizing the output shake information based on view angle information; and a shake changing unit configured to perform a shake changing process of changing a shake state of the input image data by using the normalized shake information.

[0014] The shake change processing is to change the state of the shake by reducing the shake occurring in the animation or adding shake. The shake change unit performs the shake change processing based on the output shake information, and in this case, uses the output shake information normalized according to the viewing angle of the image representing the shake.

[0015] In the image processing apparatus according to the present technology described above, it is conceivable that the shake changing unit performs dither addition on the input image data based on the normalized shake information as the shake changing process.

[0016] That is, as one aspect of dither change, dither addition is performed on the image.

[0017] In the image processing apparatus according to the present technology described above, it is conceivable that the shake changing unit performs shake portion removal of the input image data based on the normalized shake information as the shake changing process.

[0018] Specifically, as one aspect of shake reduction, partial shake removal of an image is performed. Shake reduction aims to remove shake applied to a moving image, such as shake that occurs in an image due to movement of an image capture device during image capture (such as camera shake), and to remove a portion of the shake. In other words, partial shake removal aims to reduce shake without completely eliminating it.

[0019] In the image processing device according to the present technology described above, it can be conceived that the angle of view information is a ratio between the angle of view of the target image to be subjected to the dither changing process and the reference angle of view.

[0020] A certain reference viewing angle is determined, and the jitter information is normalized by using the ratio of the viewing angle to the reference viewing angle.

[0021] In the image processing device according to the present technology described above, it can be conceived that the angle of view information is a ratio between the angle of view of the image to be subjected to the dither change process after the cutout and the angle of view at the time of image capture.

[0022] In the case of generating an output image by performing cutout from a captured image, shake information on the cutout image is normalized with reference to the viewing angle of the captured image.

[0023] In the image processing apparatus according to the present technology described above, it is conceivable to include a parameter setting unit configured to set a dither changing parameter related to the dither changing process.

[0024] The shake changing process changes the state of shake by reducing shake occurring in animation or adding shake, and is performed based on the shake changing parameter set by the parameter setting unit.

[0025] In the image processing device according to the above-mentioned present technology, it can be imagined to include a user interface processing unit configured to detect operation information about shake change and a parameter setting unit configured to set shake change parameters about shake change processing based on the operation information detected by the user interface processing unit.

[0026] That is, a user operation environment for specifying the degree of shake is provided.

[0027] In the image processing apparatus according to the present technology described above, it is conceivable that the shake change parameter is obtained by converting information of the angle detected by the user interface processing unit into a shake amount.

[0028] For example, the user is allowed to perform an operation of specifying shake as a shake angle, and the user sets parameters of the shake changing process based on the operation input.

[0029] In the image processing apparatus according to the present technology described above, it is conceivable that the shake information processing unit generates the output shake information based on a shake change parameter that specifies a processing amount of the shake change process.

[0030] For example, in the case where dither addition is performed on an image, output dither information corresponding to an input dither changing parameter is generated.

[0031] In the image processing apparatus according to the present technology described above, it is conceivable that the shake information processing unit obtains the output shake information by adjusting the image-capturing shake information based on a shake change parameter that specifies a processing amount of the shake change process.

[0032] For example, in the case of partially removing shake at the time of image capture in an image, the image capture-time shake information is adjusted by using a shake change parameter.

[0033] In the image processing device according to the present technology described above, it is conceivable that the image-capturing-time shake information is posture information of the image capturing device when image capturing of input image data is performed by the image capturing device.

[0034] For example, the posture information of the image capture device can be obtained from information of an angular velocity sensor or an acceleration sensor.

[0035] In the image processing device according to the present technology described above, it is conceivable that the shake changing unit pastes each frame of input image data to a celestial sphere model and performs shake changing processing by rotating each frame with normalized shake information corresponding to each frame.

[0036] For example, based on shake information (for example, quaternion) of the image capturing device obtained from information of an angular velocity sensor or an acceleration sensor, rotation processing is performed on the celestial sphere model.

[0037] In the image processing apparatus according to the present technology described above, it is conceivable to include an associating unit configured to associate input image data with normalized shake information.

[0038] The image data and the normalized shake information are associated with each other, so that it is possible to grasp what type of shake change is performed as the shake changing process.

[0039] In the image processing method according to the present technology, the image processing device performs jitter information processing of outputting output jitter information for input image data constituting an animation, jitter information normalization processing of obtaining normalized jitter information by normalizing the output jitter information based on view angle information, and jitter changing processing of changing the jitter state of the input image data by using the normalized jitter information.

[0040] This prevents jitter from being added to an image or the degree to which jitter remains from greatly varying depending on the viewing angle.

[0041] The program according to the present technology is a program that causes an information processing apparatus to execute processing corresponding to such an image processing method.

[0042] This enables the image processing of the present disclosure to be performed by various information processing apparatuses. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an explanatory diagram of the device used in the embodiment of the present technology.

[0044] Figure 2 This is a diagram illustrating information transmitted between devices in the embodiment.

[0045] Figure 3 is a block diagram of an image capturing apparatus of an embodiment.

[0046] Figure 4 It is an explanatory diagram of a blur removal process of an image in the image capturing apparatus of the embodiment.

[0047] Figure 5 It is a block diagram of the information processing apparatus of the embodiment.

[0048] Figure 6 It is an explanatory diagram of the functional configuration of the image processing apparatus as an embodiment.

[0049] Figure 7 It is an explanatory diagram of another example of the functional configuration of the image processing apparatus as the embodiment.

[0050] Figure 8 It is an explanatory diagram of still another example of the functional configuration of the image processing apparatus as the embodiment.

[0051] Figure 9 It is a normalized explanatory diagram of the embodiment.

[0052] Figure 10 It is an explanatory diagram of adding dither in the embodiment.

[0053] Figure 11 It is an explanatory diagram of the jitter removal in the embodiment.

[0054] Figure 12 It is an explanatory diagram of the contents of an image file and metadata according to the embodiment.

[0055] Figure 13 This diagram explains metadata related to lens distortion correction.

[0056] Figure 14 It is an explanatory diagram of the image processing of the embodiment.

[0057] Figure 15 This is an explanatory diagram of attaching the celestial sphere model according to the embodiment.

[0058] Figure 16 It is an explanatory diagram of the sampling timing of IMU data in the embodiment.

[0059] Figure 17 This is an explanatory diagram of jitter information adjustment for each frequency band according to the embodiment.

[0060] Figure 18 It is an explanatory diagram of the adjustment of the jitter information in each direction according to the embodiment.

[0061] Figure 19 This is an explanatory diagram of jitter information adjustment in each frequency band and direction according to the embodiment.

[0062] Figure 20 It is an explanatory diagram of the relationship between the output image and the celestial sphere model according to the embodiment.

[0063] Figure 21 It is an explanatory diagram of the rotation and perspective projection of the output coordinate plane of the embodiment.

[0064] Figure 22 It is an explanatory diagram of the cutout area of ​​the embodiment.

[0065] Figure 23 FIG. 1 is an explanatory diagram of the contents of metadata to be transmitted according to the embodiment. DETAILED DESCRIPTION

[0066] The embodiments will be described below in the following order.

[0067] <1. Configuration of Apparatus Suitable as Image Processing Apparatus>

[0068] <2. Device Configuration and Processing Functions>

[0069] <3. Jitter Change and Normalization of Jitter Information>

[0070] <4. Image Files and Metadata>

[0071] <5. Processing Example>

[0072] <6. Overview and Modification Examples>

[0073] Before describing the embodiments, some terms used in the specification will be described.

[0074] "Jitter" refers to the inter-frame jitter of the images constituting the animation. It is assumed to broadly refer to jitter components that appear between frames (inter-frame jitter of the image) (such as jitter caused by camera shake in images captured by so-called image capture devices) and jitter intentionally added through image processing.

[0075] "Jitter change (inter-frame jitter alteration)" refers to changing the state of jitter in an image, such as reducing jitter occurring in an image or adding jitter to an image.

[0076] It is assumed that this “jitter change” includes the following “jitter removal (inter-frame jitter reduction)” and “jitter addition (inter-frame jitter addition)”.

[0077] "Shake removal" refers to the elimination (complete shake removal) or reduction (partial shake removal) of shake that occurs in an image due to camera shake, etc. Specifically, it refers to making adjustments to reduce shake based on shake information from the time of image capture. Image stabilization performed in image capture devices is the performance of shake removal.

[0078] In the present embodiment, an example will be described in which a process of removing a part of a shake component is performed mainly as “shake portion removal”, and this is to perform shake change of an image so that the shake remains.

[0079] "Dithering" refers to adding dither to an image, including adding dither to an image without dithering and adding dither to an image with dithering to further increase the dithering.

[0080] The above “shake portion removal” and “shake addition” are processes for obtaining an image with shake as a result, and therefore can be regarded as processes of shake addition generation.

[0081] Note that as an example of the purpose of shaking generation, it is assumed that an image is shaken intentionally in order to give an impact to an animation scene.

[0082] "Image capture shake information" is information about shake when captured by an image capture device, and corresponds to detection information of the movement of the image capture device, information that can be calculated from the detection information, posture information indicating the posture of the image capture device, and shift and rotation information as the movement of the image capture device, etc.

[0083] In the embodiment, specific examples of “image capture-time shake information” include quaternion (QD) and IMU data, but there are also shift and rotation information, for example, and there is no particular limitation.

[0084] <1. Configuration of Apparatus Suitable as Image Processing Apparatus>

[0085] In the following embodiments, an example in which the image processing apparatus according to the present disclosure is realized mainly by an information processing apparatus such as a smartphone or a personal computer will be described, but the image processing apparatus can be realized in various devices. First, a device to which the technology of the present disclosure can be applied will be described.

[0086] Figure 1 A shows an example of an image source VS and an image processing device (TDx, TDy) that acquires an image file MF from the image source VS.

[0087] Note that the image processing device TDx is assumed to be a device that once performs dither changing processing on image data acquired from the image source VS.

[0088] On the other hand, the image processing device TDy is assumed to be a device that secondarily performs dither changing processing on image data that has been subjected to dither changing processing by another image processing device.

[0089] As the image source VS, an image capturing device 1 , a server 4 , a recording medium 5 , and the like are assumed.

[0090] As the image processing devices TDx and TDy, a mobile terminal 2 such as a smartphone or a personal computer 3 is assumed. Although not shown, various other devices such as a dedicated image editing device, a cloud server, a television device, and a video recording and playback device are assumed as the image processing devices TDx and TDy. These devices can be used as either of the image processing devices TDx and TDy.

[0091] The image capturing device 1 as the image source VS is a digital camera or the like capable of capturing moving pictures, and transmits an image file MF obtained by capturing the moving pictures to the mobile terminal 2 or the personal computer 3 or the like via wired communication or wireless communication.

[0092] The server 4 may be any of a local server, a network server, a cloud server, etc., but refers to a device that can provide the image file MF captured by the image capture device 1. It is conceivable that the server 4 transmits the image file MF to the mobile terminal 2 or the personal computer 3 via some transmission path.

[0093] The recording medium 5 may be any of a solid-state memory such as a memory card, a disk-shaped recording medium such as an optical disk, a tape-shaped recording medium such as a magnetic tape, and the like, but refers to a removable recording medium that records the image file MF captured by the image capture device 1. It is conceivable that the image file MF read from the recording medium 5 is read by the mobile terminal 2, the personal computer 3, or the like.

[0094] The mobile terminal 2 and the personal computer 3 as the image processing devices TDx and TDy can perform image processing on the image file MF obtained from the above-mentioned image source VS. The image processing mentioned here includes dither changing processing (dither addition or dither portion removal).

[0095] For example, the shake changing process is performed by performing a pasting process on the celestial sphere model for each frame of image data constituting the animation and then by rotating it with the posture information corresponding to the frame.

[0096] Note that a certain mobile terminal 2 or personal computer 3 sometimes serves as an image source VS for another mobile terminal 2 or personal computer 3 serving as the image processing devices TDx and TDy.

[0097] Figure 1 B shows an image capture device 1 and a mobile terminal 2 as one piece of equipment that can function as both an image source VS and an image processing device TDx.

[0098] For example, a microcomputer or the like within the image capturing apparatus 1 executes the shake changing process.

[0099] That is, the image capturing apparatus 1 is assumed to be capable of performing image output as a result of image processing to which shake portion removal or shake addition is applied, by performing shake changing processing on the image file MF generated by image capturing.

[0100] By including the image capturing function, the mobile terminal 2 can similarly be the image source VS and thus can perform image output as a result of image processing applied with shake portion removal or shake addition by performing shake changing processing on the image file MF generated by image capturing.

[0101] Of course, not limited to the image capturing apparatus 1 and the mobile terminal 2 , there are various other devices that can be used as the image source and the image processing apparatus.

[0102] As described above, there are various devices that serve as image processing devices TDx and TDy and image sources VS as embodiments, but in the following description, image sources VS such as the image capture device 1, image processing devices TDx such as the mobile terminal 2, and other image processing devices TDy will be described as separate devices.

[0103] Figure 2 The state of information transfer among the image source VS, the image processing device TDx, and the image processing device TDy is shown.

[0104] Image data VD1 and metadata MTD1 are transmitted from an image source VS to an image processing device TDx via wired communication, wireless communication, or a recording medium.

[0105] As will be described later, the image data VD1 and the metadata MTD1 are information transmitted as, for example, an image file MF.

[0106] The metadata MTD1 may include coordinate transformation parameters HP as information on shake removal at the time of image capturing performed as image stabilization or the like, for example.

[0107] The image processing device TDx can perform various types of processing in response to the image data VD1 , the metadata MTD1 , and the coordinate transformation parameters HP.

[0108] For example, the image processing device TDx can perform the shake changing process on the image data VD1 by using the image-capturing-time shake information included in the metadata MTD1.

[0109] Furthermore, for example, the image processing device TDx can also cancel shake removal applied to the image data VD1 at the time of image capturing by using the coordinate transformation parameters HP included in the metadata MTD1.

[0110] When the image processing device TDx has performed the shake change process, the image processing device TDx can perform a process of associating image data, shake information at the time of image capture, and shake change information SMI and normalized shake information SCI that can specify the processing amount of the shake change process.

[0111] The dither change information SMI is information of the dither change processing of the image data, and only needs to be information that can specify the processing amount of the dither change processing.

[0112] In the embodiment, specific examples of the “jitter change information” include a jitter change parameter (PRM) and a jitter change quaternion (eQD), but are not limited thereto.

[0113] The normalized jitter information SCI is jitter information that has been normalized. Although a normalized quaternion (nQD) will be described later as an example, in this embodiment, the jitter changing process is performed based on the jitter information that has been normalized.

[0114] Then, the image processing device TDx may transmit the associated image data, image-capturing jitter information, jitter change information SMI, and normalized jitter information SCI together or separately to the image processing device TDy via wired communication, wireless communication, or a recording medium.

[0115] Alternatively, at least the image data and the normalized dither information SCI are transmitted in association with each other.

[0116] Here, the term "association" means that, for example, when processing one piece of information (data, command, and program, etc.), another piece of information can be used (linked). That is, the mutually associated pieces of information can be put together as a file, etc., or can be separate pieces of information. For example, information B associated with information A can be transmitted on a transmission path different from the transmission path of information A. And, for example, information B associated with information A can be recorded in a recording medium different from the recording medium of information A (or in another recording area of ​​the same recording medium). Note that this "association" can be a part of the information, not the entire information. For example, an image and information corresponding to the image can be associated with each other in arbitrary units such as multiple frames, one frame, or a part of a frame.

[0117] More specifically, "association" includes actions such as giving the same ID (identification information) to multiple pieces of information, recording multiple pieces of information to the same recording medium, storing multiple pieces of information in the same folder, storing multiple pieces of information in the same file (giving one piece as metadata to another), embedding multiple pieces of information into the same stream, and embedding meta into an image such as a digital watermark.

[0118] Figure 2 The image data transmitted from the image processing device TDx to the image processing device TDy is shown as image data VD2. Various examples of the image data VD2 include an image from which shake removal performed by the image capture device 1 has been canceled, an image to which shake modification has been performed by the image processing device TDx, and an image before shake modification by the image processing device TDx.

[0119] and, Figure 2 The metadata MTD2 transmitted from the image processing device TDx to the image processing device TDy is shown. The metadata MTD2 is the same information as the metadata MTD1 or partially different information. However, the metadata MTD2 includes image capture shake information.

[0120] Therefore, the image processing device TDy can acquire, for example, at least the image data VD2, the image-capturing shake information contained in the metadata MTD2, the normalized shake information SCI, and the shake change information SMI in an associated state.

[0121] Note that it is also conceivable that the normalized jitter information SCI and the jitter change information SMI are also included in the metadata MTD2.

[0122] In the present embodiment, the dither changing process mainly in the image processing device TDx will be described, but the information transmission as described above can be assumed for the image processing device TDx.

[0123] <2. Device Configuration and Processing Functions>

[0124] First, refer to Figure 3 A configuration example of the image capturing device 1 serving as the image source VS is described.

[0125] Note that, assuming that the image file MF captured by the mobile terminal 2 is as shown in FIG. Figure 1 In the case of undergoing image processing by the mobile terminal 2 as described in B, the mobile terminal 2 only needs to include a configuration equivalent to the following image capturing apparatus 1 regarding the image capturing function.

[0126] Furthermore, the image capture device 1 performs processing to reduce shake in images caused by the movement of the image capture device during image capture. This processing is called image stabilization, and this is "shake removal" performed by the image capture device. On the other hand, the "shake addition" and "shake portion removal" performed by the image processing devices TDx and TDy are separate processes independent of the "shake removal" performed when the image capture device 1 captures images.

[0127] like Figure 3 As shown, the image capturing device 1 includes, for example, a lens system 11, an image capturing element unit 12, a camera signal processing unit 13, a recording control unit 14, a display unit 15, an output unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, a driver unit 22, and a sensor unit 23.

[0128] The lens system 11 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture mechanism. Light from an object (incident light) is guided by the lens system 11 and collected on the image capturing element unit 12.

[0129] Note that, although not shown, there is a case where the lens system 11 is provided with an optical image stabilization mechanism that corrects inter-frame shake and blur of an image due to camera shake or the like.

[0130] The image capturing element unit 12 includes an image sensor 12 a (image capturing element) of, for example, a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type.

[0131] The image capturing element unit 12 performs, for example, correlated double sampling (CDS) processing and automatic gain control (AGC) processing on the electrical signal obtained by photoelectric conversion of the light received by the image sensor 12a, and further performs analog / digital (A / D) conversion processing. Then, in the subsequent stage, the image capturing signal as digital data is output to the camera signal processing unit 13 and the camera control unit 18.

[0132] Note that as an optical image stabilization mechanism not shown, there is a case where the shake in the image is corrected by moving the image sensor 12a side instead of moving the lens system 11 side, and a case where a balanced optical image stabilization mechanism using a gimbal is used, etc., and any method can be used.

[0133] In the optical image stabilization mechanism, in addition to shake, blur in the frame is also corrected as described below.

[0134] The camera signal processing unit 13 is configured as an image processing processor by, for example, a digital signal processor (DSP) or the like. This camera signal processing unit 13 performs various types of signal processing on the digital signal (captured image signal) from the image capturing element unit 12. For example, as camera processing, the camera signal processing unit 13 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, and the like.

[0135] Furthermore, the camera signal processing unit 13 also performs various types of correction processing. However, there are cases where image stabilization is performed in the image capture apparatus 1 or cases where image stabilization is not performed.

[0136] The pre-processing includes a clamping process of clamping the black levels of R, G, and B to predetermined levels, a correction process between the color channels of R, G, and B, and the like for the captured image signal from the image capturing element unit 12 .

[0137] The synchronization process includes color separation processing for the image data of each pixel to have all color components of R, G, and B. For example, in the case of an image capturing element using a Bayer array color filter, demosaicing processing is performed as the color separation processing.

[0138] In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data.

[0139] In the resolution conversion process, the resolution conversion process is performed on the image data subjected to various types of signal processing.

[0140] Figure 4 Examples are given of various types of correction processing (internal correction of the image capturing apparatus 1 ) performed by the camera signal processing unit 13 . Figure 4 The correction processing performed by the camera signal processing unit 13 and the optical image stabilization performed by the lens system 11 are illustrated in the order of execution.

[0141] In the optical image stabilization as processing F1, in-lens image stabilization by shifting in the pan direction and pitch direction of the lens system 11 and in-body image stabilization by shifting in the pan direction and pitch direction of the image sensor 12a are performed, so that an image of the subject is formed on the image sensor 12a in a state in which the influence of camera shake is physically canceled.

[0142] There are cases where only one of in-lens image stabilization and in-vivo image stabilization is used, and there are cases where both are used simultaneously. When in-lens image stabilization and in-vivo image stabilization are used simultaneously, it is conceivable that in-vivo image stabilization does not perform shifts in the pan and tilt directions.

[0143] Also, there are cases where neither in-lens image stabilization nor in-body image stabilization is employed, and only electrical image stabilization or only optical image stabilization is performed on camera shake.

[0144] In the camera signal processing unit 13 , the processes from the process F2 to the process F7 are executed by space coordinate conversion for each pixel.

[0145] In process F2, lens distortion correction is performed.

[0146] In the process F3, focal plane distortion correction is performed as one element of the electrical image stabilization. Note that this is to correct distortion when, for example, the CMOS image sensor 12a performs reading using the rolling shutter method.

[0147] In process F4, roll correction is performed. That is, correction of the roll component, which is one element of electronic image stabilization, is performed.

[0148] In process F5, keystone correction is performed on the keystone amount caused by the electrical image stabilization. The keystone amount caused by the electrical image stabilization is perspective distortion caused by cropping a position away from the center of the image.

[0149] In the process F6 , as one element of the electrical image stabilization, shifting and shearing in the pitch direction and the pan direction are performed.

[0150] For example, through the above-described process, image stabilization, lens distortion correction, and keystone correction are performed.

[0151] Note that all the processing described here does not necessarily have to be performed, and the order of the processing may be switched appropriately.

[0152] exist Figure 3 In the codec processing in the camera signal processing unit 13, for example, encoding processing and file generation for recording or communication are performed on the image data subjected to the above-mentioned various types of processing. For example, the image file MF is generated in the MP4 format for recording moving images and audio conforming to MPEG-4. It is also conceivable that a file in a format such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Graphics Interchange Format (GIF) is generated as a still image file.

[0153] Note that the camera signal processing unit 13 also generates metadata to be added to the image file MF by using information from the camera control unit 18 and the like.

[0154] And, despite Figure 3 The audio processing system is not shown in FIG. 1 , but actually includes an audio recording system and an audio processing system, and the image file MF may include audio data as well as image data as animation.

[0155] The recording control unit 14 performs recording and reproduction on the recording medium through, for example, a nonvolatile memory. For example, the recording control unit 14 performs processing for recording image files MF and thumbnails of moving image data and still image data on the recording medium.

[0156] The actual form of the recording control unit 14 can be conceived in various ways. For example, the recording control unit 14 may be configured as a flash memory and its write / read circuit built into the image capture device 1, or may be in the form of a card recording and reproducing unit (e.g., a memory card (portable flash memory, etc.)) configured to perform recording and reproduction access to a recording medium that can be attached to and detached from the image capture device 1. Furthermore, as a form built into the image capture device 1, there are cases where the recording control unit 14 is implemented as a hard disk drive (HDD) or the like.

[0157] The display unit 15 is a display unit configured to perform various types of displays on the image capturing person, and is, for example, a display panel or viewfinder of a display device such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display provided in the housing of the image capturing apparatus 1.

[0158] The display unit 15 performs various types of display on the display screen based on instructions from the camera control unit 18 .

[0159] For example, the display unit 15 displays a reproduced image of image data read from the recording medium in the recording control unit 14 .

[0160] Furthermore, there is a case where the image data of the captured image subjected to display resolution conversion by the camera signal processing unit 13 is supplied to the display unit 15, and the display unit 15 performs display based on the image data of the captured image in response to an instruction from the camera control unit 18. Thus, a so-called through-the-lens image (subject monitoring image), which is an image captured during the composition check, is displayed.

[0161] Also, based on an instruction from the camera control unit 18 , the display unit 15 performs display of various operation menus, icons, messages, and the like on the screen as a graphical user interface (GUI).

[0162] The output unit 16 performs data communication and network communication with external devices in a wired or wireless manner.

[0163] The output unit 16 transfers and outputs the captured image data (still image file or moving image file) to, for example, an external display device, a recording apparatus, a reproducing apparatus, or the like.

[0164] Also, as a network communication unit, the output unit 16 can perform communication via various networks such as the Internet, a home network, and a local area network (LAN), and transmit and receive various data to and from a server, a terminal, and the like on the network.

[0165] The operation unit 17 collectively indicates an input device for a user to perform various types of operation inputs. Specifically, the operation unit 17 indicates various operators (keys, dials, touch screen, touch panel, etc.) provided in the housing of the image capturing apparatus 1.

[0166] The user's operation is detected by the operation unit 17 , and a signal corresponding to the input operation is transmitted to the camera control unit 18 .

[0167] The camera control unit 18 includes a microcomputer (arithmetic processing device) including a central processing unit (CPU).

[0168] The memory unit 19 stores information and the like used for processing of the camera control unit 18. The memory unit 19, shown generally, exhibits, for example, a read-only memory (ROM), a random access memory (RAM), a flash memory, and the like.

[0169] The memory unit 19 may be a memory area built in a microcomputer chip as the camera control unit 18 , or may be configured by a separate memory chip.

[0170] The camera control unit 18 controls the entire image capturing apparatus 1 by executing a program stored in the ROM or flash memory or the like of the memory unit 19 .

[0171] For example, the camera control unit 18 controls the actions of the necessary units, such as control of the shutter speed of the image capturing element unit 12, instructions for various types of signal processing in the camera signal processing unit 13, image capturing actions and recording actions according to user operations, reproduction actions of recorded image files, actions of the lens system 11 in the lens barrel such as zooming, focusing, and aperture adjustment, and user interface actions.

[0172] The RAM in the memory unit 19 is used as a work area for temporary storage of data, programs, and the like during various data processing by the CPU of the camera control unit 18 .

[0173] The ROM and flash memory (nonvolatile memory) in the memory unit 19 are used to store an operating system (OS) for the CPU to control each unit, content files such as image files, application programs for various operations, firmware, and the like.

[0174] The driver unit 22 is provided with, for example, a motor driver for a zoom lens driving motor, a motor driver for a focus lens driving motor, a motor driver for a motor of an aperture mechanism, and the like.

[0175] These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18 and cause the drivers to perform movement of the focus lens and zoom lens, opening and closing of the aperture blades of the aperture mechanism, and the like.

[0176] The sensor unit 23 comprehensively indicates various sensors mounted on the image capturing device.

[0177] The sensor unit 23 is mounted with, for example, an inertial measurement unit (IMU) in which, for example, an angular velocity (gyro) sensor of three axes of pitch, pan, and roll can detect angular velocity, and an acceleration sensor can detect acceleration.

[0178] Note that the sensor unit 23 only needs to include a sensor capable of detecting camera shake at the time of image capturing, and does not need to include both a gyro sensor and an acceleration sensor.

[0179] Furthermore, as the sensor unit 23 , a position information sensor, an illuminance sensor, or the like may be installed.

[0180] For example, an image file MF that is a moving picture captured and generated by the above-described image capture device 1 may be transmitted to image processing devices TDx and TDy such as the mobile terminal 2 and subjected to image processing.

[0181] For example, the mobile terminal 2 and the personal computer 3 serving as the image processing devices TDx and TDy may be implemented as a system including Figure 5 Note that it is possible to similarly Figure 5 The information processing device of the configuration implements the server 4.

[0182] exist Figure 5 In the information processing apparatus 70, the CPU 71 executes various types of processing according to the program stored in the ROM 72 or the program loaded from the storage unit 79 into the RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various types of processing.

[0183] The CPU 71, the ROM 72, and the RAM 73 are connected in relation via a bus 74. To this bus 74, an input / output interface 75 is also connected.

[0184] An input unit 76 including an operator and an operating device is connected to the input / output interface 75 .

[0185] For example, as the input unit 76 , various operators and operating devices such as a keyboard, a mouse, keys, a dial, a touch screen, a touch panel, and a remote controller are assumed.

[0186] The user's operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

[0187] Also, a display unit 77 including an LCD or an organic EL panel and an audio output unit 78 including a speaker are integrally or individually connected to the input / output interface 75 .

[0188] The display unit 77 is a display unit configured to perform various types of display, and includes, for example, a display device provided in the casing of the information processing apparatus 70 or a separate display device connected to the information processing apparatus 70 .

[0189] The display unit 77 displays images used for various types of image processing and animations of processing targets on a display screen based on instructions from the CPU 71. Furthermore, based on instructions from the CPU 71, the display unit 77 displays various operation menus, icons, messages, and the like, i.e., displays them as a graphical user interface (GUI).

[0190] In some cases, a storage unit 79 including a hard disk, a solid-state memory, or the like and a communication unit 80 including a modem or the like are connected to the input / output interface 75 .

[0191] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various types of devices, bus communication, and the like.

[0192] The drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

[0193] Data files such as image files MF and various computer programs can be read from the removable recording medium 81 by the drive 82. The read data files are stored in the storage unit 79, and the images and audio contained in the data files are output by the display unit 77 and the audio output unit 78. Also, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as needed.

[0194] In this information processing apparatus 70, software for image processing as the image processing apparatus of the present disclosure, for example, can be installed via network communication through the communication unit 80 or the removable recording medium 81. Alternatively, the software may be pre-stored in the ROM 72 or the storage unit 79 or the like.

[0195] For example, by such software (application), a program such as Figure 6 Functional configuration shown.

[0196] Figure 6 The functions provided in the information processing device 70 serving as the image processing device TDx are shown. That is, the information processing device 70 (CPU 71) includes functions as a shake changing unit 100, a shake information processing unit 101, a parameter setting unit 102, a user interface processing unit 103, a shake information normalization unit 105, and an association unit 106.

[0197] Note that “user interface” is also referred to as “UI”, and the user interface processing unit 103 is also referred to as “UI processing unit 103 ” hereinafter.

[0198] The jitter information processing unit 101 performs processing to output jitter information for input image data used to compose an animation. For example, the jitter information processing unit 101 adjusts jitter information during image capture to obtain and output jitter information for use in removing or adding jitter. Alternatively, the jitter information processing unit 101 generates and outputs jitter information for removing or adding jitter based on input jitter change parameters PRM.

[0199] The jitter modification parameter PRM is a processing parameter generated by user operation or automatic control.

[0200] The jitter information normalization unit 105 performs processing for normalizing the output jitter information output by the adjustment processing and generation processing in the jitter information processing unit 101 based on the viewing angle information. Then, the normalized jitter information SCI is output to the jitter changing unit 100. The normalization processing will be described later.

[0201] The jitter changing unit 100 performs a jitter changing process of changing a jitter state of image data by using normalized jitter information SCI.

[0202] The jitter changing unit 100 includes a function as a jitter adding unit 100 a and particularly performs a process of adding jitter by using normalized jitter information SCI.

[0203] Also, the jitter changing unit 100 has a function as a jitter portion removing unit 100 b , and particularly performs a process of removing a portion of jitter by using normalized jitter information SCI.

[0204] Note that the jitter adding unit 100a and the jitter portion removing unit 100b are not necessarily formed by different algorithms, and it can be understood that jitter addition or jitter portion removal is performed as a result of jitter change processing using the normalized jitter information SCI. The reason why the jitter adding unit 100a and the jitter portion removing unit 100b are shown in the drawings is to indicate that jitter addition or jitter portion removal is performed as a jitter change processing.

[0205] The jitter changing unit 100 outputs jitter-changed image data to which jitter addition or jitter portion removal has been performed.

[0206] The UI processing unit 103 is a function of a process of presenting an operator related to a shake change to the user and acquiring operation information through the operator.

[0207] For example, the UI processing unit 103 performs processing to cause the display unit 77 to display an image indicating information about the operator and the image as a UI image. In addition, the UI processing unit 103 detects an operation performed by the user using the input unit 76. For example, a touch operation on the UI image is detected.

[0208] The parameter setting unit 102 is a function of setting parameters (shake change parameters PRM) for the shake change process based on the operation information acquired by the UI processing unit 103. That is, the user operation content detected by the UI processing unit 103 is converted into the shake change parameters PRM and supplied to the shake information processing unit 101, so that the shake change process according to the user operation is performed on the image data.

[0209] Note that parameter setting unit 102 does not necessarily need to perform parameter setting based on user operations. For example, parameter setting can be performed to add a fixed shaking pattern. Furthermore, for example, the shaking change parameter PRM can be automatically set based on the image content. For example, as archived shaking pattern data, information can be obtained from a database for shaking patterns corresponding to "a person walking slowly," "a person walking," and "a person running," shaking patterns corresponding to "rabbits," "cats," and "horses," as well as shaking patterns corresponding to cars, ships, and airplanes. The shaking information can be selected based on the image content, and the shaking change parameter PRM can be set based on the shaking pattern.

[0210] Of course, it is also conceivable that the user selects a dither mode and sets the dither change parameter PRM according to the selection.

[0211] Note that the “jitter information” mentioned in this disclosure may be considered as any of the following information:

[0212] Information representing the amount of jitter present in the image

[0213] Information representing the amount of jitter added to the image.

[0214] When the shake modification process is executed in a direction that suppresses the amount of shake indicated by the "information representing the amount of shake present in the image," shake removal is performed. In this case, if the shake modification process is executed in a direction that adjusts the amount of shake to be small and suppresses the amount of shake, shake removal is performed.

[0215] On the other hand, also when shaking is removed based on "information representing the amount of shaking present in the image" and then added based on shaking information in which "information representing the amount of shaking present in the image" is adjusted, shaking portion removal is performed as a result.

[0216] Also, when jitter change is performed to add jitter to a jitter-free image, an image from which jitter has been removed, or an image with jitter according to "information representing the amount of jitter added to the image", jitter addition is performed.

[0217] That is, the specific arithmetic processing of the jitter change differs depending on which of the above-mentioned “jitter information” is used, but the jitter change of the present embodiment is possible in either case.

[0218] The associating unit 106 is a function that performs a process of associating image data with normalized shake information SCI.

[0219] For example, the associating unit 106 may store the image data and the normalized shake information SCI in association with each other. Alternatively, the associating unit 106 may generate and store information for associating the image data and the normalized shake information SCI with each other.

[0220] In this case, the image data is image data before the jitter changing process in the jitter changing unit 100 is performed.

[0221] The association unit 106 may also associate the normalized jitter information SCI with the jitter change information SMI (see Figure 2 ) is associated with image capture time jitter information. The jitter change information SMI is, for example, a jitter change parameter PRM input for jitter change of image data.

[0222] Figure 7 Show changes Figure 6 Function configuration.

[0223] exist Figure 7 In this case, the jitter changing unit 100 includes only the jitter adding unit 100a.

[0224] The jitter information processing unit 101 generates output jitter information based on the jitter changing parameter PRM from the parameter setting unit 102 , and the output jitter information is normalized by the jitter information normalization unit 105 and sent to the jitter changing unit 100 .

[0225] The jitter changing unit 100 performs jitter addition based on the normalized jitter information SCI.

[0226] Figure 8 Also shows changes Figure 6 Function configuration.

[0227] exist Figure 8 In this case, the jitter changing unit 100 includes only the jitter portion removing unit 100b.

[0228] The shake information processing unit 101 outputs output shake information in which shake information at the time of image capture is adjusted based on the shake changing parameter PRM from the parameter setting unit 102 , and the output shake information is normalized by the shake information normalization unit 105 and sent to the shake changing unit 100 .

[0229] The jitter changing unit 100 performs jitter portion removal based on the normalized jitter information SCI.

[0230] Although the above Figure 7and Figure 8 is a more simplified functional configuration example, but other configuration examples can be envisioned. For example, in Figure 8 In a configuration such as the one shown in FIG. 1 , the jitter changing unit 100 may include only the jitter adding unit 100a. For example, based on the jitter changing parameter PRM from the parameter setting unit 102, the jitter information processing unit 101 outputs jitter information adjusted for the jitter information during image capture. The jitter information normalization unit 105 then normalizes the jitter information and transmits it to the jitter changing unit 100. The jitter changing unit 100 then adds jitter based on the normalized jitter information SCI. This is an example of processing that further increases the jitter of the original image data.

[0231] Note that although the above has been described as a functional configuration of the image processing device TDx, it can also be regarded as a functional configuration of the image processing device TDy.

[0232] However, in the case of the image processing device TDy, the image data and the normalized shake information SCI as the processing targets of the shake changing unit 100 are data associated by the image processing device TDx or the like at the time of the previous shake change.

[0233] Furthermore, the image source VS and the image processing device TDx are configured by referring to, for example, Figure 1 In the case where the image capture device 1 described in B is implemented, it is also conceivable that Figure 6 、 Figure 7 and Figure 8 The function is included in the camera signal processing unit 13 or the camera control unit 18 of the image capture device 1 to serve as the image processing device TDx.

[0234] <3. Jitter Change and Normalization of Jitter Information>

[0235] Here, the processing of jitter addition and jitter portion removal of the embodiment, and normalization of jitter information by the jitter information normalization unit 105 will be described.

[0236] It can be considered that the processing of this embodiment adjusts the intensity of shaking according to the viewing angle of the image to be processed.

[0237] When the shake amount is specified by angle (rotation angle of celestial sphere model MT described later) as shake addition or shake partial removal, even if the same instruction value is used, the behavior (ratio of shake amplitude to viewing angle) varies depending on the viewing angle of the applicable image.

[0238] For this reason, for example, in the case of applying the same dithered data to two images having different angles of view, in order to obtain the same degree of dither generation effect in terms of the ratio with respect to the angles of view, different intensities need to be set, which is complicated.

[0239] Therefore, in this embodiment, the amplitude of the dither applied by the dither change is normalized by the viewing angle of the image.

[0240] For example, a reference angle of view assumed as dither data is determined in advance, and the magnitude of dithering is adjusted according to the relationship between the reference angle of view and the angle of view of an image to be actually subjected to dithering changes.

[0241] For example, in the case where dither is newly added to a dither-free image, etc., the following relationship holds:

[0242] (Normalized dither addition amount) = (Original dither addition amount) × ((Target image viewing angle) / (Reference viewing angle))

[0243] Note that the original jitter addition amount is output jitter information generated by the jitter information processing unit 101. For example, it is equivalent to a jitter change quaternion eQD described later having been subjected to gain adjustment according to a user operation or the like.

[0244] The reference viewing angle is a certain viewing angle predetermined as a reference.

[0245] Also, in the case where shake portion removal is performed by adjusting the shake removal amount for an image having shake, the following relationship holds true:

[0246] (normalized residual jitter amount) = (original residual jitter amount) × ((angle of view of the cropped image) / (angle of view of the captured image))

[0247] Note that the original jitter remainder is output jitter information adjusted by the jitter information processing unit 101. This is also equivalent to, for example, the jitter change quaternion eQD described later having been subjected to gain adjustment according to a user operation or the like.

[0248] For example, it is conceivable that such normalization is performed based on all images being aligned with respect to the vertical viewing angle, since the vertical viewing angle generally becomes the same when the same scene is captured with different aspect ratios.

[0249] Then, in order to perform normalization according to the viewing angle of the shake amount, as a reference viewing angle, information of at least the vertical viewing angle is retained as metadata.

[0250] Figure 9 A shows the vertical viewing angle FOV_Vstd and the horizontal viewing angle FOV_Hstd as reference viewing angles. Figure 9 B shows a vertical angle of view FOV_Vtgt and a horizontal angle of view FOV_Htgt of the angle of view of the image as a processing target.

[0251] In the case of this diagram, FOV_Vstd≈27 [degrees] and FOV_Vtgt≈14.4 [degrees].

[0252] Here, “(target image angle of view) / (reference angle of view)” and “(angle of view of the cut-out image) / (angle of view of the captured image)” of the above-mentioned normalization operation expression are defined as normalization coefficients α.

[0253] Then, assuming that the “normalized jitter addition amount” and the “normalized jitter residual amount” are the normalized jitter information SCI described above, and the “original jitter addition amount” and the “original jitter residual amount” are the jitter information output by the jitter information processing unit 101,

[0254] Normalized jitter information SCI=(jitter information)×α holds true.

[0255] Then, consider Figure 9 In the case of vertical viewing angle processing,

[0256] Normalization coefficient α = FOV_Vtgt / FOV_Vstd holds. That is, the following relationship holds:

[0257] Normalized jitter information SCI = (jitter information) × (FOV_Vtgt / FOV_Vstd)

[0258] For example, by performing such normalization processing and performing shake change based on the normalized shake information SCI, the degree of shake (the way shake appears) does not greatly change depending on the viewing angle.

[0259] Note that although an example using a vertical viewing angle has been described above, the normalization coefficient α may be obtained by a horizontal viewing angle or an oblique (diagonal) viewing angle.

[0260] Figure 10 The following schematically illustrates the case where a shake modification is performed on images P1 and P2, both of which have a relatively small angle of view. Image P1 is assumed to have been captured by a camera on a studio set, for example, with a composite background. Furthermore, image P2 is assumed to have been captured using a different camera on the same studio set, using zoom, and with a composite background. In this case, it is assumed that a ship shake is added based on a shake pattern stored, for example, as a shake profile.

[0261] When dither information is generated by using the dither changing parameter PRM for adding dither and dither addition is performed, even dither that appears appropriate for the image P1 becomes visually severe dither of the image P2.

[0262] Therefore, in this case, by using the image capturing angle of view of each target image and the reference angle of view, the shake information is normalized by the normalization coefficient α, and the shake change is performed by using the normalized shake information SCI. As shown in the figure, this allows both to express moderate shake.

[0263] Figure 11 The example shows a case where images P4 and P5 having different perspectives are cut from an image P3 that initially has severe shaking. In this case, for each of the images P4 and P5 to be cut, shaking is changed based on the input image data, the perspective of the captured image, the perspective of the cut image, and the shaking information during image capture.

[0264] Also, for the jitter portion removal, for example, a jitter change parameter according to the jitter remaining amount specified by the user is input.

[0265] If shake modification is performed without normalizing the shake information obtained from the shake modification parameters alone during image capture, it is assumed that images P4 and P5 will not experience the residual shake desired by the user. Therefore, shake modification is performed using normalized shake information SCI, which is normalized using a normalization coefficient α derived from the angles of view of the captured image and the angles of view of the cropped image. As a result, images P4 and P5 visually exhibit the same degree of shake.

[0266] <4. Image Files and Metadata>

[0267] Hereinafter, an example will be described in which the above-described normalization of the shake information is performed on the image file MF captured by the image capturing device 1 serving as the image source VS and input to the image processing device TDx.

[0268] First, the contents of an image file MF and the contents of metadata to be transferred from an image source VS such as the image capturing device 1 to the image processing device TDx will be described.

[0269] Figure 12 A shows data included in the image file MF. As shown in the figure, the image file MF includes various data such as "title", "sound", "movie" and "metadata".

[0270] In the “title”, information indicating the presence or absence of metadata and the like is described together with information such as the file name and file size.

[0271] "Sound" is audio data recorded with the animation. For example, it stores two-channel stereo audio data.

[0272] "Movie" is movie data, and includes image data of frames (#1, #2, #3, ...) constituting the movie.

[0273] As "metadata", additional information associated with each frame (#1, #2, #3, ...) constituting the animation is described.

[0274] exist Figure 12An example of the metadata content is shown in B. For example, IMU data, coordinate transformation parameters HP, timing information TM, and camera parameters CP are described for one frame. Note that these are part of the metadata content, and only information related to the image processing described later is shown here.

[0275] As IMU data, the gyroscope (angular velocity data), accelerometer (acceleration data), and sampling rate are described.

[0276] The IMU mounted on the image capture device 1 as the sensor unit 23 outputs angular velocity data and acceleration data at a predetermined sampling rate. Typically, the sampling rate is higher than the frame rate of the captured image, and thus many IMU data samples are obtained within one frame period.

[0277] Therefore, as angular velocity data, n samples are associated with one frame, such as Figure 12 C shows gyro sample #1, gyro sample #2, ..., and gyro sample #n.

[0278] Also, as acceleration data, m samples are associated with one frame, such as accelerator sample #1, accelerator sample #2, . . . , and accelerator sample #m.

[0279] There is a case where n=m, and there is a case where n≠M.

[0280] Note that although the example described here associates metadata with each frame, there are cases where, for example, IMU data is not completely synchronized with the frame. In such cases, for example, the time information associated with the time information of each frame is retained as the IMU sampling timing offset in the timing information TM.

[0281] The coordinate transformation parameter HP is a general term for parameters used for correction accompanying the coordinate transformation of each pixel in an image. It also includes nonlinear coordinate transformations such as lens distortion.

[0282] Then, the coordinate transformation parameters HP are terms that may include at least lens distortion correction parameters, keystone distortion correction parameters, focal plane distortion correction parameters, electrical image stabilization parameters, and optical image stabilization parameters.

[0283] The lens distortion correction parameter is information for directly or indirectly understanding how to correct distortions such as barrel aberration and pincushion aberration and return the image to the image before lens distortion correction. Metadata about the lens distortion correction parameter, which is one of the metadata, will be briefly described.

[0284] Figure 13A shows the image height Y, the angle β, the entrance pupil position d1 , and the exit pupil position d2 in the schematic diagram of the lens system 11 and the image sensor 12 a .

[0285] The lens distortion correction parameters are used to determine the incident angle of each pixel of the image sensor 12a during image processing. Therefore, it is only necessary to determine the relationship between the image height Y and the angle β.

[0286] Figure 13 B shows an image 110 before lens distortion correction and an image 111 after lens distortion correction. The maximum image height H0 is the maximum image height before distortion correction and is the distance from the center of the optical axis to the farthest point. The maximum image height H1 is the maximum image height after distortion correction.

[0287] In order to make the relationship between the image height Y and the angle β known, the necessary metadata is the maximum image height H0 before distortion correction and the data of the incident angles d0, d1, ..., d(N-1) for the corresponding N image heights. As an example, "N" is assumed to be approximately 10.

[0288] return Figure 12 B, the keystone correction parameter is a correction amount when correcting keystone distortion caused by shifting the cropped area from the center by the electrical image stabilization, and also has a value corresponding to the correction amount of the electrical image stabilization.

[0289] The focal plane distortion correction parameter is a value indicating the correction amount for each line of focal plane distortion.

[0290] The electrical image stabilization and the optical image stabilization are parameters indicating the correction amount in each axial direction of pan, pitch, and roll.

[0291] Note that the parameters for lens distortion correction, keystone correction, focal plane distortion correction, and electrical image stabilization are collectively referred to as coordinate transformation parameters. This is because these correction processes are correction processes for the image formed on each pixel of the image sensor 12a of the image capturing element unit 12, and they are parameters for correction processes involving coordinate transformation of each pixel. This is because, although optical image stabilization is also one of the coordinate transformation parameters, the shake correction of the inter-frame component in optical image stabilization becomes a process involving coordinate transformation of each pixel.

[0292] That is, by performing reverse correction using these parameters, the image data subjected to lens distortion correction, keystone distortion correction, focal plane distortion correction, electrical image stabilization, and optical image stabilization can be returned to the state before each correction processing, that is, the state when the image is formed on the image sensor 12a of the image capture element unit 12.

[0293] The parameters of lens distortion correction, keystone distortion correction, and focal plane distortion correction are collectively referred to as optical distortion correction parameters because they are distortion correction processing for the case where the optical image from the subject itself is an image captured in an optically distorted state, and each parameter is intended to be used for optical distortion correction.

[0294] That is, when inverse correction is performed by using these parameters, image data subjected to lens distortion correction, keystone distortion correction, and focal plane distortion correction can be returned to the state before optical distortion correction.

[0295] The timing information TM in the metadata includes information on exposure time (shutter speed), exposure start timing, readout time (curtain speed), number of exposure frames (long-second exposure information), IMU sampling offset, and frame rate.

[0296] In the image processing of this embodiment, these are mainly used to associate the rows of each frame with the IMU data.

[0297] However, even when the image sensor 12a is a CCD or a global shutter type CMOS, correction can be performed according to the exposure center by using the exposure start timing and the curtain speed when the exposure center is shifted by using the electronic shutter or the mechanical shutter.

[0298] As the camera parameters CP in the metadata, the angle of view (focal length), zoom position, and lens distortion information are described.

[0299] <5. Processing Example>

[0300] A description will be given of a processing example of the information processing device 70 serving as the image processing device TDx as the embodiment.

[0301] Figure 14 The procedures of various types of processing executed in the information processing device 70 as the image processing device TDx are shown, and the relationship between information used in each processing is shown.

[0302] Note that only the Figure 6 The functions of the jitter changing unit 100 shown in FIG. 1 and FIG. 2 perform at least the processing of step ST17 . However, steps ST11 , ST12 , ST13 , ST14 , ST18 , and ST19 may also be regarded as the processing of the jitter changing unit 100 .

[0303] Then, the process of step ST15 is performed according to the function of the jitter information processing unit 101 .

[0304] The process of step ST16 is performed according to the function of the jitter information normalization unit 105 .

[0305] The process of step ST40 is performed according to the function of the UI processing unit 103 .

[0306] The process of step ST41 is performed according to the function of the parameter setting unit 102 .

[0307] The process of step ST20 is performed according to the function of the associating unit 106 .

[0308] First, steps ST1 , ST2 , ST3 , and ST4 as preprocessing will be described.

[0309] Pre-processing is processing performed when the image file MF is imported.

[0310] The term "import" used here refers to setting an image file MF or the like that can be accessed by, for example, being taken into the storage unit 79 by the information processing device 70 as an image processing target, and performing preprocessing to expand the file so that image processing can be performed. For example, it does not refer to transfer from the image capture device 1 to the mobile terminal 2 or the like.

[0311] The CPU 71 imports an image file MF designated by a user operation or the like to be an image processing target, and executes processing related to metadata added to the image file MF as preprocessing. For example, the CPU 71 executes processing to extract and store metadata corresponding to each frame of an animation.

[0312] Specifically, in this preprocessing, metadata extraction (step ST1), merging of all IMU data (step ST2), metadata retention (step ST3), and conversion into quaternions (posture information of the image capture device 1) and retention (step ST4) are performed.

[0313] As metadata extraction in step ST1, the CPU 71 reads the target image file MF, and as referring to Figure 12 The metadata contained in the image file MF is extracted as described.

[0314] Note that part or all of steps ST1, ST2, ST3, and ST4 may be performed on the image source VS side such as the image capturing device 1. In this case, in pre-processing, contents after those processes described below are acquired as metadata.

[0315] The CPU 71 performs a merging process on the IMU data (angular velocity data (gyro samples) and acceleration data (accelerator samples)) among the extracted metadata in step ST2 .

[0316] This is a process of arranging and merging all IMU data associated with all frames in a time series order and constructing IMU data corresponding to the entire sequence of the animation.

[0317] Then, integration processing is performed on the combined IMU data to calculate, store, and retain the quaternion QD representing the posture of the image capture device 1 at each time point over the animation sequence. Calculating the quaternion QD is an example.

[0318] Note that the quaternion QD can be calculated using only the angular velocity data.

[0319] The CPU 71 performs processing to retain the metadata other than the IMU data extracted in step ST3, namely, the coordinate transformation parameters HP, timing information TM, and camera parameters CP. In other words, the coordinate transformation parameters HP, timing information TM, and camera parameters CP are stored in a state corresponding to each frame.

[0320] By performing the above-described preprocessing, the CPU 71 prepares to perform various types of image processing, including dithering change, on the image data received as the image file MF.

[0321] Figure 10 The steady-state processing in indicates image processing that is performed as a target on the image data of the image file MF subjected to the preprocessing as described above.

[0322] CPU 71 performs one-frame extraction of animation (step ST11), cancellation of internal correction of the image capture device (step ST12), pasting to the celestial sphere model (step ST13), synchronization processing (step ST14), jitter information adjustment / generation (step ST15), jitter information normalization (step ST16), jitter change (step ST17), plane projection and cutting (step ST18), output area designation (step ST19) and associated recording / reproduction (step ST20).

[0323] When reproducing the image of the image file MF, the CPU 71 executes the processing of steps ST11 to ST20 for each frame.

[0324] In step ST11, the CPU 71 decodes one frame of the moving image (image file MF) along the frame number FN. Then, it outputs the image data PD (#FN) of one frame. Note that "(#FN)" represents the frame number and indicates information corresponding to the frame.

[0325] Note that in the case where the animation has not been subjected to encoding processing such as compression, the decoding processing in step ST11 is unnecessary.

[0326] In step ST12 , the CPU 71 executes a process of canceling the internal correction performed by the image capturing apparatus 1 on the image data PD (#FN) for one frame.

[0327] For this purpose, the CPU 71 performs inverse correction of the correction performed by the image capture device 1, referring to the coordinate transformation parameters HP(#FN) stored in association with the frame number (#FN) during preprocessing. Consequently, image data iPD(#FN) is obtained in a state where lens distortion correction, keystone correction, focal plane distortion correction, electrical image stabilization, and optical image stabilization in the image capture device 1 have been canceled. In other words, this is image data in which shake removal and the like performed by the image capture device 1 have been canceled, leaving the effects of shake, such as camera shake, intact. This is because the correction processing during image capture is canceled to achieve a state before correction, and more accurate shake removal and shake addition using shake information (e.g., quaternion QD) during image capture are performed.

[0328] However, it is not necessary to perform the process of canceling the internal correction of the image capturing device as step ST 12. For example, the process of step ST 12 may be skipped, and the image data PD (#FN) may be output as it is.

[0329] In step ST13, the CPU 71 pastes the image data iPD(#FN) of one frame or the image data PD(#FN) of one frame to the celestial sphere model. At this time, the camera parameters CP(#FN) stored corresponding to the frame number (#FN), namely, the angle of view, zoom position, and lens distortion information, are referenced.

[0330] Figure 15 Shows the outline pasted onto the celestial sphere model.

[0331] Figure 15 A shows image data iPD. Note that there is also a case of image data PD, but the same applies hereinafter.

[0332] The image height h is the distance from the center of the image. The circles in the figure represent the positions where the image height h becomes equal.

[0333] The image sensor surface and incident angle in the frame are calculated from the viewing angle, zoom position and lens distortion information of the frame of the image data iPD. The relationship between "data 0" ... "data N-1" is set as "data 0" ... "data N-1" at each position on the image sensor surface. Then, from "data 0" ... "data N-1", the relationship is expressed as follows Figure 15 B shows the image height h and the incident angle A one-dimensional graph of the relationship between . Angle of incidence is the angle of the beam (as viewed from the optical axis).

[0334] The one-dimensional graph is rotated once around the center of the captured image, and the relationship between each pixel and the incident angle is obtained.

[0335] Therefore, a mapping of each pixel of the image data iPD to the celestial sphere model MT is performed, such as Figure 15 Pixel G1 in C to pixel G2 on the celestial coordinates.

[0336] As described above, the captured image is pasted onto the surface of an ideal celestial sphere with lens distortion removed, resulting in an image (data) of a celestial sphere model MT. In this celestial sphere model MT, the parameters and distortion unique to the image capture device 1 that originally captured the image data iPD are removed, and the range visible to the ideal pinhole camera is the range pasted onto the surface of the celestial sphere.

[0337] Therefore, by rotating the image of the celestial sphere model MT in a predetermined direction in this state, shake changing processing as shake removal or shake addition can be realized.

[0338] Here, the posture information (quaternion QD) of the image capture device 1 is used for the shake change processing. For this purpose, the CPU 71 performs synchronization processing in step ST14.

[0339] In the synchronization process, a process of specifying and acquiring the quaternion QD (#LN) appropriate for each line is executed corresponding to the frame number FN. Note that "(#LN)" represents a line number in a frame and indicates information corresponding to the line.

[0340] Note that the reason why the quaternion QD(#LN) is used for each row is that, in a case where the image sensor 12 a is of a CMOS type and rolling shutter image capturing is performed, the amount of shake differs for each row.

[0341] For example, in the case where the image sensor 12 a is of CCD type and performs global shutter image capturing, it is sufficient to use the quaternion QD(#FN) in units of frames.

[0342] Note that even in the case of a global shutter of a CCD or CMOS as the image sensor 12a, the center of gravity shifts when an electronic shutter is used (the same applies to a mechanical shutter), and therefore, it is preferable to use a quaternion with timing at the center of the exposure period of the frame (according to the shutter speed shift of the electronic shutter).

[0343] Here, blur occurring in an image is considered.

[0344] Blurring occurs in images due to relative motion between the image capture device and the subject within the same frame. That is, image blur is caused by jitter during exposure time. The longer the exposure time, the greater the effect of blurring.

[0345] In electrical image stabilization, when a method of controlling the image range cut for each frame is used, "jitter" occurring between frames can be reduced / eliminated, but relative jitter within the exposure time cannot be reduced by this electrical image stabilization.

[0346] Furthermore, when changing the cropping area through image stabilization, pose information for each frame is used. However, if pose information deviates from the center of the exposure period, such as at the start or end of the exposure period, the direction of shake during the exposure time based on the pose is offset, and blurring is easily noticeable. Furthermore, in a CMOS rolling shutter, the exposure period varies for each row.

[0347] Therefore, in the synchronization process in step ST14 , for each frame of image data, the quaternion QD is acquired with reference to the timing of the exposure gravity center of each line.

[0348] Figure 16 The synchronization signal cV in the vertical period of the image capture device 1 , the synchronization signal sV of the image sensor 12 a generated from the synchronization signal cV, and the sampling timing of the IMU data are shown, and the exposure timing range 120 is also shown.

[0349] The parallelogram schematically shows the exposure timing range for each row of a frame when exposure time t4 is set using the rolling shutter method. Also shown are the time offset t0 between the synchronization signals cV and sV, the IMU sampling timing offset t1, the read start timing t2, the read time (curtain speed) t3, and the exposure time t4. Note that read start timing t2 is the time after a predetermined time t2of has passed from the synchronization signal sV.

[0350] Each IMU data obtained at each IMU sampling timing is associated with a frame. For example, the IMU data in period FH1 is metadata associated with the current frame indicating the exposure period in the parallelogram, and the IMU data in period FH2 is metadata associated with the next frame. However, by Figure 14 In step ST2, all IMU data are merged, the association between each frame and the IMU data is released, and the IMU data can be managed in a time series.

[0351] In this case, IMU data corresponding to the exposure center of gravity of each line of the current frame (timing of the dotted line W) is specified. If the time relationship between the IMU data and the effective pixel area of ​​the image sensor 12a is known, it can be calculated.

[0352] Therefore, by using the information that can be acquired as the timing information TM corresponding to the frame (#FN), the IMU data corresponding to the exposure center of gravity of each line (the timing of the broken line W) is specified.

[0353] That is, it is information of exposure time, exposure start timing, read time, number of exposure frames, IMU sampling offset, and frame rate.

[0354] Then, the quaternion QD calculated from the IMU data of the exposure center of gravity is designated and set as the quaternion QD (#LN) serving as the posture information of each row.

[0355] This quaternion QD(#LN) is supplied to the dither information adjustment / generation process in step ST15.

[0356] In the dither information adjustment / generation process, the CPU 71 adjusts or generates the quaternion QD according to the dither change parameter PRM that has been input.

[0357] The jitter change parameter PRM is a parameter input according to a user operation or a parameter generated by automatic control.

[0358] The jitter changing parameter PRM is generated in the parameter setting process of step ST41 .

[0359] Furthermore, through the UI processing in step ST40 , the user can perform an operation input for instructing a change in shake, that is, perform an operation for instructing to add shake as a shake generation operation or an operation for instructing the degree of shake removal, etc.

[0360] For example, the user can input the shake changing parameter PRM itself or select parameters to add an arbitrary degree of shake to the image. In this case, in step ST41, the shake changing parameter PRM according to the user input is set and used for the shake information adjustment / generation process.

[0361] Furthermore, as parameter setting processing in step ST41, the CPU 71 can generate and use the jitter change parameter PRM for the jitter information adjustment / generation processing through automatic control according to image analysis, image type or user selection operation of the jitter model, and use the jitter change parameter PRM.

[0362] Then, in the shake information adjustment / generation processing in step ST15, the CPU 71 adjusts or generates and outputs the shake change quaternion eQD for adding shake to the image or reducing the shake amount as output shake information based on the shake change parameter PRM and the quaternion QD as the shake information at the time of image capture.

[0363] refer to Figure 17 、 Figure 18 and Figure 19 , an adjustment example of using the quaternion QD as the shake information and the shake change parameter PRM at the time of image capture (ie, a generation example of the shake change quaternion eQD) will be described.

[0364] Figure 17 An example is shown in which a dither change quaternion eQD is generated according to a band gain instruction using a dither change parameter PRM.

[0365] The frequency band is a band of jitter frequencies. For ease of description, it is assumed that the band is divided into three bands: a low band, a middle band, and a high band. Of course, this is just an example, and the number of bands only needs to be two or more.

[0366] As the jitter change parameter PRM, a low gain LG, a middle gain MG, and a high gain HG are provided.

[0367] Figure 17 The adjustment processing system in includes a low-pass filter 41 , a medium-pass filter 42 , a high-pass filter 43 , gain operation units 44 , 45 and 46 , and a synthesis unit 47 .

[0368] The "quaternion QDs for dithering" is input to the adjustment processing system. This is the conjugation of the quaternion QDs as dithering information during image capture.

[0369] The values ​​q of the current frame and the previous and next predetermined frames as the quaternion QDs for dithering are input to the low-pass filter 41 to obtain the low component q low .

[0370] [Expression 1]

[0371] q low =mean(q,n)

[0372] The gain operation unit 44 adds the low component q low Gives low gain LG.

[0373] Note that the average(q,n) in the expression represents the average of the n values ​​before and after q.

[0374] And, the value q of the quaternion QDs for dithering is input to the middle pass filter 42 to obtain the middle component q mid .

[0375] [Expression 2]

[0376]

[0377] Where n>m

[0378] Note that q* low It's q low The conjugation of .

[0379] And, “×” is quaternion multiplication.

[0380] The gain operation unit 45 gives the intermediate gain MG to the intermediate component q mid.

[0381] And, the value q of the quaternion QDs for dithering is input to the high-pass filter 43 to obtain the high component q high .

[0382] [Expression 3]

[0383]

[0384] Note that q* mid It is q mid The conjugation of .

[0385] The gain operation unit 46 gives a high gain HG to the high component q high .

[0386] These gain operation units 44, 45 and 46 assume that the input is "q in ”.

[0387] [Expression 4]

[0388]

[0389] In this case, the next “q” is output as “θ′=θ*gain” out ”.

[0390] (Gains are low gain LG, medium gain MG, and high gain HG.)

[0391] [Expression 5]

[0392]

[0393] By such gain operation units 44, 45 and 46, the low component q is obtained to which the low gain LG, the middle gain MG and the high gain HG are respectively given. low , medium component q mid and high-component q high These are combined by the combining unit 47 to obtain the value q mixed .

[0394] [Expression 6]

[0395] q mixed =q low ×q mid ×q high

[0396] Note that "×" is quaternion multiplication.

[0397] The value q obtained from this mixed Change the value of the quaternion eQD to dither.

[0398] Although the above is an example of band division, it is also conceivable to generate a dither change quaternion eQD that gives a gain according to the dither change parameter PRM without band division.

[0399] Next, Figure 18 An example is shown in which a dither change quaternion eQD is generated according to an instruction of a gain in each direction by a dither change parameter PRM.

[0400] The direction is the direction of shaking, ie, the direction of pan, pitch, and roll.

[0401] As the shake change parameters PRM, a pan gain YG, a pitch gain PG, and a roll gain RG are given.

[0402] Figure 18 The adjustment processing system in φA includes a pan component extraction unit 51 , a pitch component extraction unit 52 , a roll component extraction unit 53 , gain operation units 54 , 55 , and 56 , and a synthesis unit 57 .

[0403] The pan component extraction unit 51 , the pitch component extraction unit 52 , and the roll component extraction unit 53 are supplied with information on the pan axis, the pitch axis, and the roll axis, respectively.

[0404] The corresponding values ​​q of the current frame and the previous and next predetermined frames as the quaternion QDs for shaking are respectively input to the pan component extraction unit 51, the pitch component extraction unit 52, and the roll component extraction unit 53 to obtain the pan component q yaw , pitch component q pitch and the roll component q roll .

[0405] Each component extraction process assumes that the next "q in " as input.

[0406] [Expression 7]

[0407]

[0408] u=[u x u y u z ]

[0409] u is a unit vector indicating the direction of an axis such as a pan axis, a pitch axis, or a roll axis.

[0410] In this case, the next “q out ”.

[0411] [Expression 8]

[0412]

[0413] For the panning component q obtained by this component extraction yaw , pitch component q pitch and the roll component q roll , the gain operation units 54, 55 and 56 respectively provide the pan gain YG, the pitch gain PG and the roll gain RG.

[0414] Then, the pan component q is subjected to the gain operation yaw , pitch component q pitch and the roll component q roll The value q is synthesized by the synthesizing unit 47 to obtain mixed .

[0415] [Expression 9]

[0416] q mixed =q yaw ×q pitch ×q roll

[0417] Note that "×" in this case is also a quaternion multiplication.

[0418] The value q obtained from this mixed Change the value of the quaternion eQD to dither.

[0419] Figure 19 An example of combining frequency bands and directions is shown.

[0420] The adjustment processing system includes a low-pass filter 41 , a medium-pass filter 42 , a high-pass filter 43 , direction processing units 58 , 59 , and 60 , gain operation units 44 , 45 , and 46 , and a synthesis unit 61 .

[0421] According to the shake change parameter PRM, a low gain LG, a middle gain MG, a high gain HG, and a pan gain YG, a pitch gain PG, and a roll gain RG, not shown, are given.

[0422] In this adjustment processing system, the values ​​q of the current frame and the preceding and following frames are fed as quaternions QDs for dithering to low-pass filter 41, mid-pass filter 42, and high-pass filter 43, respectively, to obtain their respective band components. These band components are then input to direction processing units 58, 59, and 60.

[0423] Each of the direction processing units 58, 59 and 60 is assumed to include Figure 18 The pan component extraction unit 51, the pitch component extraction unit 52, the roll component extraction unit 53, the gain operation units 54, 55 and 56, and the synthesis unit 57 in FIG.

[0424] That is, the direction processing unit 58 separates the low component of the quaternion QDs for shake into components in the pan, roll, and pitch directions, performs gain operation by using the pan gain YG, pitch gain PG, and roll gain RG, and then synthesizes them.

[0425] The direction processing unit 59 separates the intermediate component of the quaternion QDs for shake into components in the pan direction, the roll direction, and the pitch direction, similarly performs gain operation, and then synthesizes them.

[0426] The direction processing unit 60 separates the high component of the quaternion QDs for shaking into components in the pan direction, the roll direction, and the pitch direction, similarly performs gain operation, and then synthesizes them.

[0427] The outputs of these direction processing units 58, 59 and 60 are respectively supplied to gain operation units 44, 45 and 46 and are respectively given low gain LG, medium gain MG and high gain HG. Then, they are combined by a combination unit 61 and output as the value of the dither change quaternion eQD.

[0428] In the above description Figure 19 In the example of , after first dividing into frequency bands, directional processing is applied to each band component, but this can be reversed. That is, after first dividing into each direction, frequency band processing can be applied to each directional component.

[0429] exist Figure 10 In step ST15 , for example, the dither change quaternion eQD is generated by the above-described processing example.

[0430] However, the above description can be viewed as an example of a case where a jitter-changing quaternion eQD is generated by adjusting the quaternion QD, which serves as jitter information during image capture. For example, this becomes an example of a process in which jitter is partially removed. Alternatively, this becomes an example of a process in which jitter is added to the original jitter.

[0431] In the case where a jitter-free image (e.g., image data iPD) is assumed as the target of jitter change and jitter addition of newly added jitter is performed, it is only necessary to generate a jitter change quaternion eQD based on the jitter change parameter PRM without using the quaternion QD, which is the jitter information at the time of image capture.

[0432] Incidentally, in the above example, the low gain LG, the middle gain MG, and the high gain HG are given as the jitter changing parameters PRM, and these are the jitter changing parameters PRM according to the result of detecting a user operation as, for example, UI processing.

[0433] In this case, if the amount of change in jitter (jitter remaining amount) can be input and recognized as an angle from the user's perspective, it is possible to provide an intuitively understandable operation. The angle mentioned here refers to the rotation angle on the celestial sphere model MT. It is preferable to be able to operate in angles even when jitter is added.

[0434] As the desired rotation, that is, the amount of jitter, the unit vector a representing its rotation axis is expressed as:

[0435] a=(a x a y a z ) T

[0436] The rotation angle (rotation amount) is θ[rad],

[0437] The quaternion q corresponding to the rotation is as follows (Expression 10).

[0438] [Expression 10]

[0439]

[0440] For example, in the parameter setting process of step ST41 , the user's angle input is converted into a quaternion q regarding rotation in this manner, and this only needs to be applied to the above-mentioned (Expression 4) (Expression 7) and the like.

[0441] The shake change quaternion eQD generated in the shake information adjustment / generation process of step ST15 is supplied to the shake information normalization process of step ST16 .

[0442] In the dither information normalization process, the dither change quaternion eQD is normalized by using a normalization coefficient α.

[0443] First, the normalized jitter information SCI is obtained by the following relationship:

[0444] Normalized dither information SCI=(dither information)×α, and a specific example of “dither information” in this expression becomes the dither change quaternion eQD Then, in step ST16 , the normalized quaternion nQD is obtained as the normalized dither information SCI.

[0445] Now, the value of the dither-changed quaternion eQD as the adjusted / generated "dither information" is assumed to be the next "q org ”.

[0446] [Expression 11]

[0447]

[0448] Then, the value of the normalized quaternion nQD "q norm "as follows.

[0449] [Expression 12]

[0450]

[0451] Note that the normalization coefficient α is obtained by “(target image angle of view) / (reference angle of view)” and “(image angle of view after cropping) / (captured image angle of view)” as described above.

[0452] The normalized quaternion nQD generated as described above is supplied to the dither changing process in step ST17.

[0453] The dither change in step ST17 can be regarded as, for example, adding dither by applying the normalized quaternion nQD to the image data iPD in a state where the dither is stopped, or adjusting the image stabilization amount of the image data PD with dither.

[0454] In the image stabilization in step ST17, the CPU 71 adds or partially removes the shake by rotating the image of the celestial sphere model MT to which the frame image is attached in step ST13 using the normalized quaternion nQD (#LN) for each line. The image of the celestial sphere model hMT in which the shake has been changed is sent to the process of step ST18.

[0455] Then, in step ST18 , the CPU 71 projects the image of the celestial sphere model hMT, in which the jitter has been changed, onto a plane, and cuts the image so that an image (output image data oPD) that has been subjected to a jitter change is obtained.

[0456] In this case, the jitter is reduced by rotating the sphere model MT. Using the sphere model MT prevents trapezoidal distortion even when any portion is clipped. Furthermore, as described above, the visible range of an ideal pinhole camera in the sphere model MT is the range adhered to the surface of the celestial sphere, eliminating lens distortion. Since the sphere model MT is rotated according to the normalized quaternion nQD(#LN) for each line, focal plane distortion correction is also eliminated.

[0457] Furthermore, since the quaternion QD(#LN) corresponds to the exposure center of gravity of each line, blur is not visible in the image.

[0458] The association between the image after being subjected to the planar projection in step ST18 and the celestial sphere model MT is as follows.

[0459] Figure 20A shows an example of the rectangular coordinate plane 131 subjected to planar projection. The coordinates of the image subjected to planar projection are assumed to be (x, y).

[0460] like Figure 20 B, the coordinate plane 131 is arranged (normalized) in three-dimensional space so as to contact the center directly above the celestial sphere model MT. That is, the center of the coordinate plane 131 is arranged at a position that coincides with the center of the celestial sphere model MT and contacts the celestial sphere model MT.

[0461] In this case, the coordinates are normalized based on the zoom factor and the size of the cropped area. For example, Figure 20 As shown in A, in the case where the horizontal coordinates of the coordinate plane 131 are 0 to outh and the vertical coordinates are 0 to outv, outh and outv are image sizes, and then, for example, the coordinates are normalized by the following expression.

[0462] [Expression 13]

[0463]

[0464]

[0465] z norm =1

[0466] where r = min(outh, outv) / 2

[0467] In the above (Expression 13), min(A, B) is a function that returns the smaller value of A and B. And, "zoom" is a parameter for controlling zoom.

[0468] And, x norm 、y norm and z norm are the normalized x, y, and z coordinates.

[0469] By each expression of the above-mentioned (Expression 13), the coordinates of the coordinate plane 131 are normalized to the coordinates on the spherical surface of the hemisphere with a radius of 1.0.

[0470] To rotate to get the orientation of the clipping region, as Figure 21 Coordinate plane 131 is rotated by a rotation matrix operation as shown in FIG. That is, the rotation matrix shown in FIG. 14 is used to perform rotation using the pan angle, tilt angle, and roll angle. Here, the pan angle is the rotation angle of the coordinates around the z-axis. The tilt angle is the rotation angle of the coordinates around the x-axis, and the roll angle is the rotation angle of the coordinates around the y-axis.

[0471] [Expression 14]

[0472]

[0473] In the above (Expression 14), "Rt" is the tilt angle, "Rr" is the roll angle, and "Rp" is the pan angle. And, (x rot ,y rot , z rot ) are the rotated coordinates.

[0474] The coordinate (x rot ,y rot , z rot ) is used to calculate corresponding points on the celestial sphere in perspective projection.

[0475] like Figure 21 As shown in B, the coordinate plane 131 is perspectively projected onto the surface of the celestial sphere (region 132). That is, when a straight line is drawn from the coordinate plane to the center of the celestial sphere, the point where it intersects the spherical surface is obtained. The calculation of each coordinate is as follows.

[0476] [Expression 15]

[0477]

[0478]

[0479]

[0480] In (Expression 15), x sph 、y sph and z sph y are coordinates obtained by projecting the coordinates on the coordinate plane 131 onto the surface of the celestial sphere model MT.

[0481] Image data on which planar projection is performed in this relationship is obtained.

[0482] For example, in Figure 14 In step ST19, a clipping area for the image projected onto the plane using the above-described technique is set.

[0483] In step ST19 , based on the tracking process by image analysis (subject recognition) or the cropping area instruction information CRC according to the user operation, the cropping area information CRA in the current frame is set.

[0484] For example, Figure 22 A and Figure 22 B shows the cropping area information CRA set for an image of a certain frame in the frame state.

[0485] Such cropping area command information CRC is set for each frame.

[0486] Note that the cropping area information CRA also reflects the user's or automatic control's indication of the aspect ratio of the image.

[0487] The clipping region information CRA is reflected in the process of step ST18. That is, as described above, the region corresponding to the clipping region information CRA is planar-projected onto the celestial sphere model MT, and the output image data oPD is obtained.

[0488] The output image data oPD thus obtained is image data subjected to the dither changing process in step ST16 .

[0489] By executing each frame Figure 14 When the output image data oPD is reproduced and displayed, an image with added jitter is displayed as jittered. Therefore, when the user performs an operation for inputting a jitter change parameter PRM, an image with added jitter according to the user's intention is obtained. This image data is displayed or stored as an image subjected to jitter.

[0490] Here, in the present embodiment, for example, the associated recording / reproduction is performed as step ST20. For example, the CPU 71 may perform associated recording on each frame (for example, Figure 5 The data is recorded in the storage unit 79 or the removable recording medium 81), and can be reproduced / transmitted and output as needed.

[0491] As the associated recording process for each frame, for example, normalized shake information SCI is recorded in association with image data in units of frames, and can also be recorded in association with image capture shake information and shake change information SMI.

[0492] First, as image data in frame units, in step ST12, CPU 71 performs recording processing on the image data iPD (or image data PD) from which the internal correction of the image capture device has been canceled, and in response thereto, performs recording processing of the normalized jitter information SCI, i.e., the normalized quaternion nQD.

[0493] Furthermore, the CPU 71 can perform recording processing of the quaternion QD of the frame, the timing information TM, and the camera parameter CP as image-capturing-time shake information associated with the image data iPD.

[0494] Figure 23 Metadata MTD1 supplied from the image source VS to the image processing device TDx and metadata MTD2 supplied from the image processing device TDx to the image processing device TDy are shown, and include normalized quaternions nQD. Information corresponding to the metadata MTD2 is associated with each frame of the image data iPD.

[0495] Furthermore, the CPU 71 may perform a recording process of the shake change parameter PRM as the shake change information SMI associated with the image data iPD. Alternatively, the shake change quaternion eQD may be recorded instead of or together with the shake change parameter PRM.

[0496] The image data iPD and normalized shake information SCI (normalized quaternion nQD) of each frame recorded in association in this manner and the like are transmitted to an external device and made available.

[0497] Figure 2 1 shows that the image data VD2, metadata MTD2, normalized dither information SCI, and dither change information SMI are transmitted from the image processing device TDx to the image processing device TDy, but in Figure 14 In the case of the example of , the image data VD2 is image data including the image data iPD. And, the metadata MTD2 includes, for example, normalized jitter information SCI as Figure 23 The content shown.

[0498] Therefore, the image processing device TDy becomes able to easily perform reproduction of shake addition and shake portion removal based on the normalized shake information SCI applied in the past in the image processing device TDx, adjustment and correction of shake addition, and accurate elimination of shake change.

[0499] <6. Overview and Modification Examples>

[0500] In the above-described embodiment, the following effects can be obtained.

[0501] The image processing device TDx of this embodiment includes a shake information processing unit 101 that outputs output shake information for image data (input image data) constituting an animation (step ST15). The image processing device TDx also includes a shake information normalization unit 105 that obtains normalized shake information by normalizing the output shake information based on view angle information (step ST16). The image processing device TDx also includes a shake changing unit 100 that performs a shake changing process that changes the shake state of the image data using the normalized shake information SCI (step ST17).

[0502] This enables jitter addition (or jitter remaining due to partial jitter removal) that is appropriate for the jitter amount of the output image data oPD. That is, jitter is no longer stronger than expected, or the jitter amount is no longer too small, depending on the angle of view after cropping, and jitter addition (or jitter remaining due to partial jitter removal) that is in accordance with the user's intention can be achieved regardless of the angle of view.

[0503] In the embodiment, an example is described in which the jitter changing unit 100 performs jitter addition on the input image data based on the normalized jitter information SCI as the jitter changing process.

[0504] For example, in the image processing device TDx, a user performs an operation to instruct the amount of jitter as a jitter generation. In response to this, a jitter change parameter PRM is input, a jitter change quaternion eQD is generated and normalized, and jitter change processing is performed using the normalized quaternion nQD to add jitter as a jitter generation. For example, jitter can be added as a jitter generation to a jitter-free image, and further jitter can be added to an image with jitter. This enables diversification of image expression. Furthermore, in this case, an appropriate amount of jitter can be added regardless of the viewing angle of the output image data oPD.

[0505] In the embodiment, an example in which the shake changing unit 100 performs shake portion removal of input image data based on the normalized shake information SCI is described as the shake changing process.

[0506] In step ST17, the image processing device TDx can generate an image with reduced shake by, for example, performing partial shake removal. This is suitable, for example, for situations where it is desirable to suppress, rather than eliminate, severe image shake. In this case as well, by performing shake modification using the normalized quaternion nQD, the residual degree of shake no longer varies depending on the viewing angle, and it is possible to achieve residual shake in accordance with the user's intention, regardless of the viewing angle.

[0507] In the embodiment, an example is described in which the angle-of-view information used for normalization is the ratio between the angle of view of the target image to be subjected to the dither changing process and the reference angle of view.

[0508] That is, a certain reference viewing angle is determined, and the jitter information is normalized by the ratio of the viewing angle to the reference viewing angle. This makes it possible to express the degree of jitter at the reference viewing angle and the equivalent jitter regardless of the viewing angle.

[0509] In the embodiment, an example is described in which the angle of view information used for normalization is the ratio between the angle of view after the cutout and the angle of view at the time of image capture of the image to be subjected to the dither change process.

[0510] That is, when generating an output image by cropping from a captured image, the shake information about the cropped image is normalized with reference to the viewing angle of the captured image. This makes it possible to express the perceived shake in the cropped image as visually equivalent to the degree of shake in the captured image.

[0511] In the embodiment, the example including the parameter setting unit 102 for setting the dither changing parameter regarding the dither changing process has been described.

[0512] The shake changing process changes the state of shake by reducing shake occurring in the animation or adding shake, and is performed based on the shake changing parameter PRM set by the parameter setting unit 102. For example, by generating the shake changing parameter by using archives of various shakes or by using user input, it is possible to add a shake as newly generated shake to an image instead of the shake at the time of image capture.

[0513] In the embodiment, an example is described including the UI processing unit 103 that detects operation information about the shake change and the parameter setting unit 102 that sets the shake change parameter about the shake change process based on the operation information detected by the UI processing unit 103 .

[0514] This makes it possible to provide the user with an operating environment for causing execution of image processing for adding jitter to an image or partially removing jitter from an image.

[0515] In particular, by enabling parameters of dither change to be visually manipulated on a UI screen, a dither manipulation environment that is easy for the user to understand can be provided.

[0516] In the embodiment, an example in which the shake change parameter PRM is obtained by converting information of the angle detected by the UI processing unit 103 into the shake amount is described.

[0517] Specifically, the parameter setting unit 102 converts the shake amount specified in angles on the UI screen, etc. into quaternions. This allows the user to specify the changed shake amount through an operation that makes it easy to image the shake amount by angles.

[0518] In the embodiment, an example is described in which the jitter information processing unit 101 generates the output jitter information (for example, the jitter change quaternion eQD) based on the jitter change parameter PRM for specifying the processing amount of the jitter changing process.

[0519] This enables the execution of a process of adding shake to a shake-free image or increasing shake, and enables the diversification of image expression.

[0520] In the embodiment, an example is described in which the shake information processing unit 101 adjusts the shake information (quaternion QD) at image capture time based on the shake change parameter PRM for specifying the processing amount of the shake change process and obtains output shake information (e.g., shake change quaternion eQD).

[0521] This enables execution of processing for generating an image with a sense of reality by reducing shake applied to an image at the time of image capture and retaining the shake without impairing visibility, and enables diversification of image expression.

[0522] In the embodiment, an example is described in which the image-capturing-time shake information is the posture information of the image capturing apparatus 1 when input image data is image-captured by the image capturing apparatus 1 .

[0523] That is, the quaternion QD obtained as posture information from the IMU data is used as image capture shake information. This makes it possible to perform shake change based on information for appropriately expressing shake at the time of image capture.

[0524] In the embodiment, an example is described in which the shake changing unit 100 pastes each frame of input image data of a processing target to the celestial sphere model MT and performs shake change by rotating it with normalized shake information SCI (normalized quaternion nQD) corresponding to each frame.

[0525] By performing shake change correction on the celestial sphere model MT by rotating it to increase or decrease the shake of each frame, shake change can be performed without causing keystone distortion. Therefore, a high-quality image with less distortion can be obtained as an image subjected to shake generation.

[0526] The image processing device TDx of this embodiment includes an associating unit 106 that performs processing to associate image data with normalized dither information SCI (e.g., normalized quaternion nQD) (step ST20). This enables, for example, the image processing device TDy to understand what type of dither change has been performed by the dither change processing in the image processing device TDx.

[0527] Then, in the image processing device TDy, it becomes possible to perform processing such as re-performing the shake changing processing as the shake generation, for example, adjusting the shake degree of the shake generation that has been performed and eliminating the shake generation.

[0528] Furthermore, in the image processing device TDy, it is also possible to perform blur removal after canceling blur generation performed by the image processing device TDx.

[0529] That is, each image processing device enables the shake state to be changed arbitrarily and differently.

[0530] Note that, for example, it is conceivable that the image data and normalized jitter information associated with each other are recorded in the same file. For example, a file including the image data (and audio data) constituting the animation, metadata, and normalized jitter change information can be configured. For example, the normalized jitter change information can be included as metadata accompanying the image data.

[0531] However, they are not necessarily recorded in the same file, etc. Even if the normalized jitter information is recorded in a data file different from the data file of the image data, it is sufficient to use a form that can be later associated with each frame of the image data. In other words, any recording form can be used as long as the image processing device TDy at the transmission destination can obtain the information in an associated manner.

[0532] Furthermore, the image capture shake information can also be associated.

[0533] The program of the embodiment is used to cause a CPU, a DSP or a device including them to execute the following Figure 14 The processing procedure is shown.

[0534] That is, the program of the embodiment is a program that causes the information processing device to perform jitter information processing (ST15) of outputting output jitter information for input image data constituting an animation, jitter information normalization processing (ST16) of obtaining normalized jitter information by normalizing the output jitter information based on view angle information, and jitter changing processing (ST17) of changing the jitter state of the input image data by using the normalized jitter information.

[0535] Such a program enables the above-described image processing device TDx to be realized in a device such as the mobile terminal 2 , the personal computer 3 , or the image capturing device 1 .

[0536] Such a program may be recorded in advance in an HDD as a recording medium built into a device such as a computer apparatus or a ROM or the like in a microcomputer having a CPU.

[0537] Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disc read only memory (CD-ROM), a magneto-optical disc (MO), a digital versatile disc (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, and a memory card. Such a removable recording medium may be provided as a so-called software package.

[0538] Also, such a program can be installed from a removable recording medium to a personal computer or the like, and can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0539] Furthermore, this program is applicable to a wide range of image processing devices TDx of the present invention. For example, by downloading the program to a personal computer, portable information processing device, mobile phone, game console, video equipment, personal digital assistant (PDA), etc., the personal computer or the like can be used as the image processing device of the present disclosure.

[0540] Note that the effects described in this specification are merely examples and are not limited thereto, and there may be other effects.

[0541] Note that the present technology can also have the following configurations. (1)

[0543] An image processing device, comprising:

[0544] a jitter information processing unit configured to output output jitter information for input image data constituting an animation;

[0545] a jitter information normalizing unit configured to obtain normalized jitter information by normalizing the output jitter information based on viewing angle information; and

[0546] The shake changing unit is configured to perform a shake changing process of changing a shake state of the input image data by using the normalized shake information. (2)

[0548] The image processing device according to (1), wherein

[0549] The jitter changing unit performs jitter addition on the input image data as jitter changing processing based on the normalized jitter information. (3)

[0551] The image processing device according to (1) or (2), wherein

[0552] The shake changing unit performs shake portion removal of the input image data as a shake changing process based on the normalized shake information. (4)

[0554] The image processing device according to any one of (1) to (3), wherein

[0555] The angle of view information is a ratio between the angle of view of a target image to be subjected to dither changing processing and a reference angle of view. (5)

[0557] The image processing device according to any one of (1) to (3), wherein

[0558] The angle of view information is a ratio between the angle of view of the image to be subjected to the dither change process after being cut out and the angle of view at the time of image capture. (6)

[0560] The image processing device according to any one of (1) to (5), further comprising:

[0561] The parameter setting unit is configured to set a jitter change parameter related to the jitter change process. (7)

[0563] The image processing device according to any one of (1) to (6), further comprising:

[0564] a user interface processing unit configured to detect operation information regarding a change in jitter; and

[0565] The parameter setting unit is configured to set a dither changing parameter related to the dither changing process based on the operation information detected by the user interface processing unit. (8)

[0567] The image processing device according to (7), wherein

[0568] The shake change parameter is obtained by converting information of the angle detected by the user interface processing unit into a shake amount. (9)

[0570] The image processing device according to any one of (1) to (8), wherein

[0571] The jitter information processing unit generates output jitter information based on a jitter change parameter that specifies a processing amount of the jitter change process. (10)

[0573] The image processing device according to any one of (1) to (9), wherein

[0574] The shake information processing unit obtains output shake information by adjusting the image-capturing-time shake information based on a shake change parameter that specifies a processing amount of the shake change process. (11)

[0576] The image processing device according to (10), wherein

[0577] The image-capturing-time shake information is posture information of the image capturing device when image capturing of input image data is performed by the image capturing device. (12)

[0579] The image processing device according to any one of (1) to (11), wherein

[0580] The shake changing unit pastes each frame of the input image data to the celestial sphere model, and performs shake changing processing by rotating each frame using normalized shake information corresponding to each frame. (13)

[0582] The image processing device according to any one of (1) to (12), further comprising:

[0583] The associating unit is configured to associate the input image data with the normalized dither information. (14)

[0585] An image processing method, wherein:

[0586] The image processing device performs the following processing:

[0587] a shake information process for outputting shake information for input image data constituting an animation;

[0588] jitter information normalization processing by normalizing the output jitter information based on the viewing angle information to obtain normalized jitter information; and

[0589] A shake changing process that changes the shake state of input image data by using normalized shake information. (15)

[0591] A program causing an information processing device to execute the following processing:

[0592] a shake information process for outputting shake information for input image data constituting an animation;

[0593] jitter information normalization processing by normalizing the output jitter information based on the viewing angle information to obtain normalized jitter information; and

[0594] A shake changing process that changes the shake state of input image data by using normalized shake information.

[0595] List of Reference Numerals

[0596] 1 Image capture device

[0597] 2 Mobile Terminals

[0598] 3 personal computers

[0599] 4 Servers

[0600] 5 Recording Media

[0601] 70 Information processing device

[0602] 71 CPU

[0603] 100 jitter change units

[0604] 100a Jitter Adding Unit

[0605] 100b jitter removal unit

[0606] 101 Jitter Information Processing Unit

[0607] 102 parameter setting unit

[0608] 103 UI processing unit

[0609] 105 Jitter information normalization unit

[0610] 106 associated units

Claims

1. An image processing device, comprising: a jitter information processing unit configured to output output jitter information for input image data constituting an animation; a jitter information normalization unit configured to obtain normalized jitter information by normalizing the output jitter information based on viewing angle information; and a shake changing unit configured to perform a shake changing process of changing a shake state of input image data by using normalized shake information, Here, the jitter changing unit performs jitter addition on the input image data as jitter changing processing based on the normalized jitter information.

2. The image processing apparatus according to claim 1, wherein: The shake changing unit performs shake portion removal of the input image data as a shake changing process based on the normalized shake information.

3. The image processing apparatus according to claim 1, wherein: The angle of view information is a ratio between the angle of view of a target image to be subjected to dither changing processing and a reference angle of view.

4. The image processing apparatus according to claim 1, wherein: The angle of view information is a ratio between the angle of view of the image to be subjected to the dither change process after being cut out and the angle of view at the time of image capture.

5. The image processing apparatus according to claim 1 , further comprising: The parameter setting unit is configured to set a jitter change parameter related to the jitter change process.

6. The image processing apparatus according to claim 1, further comprising: a user interface processing unit configured to detect operation information regarding a change in jitter; and The parameter setting unit is configured to set a dither changing parameter related to the dither changing process based on the operation information detected by the user interface processing unit.

7. The image processing apparatus according to claim 6, wherein: The shake change parameter is obtained by converting information of the angle detected by the user interface processing unit into a shake amount.

8. The image processing apparatus according to claim 1, wherein: The jitter information processing unit generates output jitter information based on a jitter change parameter that specifies a processing amount of the jitter change process.

9. The image processing apparatus according to claim 1, wherein: The shake information processing unit obtains output shake information by adjusting the image-capturing-time shake information based on a shake change parameter that specifies a processing amount of the shake change process.

10. The image processing apparatus according to claim 9, wherein: The image-capturing-time shake information is posture information of the image capturing device when image capturing of input image data is performed by the image capturing device.

11. The image processing apparatus according to claim 1, wherein: The shake changing unit pastes each frame of the input image data to the celestial sphere model, and performs shake changing processing by rotating each frame using normalized shake information corresponding to each frame.

12. The image processing apparatus according to claim 1, further comprising: The associating unit is configured to associate the input image data with the normalized dither information.

13. An image processing method, wherein: The image processing device performs the following processing: a shake information process for outputting shake information for input image data constituting an animation; Normalizing the jitter information by normalizing the output jitter information based on the viewing angle information to obtain normalized jitter information; as well as By a shake changing process of changing the shake state of input image data using normalized shake information, Here, dither addition is performed on input image data as dither changing processing based on normalized dither information.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, a method is implemented comprising the following steps: a shake information process for outputting shake information for input image data constituting an animation; Normalizing the jitter information by normalizing the output jitter information based on the viewing angle information to obtain normalized jitter information; as well as By a shake changing process of changing the shake state of input image data using normalized shake information, Here, dither addition is performed on input image data as dither changing processing based on normalized dither information.

Images