Image processing device, image processing method, and program

The image processing device and method allow users to adjust shaking components in moving images, addressing the limitations of existing technologies by enabling precise control over shake state changes, resulting in enhanced video quality and variety.

JP7586081B2Active Publication Date: 2024-11-19SONY GROUP CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021534552
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-07-24
Filing Date
2020-05-11
Publication Date
2024-11-19
Estimated Expiration
2040-05-11

AI Technical Summary

Technical Problem

Existing image processing technologies do not adequately address the need for users to easily adjust and enhance the shaking effects in moving images, limiting the quality and variety of images produced, especially in mobile devices and video content creation.

Method used

An image processing device and method that allows users to adjust shaking components in moving images by changing the shake state, including direction and frequency bands, through a user interface that sets parameters for shake change processing, enabling adjustments in yaw, roll, and pitch directions, as well as low and high frequency bands, and applying preset data to modify shaking patterns.

Benefits of technology

Enables users to easily adjust and enhance shaking effects in images, providing more diverse and high-quality video content by allowing precise control over shaking components, enhancing the realism and impact of video scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586081000013
    Figure 0007586081000013
  • Figure 0007586081000014
    Figure 0007586081000014
  • Figure 0007586081000015
    Figure 0007586081000015
Patent Text Reader

Abstract

Provided is an image processing device, wherein the following are carried out: shake modification processing by which shake-modified image data is obtained by modifying the state of shake of input image data that constitutes a moving image; user interface processing by which operation information related to the shake modification is acquired; and parameter setting processing by which parameters for the shake modification processing are set on the basis of the operation information acquired by the user interface processing.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and in particular to image processing for image shaking. [Background technology]

[0002] 2. Description of the Related Art Techniques for performing various types of image processing, such as correction, on moving images captured by an imaging device are known. The following Patent Document 1 discloses a method of executing an anti-shake process on image data relating to a captured image, and removing the effects of the anti-shake process from the image data after the anti-shake process. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-216510 A Summary of the Invention [Problem to be solved by the invention]

[0004] Nowadays, users can easily take images and adjust images using mobile terminals such as smartphones and tablets, or cameras themselves or personal computers, and the like, and video posting is also becoming more and more common. In such an environment, it is desired that users not simply output captured images as they are, but rather create higher quality and more diverse images. Broadcasting companies and others also desire to be able to present images in a variety of ways. In view of this, the present disclosure proposes a technique that enables easy adjustment of shaking in moving images. [Means for solving the problem]

[0005] The image processing device according to the present technology includes a sway change unit that performs sway change processing to obtain sway-changed image data by changing the sway state of input image data that constitutes a moving image, a user interface processing unit that acquires operation information related to the sway change, and a parameter setting unit that sets parameters for the sway change processing based on the operation information acquired by the user interface processing unit. The wobble change process is to change the state of wobble by reducing the wobble occurring in the video, adding wobble, etc. The wobble change unit performs the wobble change process based on the parameters set by the parameter setting unit.

[0006] In the image processing device related to the above-mentioned present technology, it is conceivable that the user interface processing unit presents an operator for adjusting the amount of shaking, and the parameter setting unit sets a parameter for adjusting the amount of shaking based on operation information of the operator. That is, the amount of shaking that appears in the image can be adjusted by operation.

[0007] In the image processing device relating to the above-mentioned present technology, the user interface processing unit may present an operator for adjusting the amount of shaking for one or more shaking direction components, and the parameter setting unit may set a parameter for adjusting the amount of shaking for the shaking direction component based on operation information for the amount of shaking for the shaking direction component. The shaking that appears in an image contains multiple shaking direction components. Therefore, it is possible to adjust the shaking amount for each shaking direction component or for each multiple shaking direction components.

[0008] In the image processing device according to the present technology described above, it is considered that the shaking direction component includes at least one of a yaw direction shaking component, a roll direction shaking component, and a pitch direction shaking component. The shaking that appears in an image includes shaking components in the yaw, roll, and pitch directions. The amount of at least one, two, or all of these shaking directions can be adjusted.

[0009] In the image processing device related to the above-mentioned present technology, the user interface processing unit may present an operator for adjusting the amount of shaking for one or more frequency bands of shaking, and the parameter setting unit may set a parameter for adjusting the amount of shaking for the frequency band based on operation information for the amount of shaking for the frequency band. The shaking that appears in an image contains vibration components of multiple frequencies. Therefore, it is possible to adjust the amount of shaking for each frequency band or for multiple frequency bands.

[0010] In the image processing device related to the present technology described above, the user interface processing unit may present controls for adjusting the amount of shaking of at least a low frequency band and a high frequency band as the frequency bands, and the parameter setting unit may set a parameter for adjusting the amount of shaking of the low frequency band and a parameter for adjusting the amount of shaking of the high frequency band. For example, the vibration can be adjusted in two bands, a low frequency band and a high frequency band, or it can be adjusted in three or more bands, a low frequency band, one or more intermediate frequency bands, and a high frequency band.

[0011] In the image processing device relating to the above-mentioned present technology, the user interface processing unit enables a selection operation of preset data for a shaking waveform, and the parameter setting unit can set parameters for adjusting the shaking of the image in accordance with the selection operation of the preset data. For example, one or more preset data of vibration waveforms indicating vibration patterns are stored, and the user is allowed to apply one of the preset data or to select one of the multiple preset data.

[0012] In the image processing device according to the present technology described above, it is considered that the user interface processing unit executes presentation of a timeline indicating a shaking component in the time axis direction of the image. For example, a waveform representing vibration is displayed along the time axis of the image.

[0013] In the image processing device according to the present technology described above, it is considered that the user interface processing unit enables a user to specify a key frame on the timeline through a user operation. Allows the user to specify any point (timecode / frame) on the timeline.

[0014] In the image processing device relating to the above-mentioned present technology, the user interface processing unit may enable a user to specify a key frame on the timeline by user operation, and may perform processing to play an image based on the key frame specified on the timeline by user operation. For example, the user can specify any point (time code / frame) on the timeline as a key frame, and can also perform a preview (playback) operation.

[0015] In the image processing device relating to the above-mentioned present technology, it is considered that the user interface processing unit performs processing to repeatedly play back an image section determined based on a key frame specified by a user operation on the timeline. For example, when a user designates a certain point (time code / frame) on the timeline as a key frame, the image section is determined based on that key frame, and the image section is then repeatedly played back.

[0016] In the image processing device according to the present technology described above, it is considered that the user interface processing unit performs processing of assigning preset data selected by an operation to the timeline. For example, a user can perform an operation such as dragging an image showing preset data onto a timeline, thereby applying the preset data to a certain section on the timeline.

[0017] In the image processing device according to the present technology described above, it is considered that the user interface processing unit performs processing to present a state of shaking that changes according to assignment of preset data to the timeline. For example, when a user applies certain preset data to a certain section on the timeline, a display is made to indicate that the shaking will change in that section.

[0018] In the image processing device relating to the above-mentioned present technology, the user interface processing unit can be thought of as enabling a user to specify a key frame on the timeline by operation, and also enabling a sway change operation to be reflected in a section set based on the key frame. For example, the user can specify any point (time code / frame) on the timeline as a key frame, and the section is automatically set according to the key frame. Changes to the shaking caused by various operations are reflected in that section.

[0019] In the image processing device relating to the present technology described above, it is considered that the user interface processing unit performs processing to present the state before and after the shaking of the image caused by the shaking change operation on the timeline. For example, the vibration waveform is displayed, and the vibration waveform before and after the vibration change is simultaneously displayed or switched between.

[0020] In the image processing device relating to the present technology described above, the jitter change unit is considered to perform jitter change processing that adds jitter to input image data that is not jittered or to input image data that has been subjected to jitter reduction processing. For example, the shake correction is performed on an image captured by a fixed camera and originally free of camera shake or an image from which camera shake or the like has been removed by the imaging device.

[0021] In the image processing device according to the present technology described above, it is considered that the wobble modification unit performs wobble modification processing on input image data including a wobble component. For example, an image in which camera shake or the like has not been removed is input, and shake modification is performed on this image.

[0022] In the image processing device according to the present technology described above, it is considered that the user interface processing unit performs a process of storing the vibration information in response to a user's operation for saving the vibration information. That is, information on swing changes made in response to user operations, such as parameter settings, can be saved.

[0023] In the image processing method of the present technology, an image processing device performs a sway modification process to obtain sway-modified image data by changing the sway state of input image data that constitutes a moving image, a user interface process to obtain operation information related to the sway modification, and a parameter setting process to set parameters for the sway modification process based on the operation information obtained by the user interface process. This allows the user to perform operations for producing a shake effect or removing a shake from an image. A program according to the present technology is a program that causes an information processing device to execute processing equivalent to such an image processing method. This makes it possible to execute the image processing of the present disclosure on various information processing devices. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is an explanatory diagram of a device used in an embodiment of the present technology. [Diagram 2] FIG. 4 is an explanatory diagram of information transmitted between devices according to an embodiment. [Diagram 3] 1 is a block diagram of an imaging apparatus according to an embodiment; [Figure 4] 5A to 5C are explanatory diagrams of an image shake removal process in the imaging device according to the embodiment. [Diagram 5] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a functional configuration of an image processing apparatus according to an embodiment; [Figure 7]FIG. 2 is an explanatory diagram of an input / output mode of an image processing device according to an embodiment. [Figure 8] FIG. 2 is an explanatory diagram of an input / output mode of an image processing device according to an embodiment. [Figure 9] FIG. 2 is an explanatory diagram of an input / output mode of an image processing device according to an embodiment. [Figure 10] FIG. 2 is an explanatory diagram of an input / output mode of an image processing device according to an embodiment. [Figure 11] FIG. 2 is an explanatory diagram of an input / output mode of an image processing device according to an embodiment. [Figure 12] 3A and 3B are explanatory diagrams of the contents of an image file and metadata according to the embodiment; [Figure 13] FIG. 11 is an explanatory diagram of metadata related to lens distortion correction. [Figure 14] FIG. 2 is an explanatory diagram of image processing according to the embodiment. [Figure 15] FIG. 13 is an explanatory diagram of application to a celestial sphere model according to an embodiment. [Figure 16] FIG. 4 is an explanatory diagram of sample timing of IMU data according to the embodiment. [Figure 17] 11 is an explanatory diagram of vibration information adjustment for each frequency band according to an embodiment. FIG. [Figure 18] 11 is an explanatory diagram of the direction-specific shaking information adjustment according to the embodiment. FIG. [Figure 19] 11 is an explanatory diagram of the vibration information adjustment by frequency band and direction according to an embodiment. FIG. [Figure 20] FIG. 11 is an explanatory diagram of correspondence between an output image and a celestial sphere model according to the embodiment. [Figure 21] 4A to 4C are explanatory diagrams of rotation and perspective projection of an output coordinate plane according to an embodiment of the present invention. [Figure 22] FIG. 2 is an explanatory diagram of a cut-out region according to an embodiment; [Figure 23] FIG. 4 is an explanatory diagram of metadata transmitted in the embodiment. [Figure 24] FIG. 11 is an explanatory diagram of secondary image processing according to the embodiment. [Diagram 25] FIG. 2 is an explanatory diagram of a UI according to the first embodiment. [Figure 26]FIG. 11 is an explanatory diagram of a UI according to the second embodiment. [Figure 27] FIG. 13 is an explanatory diagram of a UI according to the third embodiment. [Figure 28] FIG. 13 is an explanatory diagram of image processing according to the third embodiment. [Figure 29] 4 is a flowchart of UI processing according to the first, second and third embodiments. [Diagram 30] FIG. 13 is an explanatory diagram of a UI according to the fourth embodiment. [Diagram 31] FIG. 13 is an explanatory diagram of a UI according to the fifth embodiment. [Diagram 32] FIG. 13 is an explanatory diagram of a UI according to the fifth embodiment. [Diagram 33] FIG. 23 is an explanatory diagram of a UI according to the sixth embodiment. [Diagram 34] 13 is a flowchart of UI processing according to the fourth, fifth and sixth embodiments. [Diagram 35] 13 is a flowchart of UI processing according to the fourth, fifth and sixth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] The embodiments will be described below in the following order. <1. Configuration of equipment that can be used as an image processing device> <2. Device configuration and processing functions> <3. Image files and metadata> 4. Image Processing of the Embodiment <5. User interface for changing sway> [5-1: UI operation of the first embodiment] [5-2: UI operation of the second embodiment] [5-3: UI operation of the third embodiment] [5-4: UI processing in the first, second and third embodiments] [5-5: UI operation of the fourth embodiment] [5-6: UI operation of the fifth embodiment] [5-7: UI operation of the sixth embodiment] [5-8: UI processing in the fourth, fifth and sixth embodiments] <6. Summary and variations>

[0026] Prior to the description of the embodiments, some terms used in the description will be explained. "Shake" refers to the interframe shake between the frames of images that make up a video. It broadly refers to vibration components that occur between frames (image shake between frames), such as shake caused by camera shake in images captured by an imaging device, or shake intentionally added through image processing.

[0027] "Interframe shake modification" refers to changing the state of shake in an image, such as reducing shake occurring in an image or adding shake to an image. This "shake modification" includes the following "interframe shake reduction" and "interframe shake production."

[0028] "Shake removal" refers to eliminating (removing all shake) or reducing (removing part of) shake that occurs in an image due to camera shake or other factors. For example, it refers to making adjustments to reduce shake based on shake information at the time of image capture. The so-called camera shake correction performed in an image capture device is equivalent to shaking removal.

[0029] "Shake production" may add shake to an image or reduce shake, and in that sense may result in the same as "shake removal," but in this embodiment, it refers to changing the amount of shake change specified by a user operation or automatic control, and changing the shake state of the image in response to the command. For example, "shake production" refers to changing shake information at the time of image capture by a user command or the like, and reducing or increasing shake by performing shake change processing based on the changed shake information, or reducing or increasing shake by changing shake based on shake-adding information generated by a user command or the like. Even when adjusting the shaking in a direction to suppress it, for example, intentionally adjusting the amount of shaking can be considered to fall under "shaking effect." One example of the purpose of the shaking effect is to intentionally shake an image to give a sense of impact to a video scene.

[0030] "Shake information during imaging" is information about the shaking when imaging with an imaging device, and includes detection information about the movement of the imaging device, information that can be calculated from the detection information, posture information that represents the posture of the imaging device, and information about shifts and rotations as the movement of the imaging device. In the embodiment, specific examples of the "imaging shake information" include quaternion (QD), IMU data, shift and rotation information (SFRO), etc. Of course, the information is not limited to these.

[0031] The "adjusted shaking information" is shaking information generated by adjusting the shaking information at the time of imaging, and is information used in shaking change processing, for example, shaking information adjusted according to a user operation or automatic control. In the embodiment, specific examples of the "adjusted shaking information" include adjusted quaternion (eQD) and adjusted IMU data (eIMU), but are not limited to these.

[0032] "Shake modification information" (denoted by the symbol "SMI" in the explanation and drawings) is information on the shake modification process of image data, and may be any information capable of specifying the processing amount of the shake modification process. In the embodiment, specific examples of the "sway modification information" include a sway modification parameter (PRM) and an adjusted quaternion (eQD), but are not limited to these.

[0033] <1. Configuration of equipment that can be used as an image processing device> In the following embodiment, an example in which an image processing device according to the present disclosure is realized by an information processing device such as a smartphone or a personal computer will be described, but the image processing device can be realized by various devices. First, a description will be given of devices to which the technology of the present disclosure can be applied.

[0034] FIG. 1A shows an example of an image source VS and image processing devices (TDx, TDy) that acquire an image file MF from the image source VS. The image processing device TDx is a device that performs primary shaking change processing on image data acquired from the image source VS. On the other hand, the image processing device TDy is a device that performs secondary shaking change processing on image data that has already been subjected to shaking change processing by another image processing device.

[0035] The image source VS may be an image capture device 1, a server 4, a recording medium 5, or the like. The image processing devices TDx and TDy are assumed to be a mobile terminal 2 such as a smartphone, a personal computer 3, etc. Although not shown, various other devices such as a dedicated image editing device, a cloud server, a television device, a video recording and playback device, etc. These devices can function as either the image processing device TDx or TDy.

[0036] An imaging device 1 serving as an image source VS is a digital camera or the like capable of capturing moving images, and transfers an image file MF obtained by capturing moving images to a mobile terminal 2, a personal computer 3, or the like via wired or wireless communication. The server 4 may be a local server, a network server, a cloud server, or the like, and refers to a device that can provide an image file MF captured by the imaging device 1. It is conceivable that this server 4 transfers the image file MF to the mobile terminal 2, the personal computer 3, or the like via some kind of transmission path.

[0037] The recording medium 5 may be any of solid-state memory such as a memory card, a disk-shaped recording medium such as an optical disk, and a tape-shaped recording medium such as a magnetic tape, but refers to a removable recording medium on which is recorded an image file MF captured by the imaging device 1. It is conceivable that the image file MF read from this recording medium 5 is read by the mobile terminal 2, the personal computer 3, or the like.

[0038] The image processing devices TDx and TDy, such as the mobile terminal 2 and the personal computer 3, are capable of performing image processing on the image file MF acquired from the image source VS. The image processing here includes shaking change processing (shake effect and shaking removal). The swing change process is performed, for example, by attaching each frame of image data constituting a video to a celestial sphere model, and then rotating the frame using the attitude information corresponding to that frame.

[0039] It should be noted that a certain mobile terminal 2 or personal computer 3 may become an image source VS for other mobile terminals 2 or personal computers 3 functioning as image processing devices TDx, TDy.

[0040] FIG. 1B shows the imaging device 1 and the mobile terminal 2 as a single device that can function as both an image source VS and an image processing device TDx. For example, a microcomputer or the like inside the imaging device 1 performs the shaking change process. That is, the imaging device 1 performs shaking change processing on the image file MF generated by imaging, thereby making it possible to output an image as an image processing result in which shaking has been removed or shaking has been produced.

[0041] The same is true for the mobile terminal 2, which can become an image source VS by being equipped with an imaging function, so that by performing the above-mentioned shaking change processing on the image file MF generated by imaging, it is possible to output an image as the result of image processing with shaking removed or shaking effects applied. Of course, the image source and image processing device may be various other devices other than the imaging device 1 and the mobile terminal 2.

[0042] As described above, there are various devices and image sources VS that function as the image processing devices TDx and TDy in the embodiments, but in the following, the image source VS such as the imaging device 1, the image processing device TDx such as the portable terminal 2, and the other image processing devices TDy will be described as being separate devices.

[0043] FIG. 2 shows information transmission among an image source VS, an image processing device TDx, and an image processing device TDy. Image data VD1 and metadata MTD1 are transmitted from the image source VS to the image processing device TDx via wired communication, wireless communication, or a recording medium. As will be described later, the image data VD1 and the metadata MTD1 are information that is transmitted as, for example, an image file MF. The metadata MTD1 may contain coordinate transformation parameters HP as information on the removal of shaking during image capture, for example, as a camera shake correction.

[0044] The image processing device TDx is capable of receiving image data VD1, metadata MTD1, and coordinate transformation parameters HP and carrying out various types of processing. For example, the image processing device TDx can perform a shake change process on the image data VD1 using the image capture shake information included in the metadata MTD1. Also, for example, the image processing device TDx can cancel the shake elimination applied to the image data VD1 at the time of capture by using the coordinate transformation parameters HP included in the metadata MTD1.

[0045] When the image processing device TDx has performed the shaking change processing, the image data, the imaging shaking information, and the shaking change information SMI that can specify the processing amount of the shaking change processing are associated with each other. The associated image data, imaging vibration information, and vibration modification information SMI can be transmitted to the image processing device TDy together or separately via wired communication, wireless communication, or a recording medium. Here, the term "associate" means, for example, making one piece of information (data, command, program, etc.) available (linked) when processing the other piece of information. In other words, pieces of information associated with each other may be collected into a single file or the like, or may be individual pieces of information. For example, information B associated with information A may be transmitted on a transmission path different from that of information A. Also, for example, information B associated with information A may be recorded on a recording medium different from that of information A (or on a different recording area of ​​the same recording medium). Note that this "association" may be a part of information, not the whole piece of information. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame. More specifically, "associating" includes, for example, acts such as assigning the same ID (identification information) to multiple pieces of information, recording multiple pieces of information on the same recording medium, storing multiple pieces of information in the same folder, storing multiple pieces of information in the same file (assigning one to the other as metadata), embedding multiple pieces of information in the same stream, and embedding meta in an image, such as a digital watermark.

[0046] 2 shows image data transmitted from the image processing device TDx to the image processing device TDy as image data VD2. There are various examples of the image data VD2, such as an image in which the shake elimination in the imaging device 1 has been cancelled, an image in which the shake elimination has been performed in the image processing device TDx, or an image before the shake change process in the image processing device TDx has been performed. 2 also shows metadata MTD2 transmitted from the image processing device TDx to the image processing device TDy. The metadata MTD2 may be the same as the metadata MTD1 or may be partially different information. However, the metadata MTD2 includes information on the shaking during imaging. Therefore, the image processing device TDy can acquire at least the image data VD2, the imaging time shaking information included in the metadata MTD2, and the shaking modification information SMI in an associated state. Note that a data format in which the swing change information SMI is also included in the metadata MTD2 is also conceivable.

[0047] In this embodiment, assuming such information transmission, image processing that can be performed by the image processing devices TDx and TDy will be described.

[0048] <2. Device configuration and processing functions> First, an example of the configuration of an image pickup device 1 serving as an image source VS will be described with reference to FIG. In addition, as explained in FIG. 1B, if it is assumed that image processing will be performed on the image file MF captured by the portable terminal 2, then the portable terminal 2 will need to have a configuration equivalent to that of the imaging device 1 described below in terms of imaging functions.

[0049] Furthermore, the imaging device 1 performs a process called image stabilization, which reduces image shaking caused by the movement of the imaging device during imaging, and this is the "shake removal" performed by the imaging device. In contrast, the "shake effect" and "shake removal" performed by the image processing devices TDx and TDy are separate processes independent of the "shake removal" performed by the imaging device 1 during imaging.

[0050] As shown in FIG. 3, the imaging device 1 includes, for example, a lens system 11, an imaging element unit 12, a camera signal processing unit 13, a recording control unit 14, a display unit 15, an output unit 16, an operation unit 17, a camera control unit 18, a memory unit 19, a driver unit 22, and a sensor unit 23.

[0051] The lens system 11 includes lenses such as a cover lens, a zoom lens, and a focus lens, an aperture mechanism, etc. The lens system 11 guides light (incident light) from a subject and collects the light on the imaging element unit 12. Although not shown, the lens system 11 may be provided with an optical image stabilization mechanism that corrects interframe shake and blur caused by camera shake or the like.

[0052] The imaging element unit 12 includes an image sensor 12a (imaging element) such as a complementary metal oxide semiconductor (CMOS) type or a charge coupled device (CCD) type. The image sensor unit 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc., on the electric signal obtained by photoelectrically converting the light received by the image sensor 12a, and further performs A / D (Analog / Digital) conversion processing. Then, the image signal as digital data is output to the downstream camera signal processor 13 and camera controller 18. Incidentally, the optical image stabilization mechanism (not shown) may be a mechanism that corrects image shaking by moving the image sensor 12a side rather than the lens system 11 side, or a balanced optical image stabilization mechanism that uses a gimbal, and any type of mechanism may be used. The optical image stabilization mechanism corrects not only interframe shake but also intra-frame blur, which will be described later.

[0053] The camera signal processing unit 13 is configured as an image processor, for example, a DSP (Digital Signal Processor) etc. The camera signal processing unit 13 performs various types of signal processing on the digital signal (captured image signal) from the imaging element unit 12. For example, as camera processes, the camera signal processing unit 13 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, etc. The camera signal processor 13 also performs various correction processes. However, it is assumed that image stabilization may or may not be performed within the imaging device 1.

[0054] In the pre-processing, the captured image signal from the imaging element unit 12 is subjected to a clamping process for clamping the R, G, and B black levels to a predetermined level, a correction process between the R, G, and B color channels, and the like. In the synchronization process, a color separation process is performed so that the image data for each pixel has all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer array color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, B image data. In the resolution conversion process, the image data that has been subjected to various types of signal processing is subjected to the resolution conversion process.

[0055] Examples of various correction processes (internal corrections of the imaging device 1) performed by the camera signal processing unit 13 are shown in Fig. 4. In Fig. 4, the correction processes performed by the camera signal processing unit 13, together with the optical image stabilization performed by the lens system 11, are illustrated in the order of execution.

[0056] In the optical image stabilization as process F1, in-lens image stabilization is performed by shifting the lens system 11 in the yaw and pitch directions, and in-body image stabilization is performed by shifting the image sensor 12a in the yaw and pitch directions, so that an image of the subject is formed on the image sensor 12a with the effects of camera shake physically canceled. This in-lens image stabilization and in-body image stabilization may be used alone, or both may be used. When using both in-lens image stabilization and in-body image stabilization, it is considered that the in-body image stabilization does not shift in the yaw and pitch directions. In some cases, neither in-lens image stabilization nor in-body image stabilization is used, and only electrical image stabilization or only optical image stabilization is used to combat camera shake.

[0057] In the camera signal processing unit 13, the processes F2 to F7 are carried out by spatial coordinate conversion for each pixel. In process F2, lens distortion correction is performed. In process F3, focal plane distortion correction is performed as one element of electronic image stabilization. This is to correct distortion that occurs when, for example, the CMOS image sensor 12a performs reading using the rolling shutter method.

[0058] In process F4, roll correction is performed. That is, correction of the roll component is performed as one element of electronic image stabilization. In process F5, keystone distortion caused by electronic image stabilization is corrected. The keystone distortion caused by electronic image stabilization is a perspective distortion caused by cutting out a portion of the image away from the center. In process F6, shifting in the pitch direction and yaw direction and cropping are performed as one element of electronic image stabilization. For example, camera shake correction, lens distortion correction, and keystone distortion correction are performed in the above-mentioned procedure. It is not essential to perform all of the processes described here, and the order of the processes may be changed as appropriate.

[0059] In the codec process in the camera signal processor 13 in Fig. 3, the image data that has been subjected to the above various processes is encoded and a file is generated for recording or communication. For example, an image file MF is generated in the MP4 format used for recording moving images and audio conforming to MPEG-4. It is also possible to generate files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) as still image files. The camera signal processing unit 13 also generates metadata to be added to the image file MF using information from the camera control unit 18 and the like.

[0060] Although an audio processing system is omitted in FIG. 3, in reality, an audio recording system and an audio processing system are provided, and the image file MF may contain audio data as well as image data as a moving image.

[0061] The recording control unit 14 performs recording and reproduction on a recording medium, such as a non-volatile memory, etc. The recording control unit 14 performs processing for recording image files MF such as moving image data and still image data, thumbnail images, etc. on the recording medium, for example. There are various possible actual forms for the recording control unit 14. For example, the recording control unit 14 may be configured as a flash memory built into the imaging device 1 and a write / read circuit for the memory memory, or may be in the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be detached from the imaging device 1, such as a memory card (such as a portable flash memory). Also, the recording control unit 14 may be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0062] The display unit 15 is a display unit that displays various information to the person taking the image, and is, for example, a display panel or viewfinder using a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display arranged on the housing of the imaging device 1. The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18 . For example, the display unit 15 displays a reproduced image of image data read from a recording medium by the recording control unit 14. Further, image data of the captured image that has been resolution-converted for display by the camera signal processing unit 13 is supplied to the display unit 15, and the display unit 15 may perform display based on the image data of the captured image in response to an instruction from the camera control unit 18. This causes a so-called through image (monitoring image of the subject), which is the captured image during composition confirmation, to be displayed. Further, based on instructions from the camera control unit 18, the display unit 15 executes display of various operation menus, icons, messages, etc., that is, GUI (Graphical User Interface) on the screen.

[0063] The output unit 16 performs data communication and network communication with external devices via wire or wirelessly. For example, the captured image data (still image files or video files) is transmitted and output to an external display device, recording device, playback device, or the like. The output unit 16 may also be a network communication unit that communicates via various networks such as the Internet, a home network, or a LAN (Local Area Network), and transmits and receives various data to and from servers, terminals, etc. on the network.

[0064] The operation unit 17 collectively refers to input devices for a user to input various operations. Specifically, the operation unit 17 refers to various operators (keys, dials, touch panel, touch pad, etc.) provided on the housing of the imaging device 1. An operation performed by the user is detected by the operation unit 17, and a signal corresponding to the input operation is sent to the camera control unit .

[0065] The camera control unit 18 is composed of a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 19 stores information and the like used for processing by the camera control unit 18. The illustrated memory unit 19 collectively includes, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, and the like. The memory unit 19 may be a memory area built into a microcomputer chip serving as the camera control unit 18, or may be configured by a separate memory chip. The camera control unit 18 executes a program stored in the ROM, flash memory, or the like of the memory unit 19 to control the entire imaging device 1 . For example, the camera control unit 18 controls the operation of each necessary unit with respect to controlling the shutter speed of the image sensor unit 12, instructing various signal processing in the camera signal processing unit 13, imaging operations and recording operations in response to user operations, playback operations of recorded image files, operations of the lens system 11 such as zoom, focus, and aperture adjustment in the lens barrel, user interface operations, etc.

[0066] The RAM in the memory unit 19 is used as a working area for various data processing by the CPU of the camera control unit 18, for temporarily storing data, programs, and the like. The ROM and flash memory (non-volatile memory) in the memory unit 19 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, etc.

[0067] The driver unit 22 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 18, thereby moving the focus lens and zoom lens, opening and closing the aperture blades of the aperture mechanism, and so on.

[0068] The sensor unit 23 collectively represents various sensors mounted on the imaging device. The sensor unit 23 is equipped with, for example, an IMU (inertial measurement unit), and can detect angular velocity using a three-axis angular velocity (gyro) sensor of pitch, yaw, and roll, and can detect acceleration using an acceleration sensor. It should be noted that the sensor unit 23 need only include a sensor capable of detecting camera shake during image capture, and does not need to include both a gyro sensor and an acceleration sensor. The sensor unit 23 may also include a position information sensor, an illuminance sensor, and the like.

[0069] For example, an image file MF serving as a moving image captured and generated by the imaging device 1 described above can be transferred to image processing devices TDx, TDy such as the portable terminal 2 and subjected to image processing. The mobile terminal 2 and the personal computer 3 serving as the image processing devices TDx and TDy can be realized as information processing devices having the configuration shown in Fig. 5. The server 4 can also be realized by an information processing device having the configuration shown in Fig. 5.

[0070] 5, a CPU 71 of an information processing device 70 executes various processes according to a program stored in a ROM 72 or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. The CPU 71, the ROM 72, and the RAM 73 are interconnected via a bus 74. To this bus 74, an input / output interface 75 is also connected.

[0071] The input / output interface 75 is connected to an input unit 76 including an operator and an operating device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0072] Further, a display unit 77 formed of an LCD or an organic EL panel, etc., and an audio output unit 78 formed of a speaker, etc. are connected to the input / output interface 75 either integrally or separately. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various types of image processing, moving images to be processed, etc., on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0073] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like. The communication unit 80 performs communication processing via a transmission path such as the Internet, and performs communication with various devices via wired / wireless communication, bus communication, and the like.

[0074] A drive 82 is also connected to the input / output interface 75 as required, and a removable recording medium 81 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 82 allows data files such as image files MF and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0075] In this information processing device 70, for example, software for image processing as the image processing device of the present disclosure can be installed via network communication by the communication unit 80 or a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0076] For example, such software (application program) configures a functional configuration as shown in FIG. 6 in the CPU 71 of the information processing device . 6 shows functions provided as the information processing device 70 functioning as the image processing devices TDx and TDy. That is, the information processing device 70 (CPU 71) has functions as a swing change unit 101, a parameter setting unit 102, and a user interface processing unit 103. It should be noted that "user interface" is also written as "UI," and the user interface processing unit 103 is also written as "UI processing unit 103" below.

[0077] The vibration modification unit 101 is a function that performs vibration modification processing to change the vibration state of image data using parameters input by the user and post-adjustment vibration information. The wobble modification unit 101 performs processing for wobble removal or wobble effect, and obtains an output image as image data from which wobble has been removed or from which wobble effect has been applied.

[0078] The UI processing unit 103 is a function that performs processing to present operators related to swing change to the user and to obtain operation information by the operators. For example, the UI processing unit 103 performs processing to display an image showing information about an operator or an image as a UI image on the display unit 77. The UI processing unit 103 also detects a user operation via the input unit 76. For example, the UI processing unit 103 detects a touch operation on the UI image.

[0079] The parameter setting unit 102 is a function for setting parameters (sway change parameters PRM) for the swing change processing based on the operation information acquired by the UI processing unit 103. In other words, the UI processing unit 103 converts the user operation content detected by the UI processing unit 103 into the swing change parameters PRM and supplies it to the swing change unit 101, so that the swing change processing according to the user operation is performed on the image data.

[0080] Here, various input / output modes for the image processing devices TDx and TDy having such functions are possible, for example, as in the following examples of Pattern 1 to Pattern 5.

[0081] 7, pattern 1 is a mode in which image data PD(NS) that is originally free of shaking such as camera shake is input to the shaking change unit 101. As the image data PD(NS), for example, image data captured by a fixed surveillance camera or the like, image data captured using a tripod or the like can be considered. In this case, the shaking change unit 101 performs a shaking effect to add shaking to the image based on the shaking change parameter PRM that is set in response to the operation of the UI processing unit 103, and it is assumed that output image data oPD with shaking added is obtained.

[0082] 8, pattern 2 is a mode in which image data PD in which shaking such as camera shake has been corrected is input to the shake change unit 101. For example, image data PD captured by an imaging device 1 equipped with a camera shake correction function is input. In this case, the shaking change unit 101 performs a shaking effect to add shaking to the image based on the shaking change parameter PRM that is set in response to the operation of the UI processing unit 103, and it is assumed that output image data oPD with shaking added is obtained.

[0083] 9, pattern 3 is a mode in which image data PD in which shaking such as camera shake has been corrected is input to the shaking change unit 101. As in pattern 2, it is assumed that image data PD captured by an imaging device 1 equipped with a camera shake correction function is input. The information processing device 70 (CPU 71) serving as the image processing devices TDx and TDy is further provided with a function as a correction canceling unit 104. The correction cancellation unit 104 performs a process of canceling the camera shake correction that has already been applied to the image data PD. The correction for removing the shake is canceled from the image data PD, and the image data PD becomes the original (pre-correction) image data iPD in a shake-free state.

[0084] It is assumed that the shaking change unit 101 performs shaking change processing on such image data iPD to adjust the shaking of the image based on the shaking change parameter PRM that is set in response to the operation of the UI processing unit 103, and obtains output image data oPD in which the shaking state has been changed.

[0085] In this case, shaking change may involve shaking elimination or shaking effect. The shake removal in the shake modification unit 101 can be considered to be a process of canceling the shake correction originally performed on the imaging device 1 side and then performing shake correction again with more advanced processing. In other words, this is a process of performing shake removal with higher performance than the camera shake correction in the imaging device 1 to obtain an image with less shake. The shaking effect produced by the shaking change unit 101 is assumed to be a process of changing or adjusting the shaking state of a shaking image according to the user's intention after canceling the shaking correction originally performed on the imaging device 1 side. The adjustment referred to here may be, for example, a process of removing a part of the shaking to make the original shaking slightly more subdued, or a process of making the original shaking more exaggerated.

[0086] 10, pattern 4 is a mode in which image data PD(S) that still contains shaking due to camera shake or the like is input to the shaking change unit 101. For example, image data PD(S) captured by an imaging device 1 that does not have a camera shake correction function, or image data PD(S) captured with the camera shake correction function turned off is input.

[0087] For such image data iPD, the jitter modifying section 101 performs processing as the functions of a jitter removing section 101a and a jitter adding section 101b. The jitter removal unit 101a and the jitter addition unit 101b are both functions that perform jitter modification processing, but in particular, the jitter removal unit 101a is a function that performs processing to remove jitter that occurs in the input image data PD(S). For example, the shake removal unit 101a performs a process of removing shake using image capture shake information acquired from the image capture device 1. In addition, assuming an image processing device TDy, the shake removal unit 101a performs a process of removing the shake added at that time using shake change information from the image processing device TDx that added the shake. The wobble adding unit 101b has a function of adding a wobble in response to a user operation.

[0088] In the jitter change unit 101, the jitter remover 101a first performs processing to remove jitter that occurred during image capture or previous image processing on the input image data PD(S). Then, in a state in which the jitter has been removed, the jitter addition unit 101b performs a jitter effect to add jitter to the image based on the jitter change parameter PRM set in response to the operation of the UI processing unit 103. Therefore, it is expected that output image data oPD to which a jitter different from that of the input image data PD(S) has been added is obtained.

[0089] 11, pattern 5 is a mode in which image data PD(S) that still contains shaking due to camera shake or the like is input to the shaking change unit 101. As in pattern 4, image data PD(S) captured by an imaging device 1 that does not have a camera shake correction function, or image data PD(S) captured with the camera shake correction function turned off, for example, is input. It is assumed that the shaking change unit 101 performs shaking change processing on such image data iPD to adjust the shaking of the image based on the shaking change parameter PRM that is set in response to the operation of the UI processing unit 103, thereby obtaining output image data oPD with the shaking adjusted. The adjustment referred to here may be, for example, removing a portion of the original shaking to make it slightly more subdued, or processing the original shaking to make it more exaggerated.

[0090] <3. Image files and metadata> The contents of the image file MF and the metadata transmitted from an image source VS such as the image capture device 1 to the image processing device TDx will be described. 12A shows the data contained in an image file MF. As shown in the figure, an image file MF contains various types of data such as a "header," "sound," "movie," and "metadata."

[0091] In the "header", information such as the file name, file size, and information indicating the presence or absence of metadata are described. "Sound" is the audio data recorded with the video. For example, 2-channel stereo audio data is stored. A "movie" is moving image data, and is made up of image data for each frame (#1, #2, #3, . . .) that makes up the moving image. As "metadata", additional information associated with each frame (#1, #2, #3, . . .) that constitutes the video is described.

[0092] An example of the metadata content is shown in Figure 12B. For example, for one frame, IMU data, coordinate transformation parameters HP, timing information TM, and camera parameters CP are described. Note that these are only a part of the metadata content, and only information related to image processing, which will be described later, is shown here.

[0093] IMU data includes gyro (angular velocity data), accelerator (acceleration data), and sampling rate. The IMU mounted on the imaging device 1 as the sensor unit 23 outputs angular velocity data and acceleration data at a predetermined sampling rate. In general, this sampling rate is higher than the frame rate of the captured image, and therefore many IMU data samples can be obtained during one frame period.

[0094] Therefore, as for the angular velocity data, n samples are associated with one frame, such as gyro sample #1, gyro sample #2, . . . , gyro sample #n, as shown in FIG. 12C. Similarly, as for acceleration data, m samples are associated with one frame, such as accelerator sample #1, accelerator sample #2, . . . accelerator sample #m. There are cases where n=m and cases where n≠m. Although the metadata is associated with each frame in the example described here, there are cases where the IMU data is not completely synchronized with the frames. In such cases, for example, time information related to the time information of each frame is provided as an IMU sample timing offset in the timing information TM.

[0095] The coordinate transformation parameters HP are a general term for parameters used in correction involving coordinate transformation of each pixel in an image. For example, nonlinear coordinate transformation such as lens distortion is also included. The coordinate transformation parameters HP are a term that can include at least lens distortion correction parameters, keystone distortion correction parameters, focal plane distortion correction parameters, electronic image stabilization parameters, and optical image stabilization parameters.

[0096] The lens distortion correction parameters are information that directly or indirectly determines how distortions such as barrel aberration and pincushion aberration have been corrected, and is used to restore an image to its pre-lens distortion state. We will briefly explain metadata related to lens distortion correction parameters, which are one type of metadata. FIG. 13A shows a schematic diagram of the lens system 11 and the image sensor 12a, and indicates the image height Y, the angle α, the entrance pupil position d1, and the exit pupil position d2. The lens distortion correction parameters are used in image processing to know the angle of incidence for each pixel of the image sensor 12a, and therefore it is sufficient to know the relationship between the image height Y and the angle α.

[0097] 13B shows an image 110 before lens distortion correction and an image 111 after lens distortion correction. The maximum image height H0 is the maximum image height before distortion correction and is the distance from the center of the optical axis to the farthest point. The maximum image height H1 is the maximum image height after distortion correction. The metadata required to understand the relationship between image height Y and angle α is the maximum image height H0 before distortion correction, and the incident angle data d0, d1, . . . d(N-1) for each of the N image heights. "N" is assumed to be around 10, for example.

[0098] Returning to FIG. 12B, the keystone distortion correction parameter is the amount of correction used when correcting keystone distortion caused by shifting the cutout area from the center due to electronic image stabilization, and is a value that corresponds to the amount of correction of electronic image stabilization.

[0099] The focal plane distortion correction parameter is a value indicating the amount of correction for the focal plane distortion for each line.

[0100] For electronic image stabilization and optical image stabilization, the parameter indicates the amount of correction in the yaw, pitch, and roll axial directions.

[0101] The parameters for lens distortion correction, keystone distortion correction, focal plane distortion correction, and electronic image stabilization are collectively referred to as coordinate transformation parameters, because these correction processes are performed on the image formed on each pixel of image sensor 12a of imaging element unit 12, and are parameters for correction processes involving coordinate transformation of each pixel. Optical image stabilization is also considered to be one of the coordinate transformation parameters, because in optical image stabilization, correction of shaking of inter-frame components is a process involving coordinate transformation of each pixel. In other words, by performing inverse correction using these parameters, it is possible to return image data that has been subjected to lens distortion correction, keystone distortion correction, focal plane distortion correction, electronic image stabilization, and optical image stabilization to the state before each correction process, i.e., the state when the image was formed on the image sensor 12a of the imaging element unit 12.

[0102] Furthermore, the lens distortion correction, trapezoidal distortion correction, and focal plane distortion correction parameters are distortion correction processes for images captured when the optical image from the subject is itself optically distorted, and since each of them aims to correct optical distortion, they are collectively referred to as optical distortion correction parameters. In other words, by performing inverse correction using these parameters, image data that has been subjected to lens distortion correction, keystone distortion correction, and focal plane distortion correction can be restored to the state before optical distortion correction.

[0103] The timing information TM in the metadata includes information on the exposure time (shutter speed), exposure start timing, readout time (curtain speed), number of exposure frames (long-second exposure information), IMU sample offset, and frame rate. In the image processing of this embodiment, these are mainly used to associate the lines of each frame with IMU data. However, even if the image sensor 12a is a CCD or a global shutter type CMOS, if the exposure center of gravity is shifted by using an electronic shutter or a mechanical shutter, correction to match the exposure center of gravity can be made using the exposure start timing and curtain speed as well.

[0104] As the camera parameters CP in the metadata, the angle of view (focal length), zoom position, and lens distortion information are described.

[0105] 4. Image Processing of the Embodiment An example of processing by the information processing device 70 serving as image processing devices TDx and TDy as an embodiment will be described. FIG. 14 shows the procedure of various processes executed in the information processing device 70 serving as the image processing device TDx, and also shows the relationship between information used in each process. Depending on the function of the swing change unit 101 in FIG. 6, the processes of steps ST13, ST14, ST15, and ST16, which are surrounded as step ST30 in FIG. 14, are performed. Depending on the function of the parameter setting unit 102, a parameter setting process of step ST41 is performed. Depending on the function of the UI processing unit 103, the UI processing of step ST40 is performed.

[0106] In addition, the example of FIG. 14 assumes pattern 3 of FIG. 9, and image data PD in which shaking due to camera shake or the like has been corrected is input to the shaking change process ST30 with the correction canceled. The process of step ST12 corresponds to the function of the correction canceling unit 104 in Fig. 9. If the process of step ST12 in Fig. 14 is turned off, the processes of the other patterns (patterns 1, 2, 4, and 5) described above can be assumed.

[0107] In Figure 14, step ST30 surrounded by a dashed line is indicated as "swing change" and step ST16 is indicated as "swing change processing", but step ST16 is processing that actually changes the state of the swing, and is "swing change" in the narrow sense. On the other hand, step ST30 is a "swing change" in a broad sense that includes the swing change process of step ST16 as well as the celestial sphere model processing and coefficient setting as preparations for that process.

[0108] As for the process in FIG. 14, first, steps ST1, ST2, ST3, and ST4 as pre-processing will be described. Pre-processing is performed when the image file MF is imported. The term "import" here refers to making the image file MF and the like accessible by the information processing device 70 by importing it into the storage unit 79, for example, the subject of image processing, and to carrying out preprocessing to develop it so that it can be processed. It does not refer to, for example, transferring it from the imaging device 1 to the mobile terminal 2, etc.

[0109] The CPU 71 imports an image file MF designated by a user operation or the like so as to be the subject of image processing, and also performs processing related to metadata added to the image file MF as preprocessing, for example, extracting and storing metadata corresponding to each frame of a moving image. Specifically, this pre-processing involves extracting metadata (step ST1), linking all IMU data (step ST2), storing the metadata (step ST3), and converting it to a quaternion (attitude information of the image capture device 1) and storing it (step ST4).

[0110] In extracting metadata in step ST1, the CPU 71 reads the target image file MF and extracts the metadata contained in the image file MF as described with reference to FIG. Note that some or all of steps ST1, ST2, ST3, and ST4 may be performed on the image source VS side, such as the imaging device 1. In that case, in the pre-processing, the contents after those processes described below are acquired as metadata.

[0111] The CPU 71 performs a linking process on the IMU data (angular velocity data (gyro sample) and acceleration data (accelerometer sample)) of the extracted metadata in step ST2. This involves taking all the associated IMU data for every frame, arranging it in chronological order and concatenating it to build the IMU data for the entire video sequence. Then, integration processing is performed on the linked IMU data to calculate a quaternion QD that represents the attitude of the image capture device 1 at each point in time on the video sequence, and this is stored and held. Calculating the quaternion QD is just one example. It is also possible to calculate the quaternion QD using only angular velocity data.

[0112] The CPU 71 performs a process of storing the extracted metadata other than the IMU data, i.e., the coordinate transformation parameters HP, timing information TM, and camera parameters CP, in step ST3. That is, the coordinate transformation parameters HP, timing information TM, and camera parameters CP are stored in a state corresponding to each frame.

[0113] By carrying out the above pre-processing, the CPU 71 is prepared to carry out various image processing including shaking modification on the image data received as the image file MF. The routine process in FIG. 14 shows image processing carried out on image data of the image file MF that has been pre-processed as described above.

[0114] The CPU 71 performs processes of extracting one frame of the video (step ST11), canceling internal correction of the imaging device (step ST12), attaching to a celestial sphere model (step ST13), synchronization processing (step ST14), adjusting shake information (step ST15), changing shake (step ST16), specifying an output region (step ST17), projecting and extracting onto a plane (step ST18), and associative recording / playback (step ST20).

[0115] The CPU 71 performs the above-mentioned processes from step ST11 to step ST20 for each frame during image reproduction of the image file MF.

[0116] In step ST11, the CPU 71 decodes one frame of the video (image file MF) according to the frame number FN. Then, it outputs image data PD(#FN) of one frame. Note that "(#FN)" indicates the frame number, and indicates that the information corresponds to that frame. If the moving image has not been subjected to encoding processing such as compression, the decoding processing in step ST11 is not necessary.

[0117] In step ST12, the CPU 71 performs a process of canceling the internal correction performed by the imaging device 1 on one frame of image data PD(#FN). For this purpose, the CPU 71 refers to the coordinate transformation parameters HP(#FN) stored corresponding to the frame number (#FN) during preprocessing, and performs the inverse correction to the correction performed by the imaging device 1. This obtains image data iPD(#FN) in a state where the lens distortion correction, keystone distortion correction, focal plane distortion correction, electronic image stabilization, and optical image stabilization in the imaging device 1 have been canceled.

[0118] In step ST13, the CPU 71 pastes one frame of image data iPD(#FN) after various corrections have been cancelled onto the celestial sphere model by referring to the camera parameters CP(#FN) stored corresponding to the frame number (#FN), i.e., the angle of view, zoom position, and lens distortion information.

[0119] Figure 15 shows an overview of pasting onto the celestial sphere model. Figure 15A shows image data iPD. Image height h is the distance from the center of the image. Each circle in the figure shows the position where the image height h is equal. From the angle of view, zoom position, and lens distortion information for the frame of this image data iPD, the "relationship between the image sensor surface and the incident angle φ" for that frame is calculated, and each position on the image sensor surface is designated as "data0"..."dataN-1." Then, from "data0"..."dataN-1," the relationship between the image height h and the incident angle φ is expressed as a one-dimensional graph as shown in Figure 15B. The incident angle φ is the angle of the light ray (angle viewed from the optical axis). This one-dimensional graph is rotated once around the center of the captured image to determine the relationship between each pixel and the angle of incidence. Accordingly, each pixel of the image data iPD is mapped onto the celestial sphere model MT, such as from pixel G1 in FIG. 15C to pixel G2 on the celestial sphere coordinates.

[0120] As a result of the above, an image (data) of the celestial sphere model MT is obtained by pasting the captured image onto the ideal celestial sphere with lens distortion removed. This celestial sphere model MT has the inherent parameters and distortion of the imaging device 1 that originally captured the image data iPD removed, and the range visible by an ideal pinhole camera is pasted onto the celestial sphere. Therefore, by rotating the image of the celestial sphere model MT in a predetermined direction in this state, it is possible to realize a process for removing shaking or changing shaking effects to produce a shaking effect.

[0121] Here, the shaking change process uses the attitude information (quaternion QD) of the image capture device 1. For this reason, the CPU 71 performs synchronization processing in step ST14. In the synchronization process, a process is performed to identify and obtain the quaternion QD(#LN) appropriate for each line corresponding to the frame number FN. Note that "(#LN)" indicates the line number within the frame, and indicates that the information corresponds to that line.

[0122] The reason why the quaternion QD (#LN) for each line is used is that when the image sensor 12a is of the CMOS type and imaging is performed by the rolling shutter method, the amount of shaking differs for each line. For example, when the image sensor 12a is a CCD type and captures images using a global shutter method, quaternion QD (#FN) in frame units may be used. Even when the image sensor 12a is a CCD or CMOS global shutter, if an electronic shutter (or a mechanical shutter) is used, the center of gravity will shift, so it is better to use a quaternion at the center of the frame exposure period (which shifts depending on the shutter speed of the electronic shutter).

[0123] Now consider the blur that appears in the image. Blur is the blurring of an image caused by the relative movement between the image capture device and the subject within the same frame. In other words, it is the blurring of an image caused by shaking during the exposure time. The longer the exposure time, the stronger the effect of the blurring becomes. When electronic image stabilization is used to control the image range cut out for each frame, it can reduce or eliminate the "shake" that occurs between frames. However, this type of electronic image stabilization cannot reduce the relative shake within the exposure time. In addition, when changing the cropping area for image stabilization, the orientation information of each frame is used, but if that orientation information is shifted from the center of the exposure period, such as the timing of the start or end of the exposure period, the direction of the shaking within the exposure time based on that orientation will be biased, making the blurring more noticeable. Furthermore, the exposure period differs for each line in the rolling shutter of a CMOS.

[0124] Therefore, in the synchronization process in step ST14, a quaternion QD is obtained for each frame of image data based on the timing of the exposure center of gravity for each line. FIG. 16 shows a synchronization signal cV in the vertical period of the imaging device 1, a synchronization signal sV of the image sensor 12a generated from this synchronization signal cV, and the sampling timing of the IMU data, and also shows an exposure timing range 120. The exposure timing range is a parallelogram that shows the exposure period of each line of one frame when the exposure time is t4 using the rolling shutter method. It also shows the time offset t0 between the synchronization signal cV and the synchronization signal sV, the IMU sample timing offset t1, the readout start timing t2, the readout time (curtain speed) t3, and the exposure time t4. The readout start timing t2 is a timing that is a predetermined time t2of after the synchronization signal sV. Each IMU data obtained at each IMU sample timing is linked to a frame. For example, the IMU data in period FH1 is treated as metadata linked to the current frame whose exposure period is indicated by a parallelogram, and the IMU data in period FH1 is treated as metadata linked to the next frame. However, by linking all the IMU data in step ST2 in Figure 14, the link between each frame and IMU data is released, making it possible to manage the IMU data in chronological order. In this case, the IMU data corresponding to the exposure center of gravity of each line of the current frame (timing of the dashed line W) is identified. This can be calculated if the temporal relationship between the IMU data and the effective pixel area of ​​the image sensor 12a is known.

[0125] Therefore, using information that can be acquired as timing information TM corresponding to the frame (#FN), the IMU data corresponding to the exposure center of each line (timing of the dashed line W) is identified. That is, information on exposure time, exposure start timing, readout time, number of exposure frames, IMU sample offset, and frame rate. Then, a quaternion QD calculated from the IMU data of the exposure center of gravity is identified, and is set as the quaternion QD (#LN), which is the attitude information for each line.

[0126] This quaternion QD(#LN) is provided to the process of adjusting the swing information in step ST15. In the swing information adjustment, the CPU 71 adjusts the quaternion QD in accordance with the input swing change parameter PRM. The swing change parameters PRM may be parameters input in response to user operations, or may be parameters generated by automatic control.

[0127] The user can input the shaking change parameter PRM so as to add a desired shaking state to the image. The CPU 71 can also generate the shaking change parameter PRM by automatic control in response to image analysis, image type, or the user's selection of a shaking model.

[0128] FIG. 14 shows the UI process in step ST40 and the parameter setting process in step ST41. Although details of the UI processing will be described later, the user can perform an operation input to instruct a shaking change, that is, an operation to instruct a shaking as a shaking effect, an operation to instruct the degree of shaking removal, and the like.

[0129] Based on the UI processing in step ST40, the CPU 71 sets parameters in step ST41. That is, vibration change parameters according to user operations are set and are provided to the vibration information adjustment processing in step ST15.

[0130] Specifically, in the shaking information adjustment process of step ST15, the CPU 71 generates an adjusted quaternion eQD for adding shaking to an image or increasing or decreasing the amount of shaking based on the quaternion QD, which is the shaking information at the time of imaging, and the shaking change parameter PRM set in step ST41.

[0131] A specific example of generating the adjusted quaternion eQD will be described with reference to FIGS. FIG. 17 shows an example of generating an adjusted quaternion eQD in response to an instruction on the gain for each frequency band by the swing change parameter PRM. A frequency band is a band of vibration frequencies. For the sake of explanation, let us divide it into three bands: low band, middle band, and high band. Of course, this is just an example, and the number of bands can be two or more. A low-frequency gain LG, a mid-frequency gain MG, and a high-frequency gain HG are given as swing change parameters PRM.

[0132] The adjustment processing system in FIG. 17 includes a low-pass filter 41, a mid-pass filter 42, a high-pass filter 43, gain calculation units 44, 45, 46, and a synthesis unit 47. The input to this adjustment processing system is "quaternion QDs for shaking." This is the conjugate of the quaternion QD as shaking information during imaging.

[0133] The values ​​q of the current frame and the preceding and succeeding frames as quaternion QDs for shaking are input to a low-pass filter 41, and the low-pass component q low Request.

[0134]

number

[0135] This low frequency component q low A gain calculator 44 gives a low-frequency gain LG to the In the formula, mean(q,n) indicates the average value of n values ​​before and after q. It goes without saying that the formula for mean(q,n) is merely an example of a low-pass filter, and other calculation methods may be used. Each formula explained below is also an example.

[0136] The value q of the quaternion QDs for the oscillation is also input to the mid-pass file 42, and the mid-pass component q mid Request.

[0137]

number

[0138] In addition, q * low q low is the conjugate of. Also, "x" is a quaternion product. This mid-range component q mid A gain calculator 45 gives a mid-range gain MG to the

[0139] Also, the value q of the quaternion QDs for fluctuating is input to the high-pass file 43, and the high-pass component q high Request.

[0140]

number

[0141] In addition, q * mid q mid is the conjugate of. This high frequency component q high A gain calculator 46 gives a high-frequency gain HG to the

[0142] These gain calculation units 44, 45, and 46 convert the input into “q in "

[0143]

number

[0144] In this case, θ'=θ*gain, and the following “q out " is output. (However, gain is the low-frequency gain LG, the mid-frequency gain MG, and the high-frequency gain HG.)

[0145]

number

[0146] The low-frequency component q' to which the low-frequency gain LG, the mid-frequency gain MG, and the high-frequency gain HG are respectively given by the gain calculation units 44, 45, and 46. low , mid-range component q' mid , high frequency component q' high The value q mixed get.

[0147]

number

[0148] Note that "x" is a quaternion product. The value q obtained in this way mixed is the value of the adjusted quaternion eQD. Although the above is an example of band division, a method of generating an adjusted quaternion eQD in which a gain according to the swing change parameter PRM is given without band division is also conceivable.

[0149] Next, FIG. 18 shows an example of generating an adjusted quaternion eQD according to the directional gain instructions by the swing change parameter PRM. The directions are the directions of motion, i.e., yaw, pitch, and roll. A yaw gain YG, a pitch gain PG, and a roll gain RG are given as swing change parameters PRM.

[0150] The adjustment processing system in FIG. 18 includes a yaw component extraction unit 51, a pitch component extraction unit 52, a roll component extraction unit 53, gain calculation units 54, 55, 56, and a synthesis unit 57. The yaw component extraction unit 51, pitch component extraction unit 52, and roll component extraction unit 53 are provided with information on the yaw axis, pitch axis, and roll axis, respectively.

[0151] The quaternion QDs for swinging the current frame and the preceding and succeeding frames are input to a yaw component extraction unit 51, a pitch component extraction unit 52, and a roll component extraction unit 53, respectively, and the yaw component q yaw , pitch component q pitch , roll component q roll Request. Each of these component extraction processes converts the input into the following “q in "

[0152]

number

[0153] u is a unit vector that represents the direction of an axis such as the yaw axis, pitch axis, or roll axis. In this case, θ'=θ*(a·u), and the following “q out " is output.

[0154]

number

[0155] The yaw component q obtained by such component extraction yaw , pitch component q pitch , roll component q roll For these, gain calculation units 54, 55, and 56 provide a yaw gain YG, a pitch gain PG, and a roll gain RG, respectively. And the yaw component q' after gain calculation yaw , pitch component q'pitch , role component q' roll The value q obtained by combining these values ​​in the combining unit 47 mixed get.

[0156]

number

[0157] In this case, "x" is also a quaternion product. The value q obtained in this way mixed is the value of the adjusted quaternion eQD.

[0158] FIG. 19 shows an example of combining the above frequency band and direction classifications. The adjustment processing system includes a low-pass filter 41 , a mid-pass filter 42 , a high-pass filter 43 , direction-specific processing units 58 , 59 , 60 , gain calculation units 44 , 45 , 46 , and a synthesis unit 61 . The swing change parameter PRM provides a low-frequency gain LG, a mid-frequency gain MG, a high-frequency gain HG, and a yaw gain YG, a pitch gain PG, and a roll gain RG (not shown).

[0159] In this adjustment processing system, the values ​​q of the current frame and the preceding and succeeding frames as quaternions QDs for perturbation are supplied to a low-pass filter 41, a mid-pass filter 42, and a high-pass filter 43 to obtain the respective band components. The respective band components are input to direction-specific processing units 58, 59, and 60. Each of the direction-specific processing units 58, 59, 60 has the yaw component extraction unit 51, pitch component extraction unit 52, roll component extraction unit 53, gain calculation units 54, 55, 56, and synthesis unit 57 shown in FIG. That is, in the directional processing unit 58, the low-frequency components of the quaternion QDs used for shaking are separated into yaw, roll, and pitch components, and gain calculations are performed using a yaw gain YG, a pitch gain PG, and a roll gain RG, and then the components are synthesized. In the direction-specific processing unit 59, the mid-range components of the quaternion QDs for shaking are divided into components in the yaw direction, roll direction, and pitch direction, and similarly subjected to gain calculations, and then synthesized. In the directional processing unit 60, the high frequency components of the quaternion QDs for shaking are divided into components in the yaw direction, roll direction, and pitch direction, and after similar gain calculations are performed on these components, they are synthesized. It is assumed that the gains used in the direction-specific processing units 58, 59, and 60 are different gain values. That is, the direction-specific processing unit 58 uses a low-frequency yaw gain YG, a low-frequency pitch gain PG, and a low-frequency roll gain RG, the direction-specific processing unit 59 uses a mid-frequency yaw gain YG, a mid-frequency pitch gain PG, and a mid-frequency roll gain RG, and the direction-specific processing unit 60 uses a high-frequency yaw gain YG, a high-frequency pitch gain PG, and a high-frequency roll gain RG. That is, it is conceivable that the direction-specific processing units 58, 59, and 60 use nine gains.

[0160] The outputs of these directional processing units 58, 59, and 60 are supplied to gain calculation units 44, 45, and 46, respectively, which provide a low-frequency gain LG, a mid-frequency gain MG, and a high-frequency gain HG, respectively. Then, the outputs are combined in a combination unit 61 and output as the value of the adjusted quaternion eQD.

[0161] In the above example of Fig. 19, the signal is first divided into frequency bands, and then the processing for each direction is applied to each band component, but the opposite is also possible, that is, the signal is first divided into directions, and then the processing for each frequency band is applied to each direction component. In this case, nine gains may be used in the processing for each frequency band. For example, in the processing for each frequency band in the yaw direction, a low-frequency gain LG for the yaw direction, a mid-frequency gain MG for the yaw direction, and a high-frequency gain HG for the yaw direction are used. In the processing for each frequency band in the pitch direction, a low-frequency gain LG for the pitch direction, a mid-frequency gain MG for the pitch direction, and a high-frequency gain HG for the pitch direction are used. In the processing for each frequency band in the roll direction, a low-frequency gain LG for the roll direction, a mid-frequency gain MG for the roll direction, and a high-frequency gain HG for the roll direction are used.

[0162] In step ST15 of FIG. 14, the adjusted quaternion eQD is generated by, for example, the above-described processing example. The generated adjusted quaternion eQD is then provided to the sway modification process in step ST16. The shaking change process in step ST16 can be considered as adding shaking by applying the adjusted quaternion eQD obtained by the processes in Figs. 17, 18 and 19 to an image in a state where shaking has stopped.

[0163] In the wobble change process in step ST16, the CPU 71 adds wobble to the image of the celestial sphere model MT to which the frame image has been pasted in step ST13 by rotating the image of the celestial sphere model hMT line by line using the adjusted quaternion eQD(#LN). The image of the wobble-changed celestial sphere model hMT is sent to the process in step ST18.

[0164] Then, in step ST18, the CPU 71 projects an image of the swing-changed celestial sphere model hMT onto a plane and cuts it out, thereby obtaining an image (output image data oPD) whose swing has been changed.

[0165] In this case, the sway change is achieved by rotating the celestial sphere model MT, and by using the celestial sphere model MT, no matter where you cut out the image, it will not become a trapezoid, so as a result, trapezoidal distortion is also eliminated. Also, as mentioned above, the celestial sphere model MT is attached to the celestial sphere surface as the area visible with an ideal pinhole camera, so there is no lens distortion. The rotation of the celestial sphere model MT is performed according to the adjusted quaternion eQD(#LN) based on the quaternion QD(#LN) for each line, and focal plane distortion correction is also eliminated. Furthermore, since the quaternion QD (#LN) corresponds to the exposure center of gravity of each line, the image has less noticeable blur.

[0166] The correspondence between the image after plane projection in step ST18 and the celestial sphere model MT is as follows: 20A shows an example of a rectangular coordinate plane 131 to be projected onto the plane. Each coordinate of the image to be projected onto the plane is represented as (x, y). 20B, the coordinate plane 131 is placed (normalized) in three-dimensional space so that it is directly above the celestial sphere model MT and touches the celestial sphere model MT at its center. In other words, the center of the coordinate plane 131 coincides with the center of the celestial sphere model MT and is placed in a position that touches the celestial sphere model MT.

[0167] In this case, the coordinates are normalized based on the zoom magnification and the size of the cropped region. For example, if the horizontal coordinate of the coordinate plane 131 is 0 to outh and the vertical coordinate is 0 to outv as shown in Fig. 20A, outh and outv are set as the image size. Then, for example, the coordinates are normalized by the following formula.

[0168]

number

[0169] In the above (Equation 10), min(A, B) is a function that returns the smaller value of A and B. Also, "zoom" is a parameter for controlling enlargement / reduction. Also, xnorm, ynorm, znorm are the normalized x, y, z coordinates. By each equation in the above (Equation 10), the coordinates on the coordinate plane 131 are normalized to coordinates on the spherical surface of a hemisphere with a radius of 1.0.

[0170] Rotation for obtaining the orientation of the cut-out region is performed by rotating the coordinate plane 131 using a rotation matrix calculation, as shown in Fig. 21A. That is, the rotation matrix shown in the following (Equation 11) is used to rotate the coordinates by the pan angle, tilt angle, and roll angle. Here, the pan angle is the rotation angle by which the coordinates are rotated around the z axis. The tilt angle is the rotation angle by which the coordinates are rotated around the x axis, and the roll angle is the rotation angle by which the coordinates are rotated around the y axis.

[0171]

number

[0172] In the above (Equation 11), "Rt" is the tilt angle, "Rr" is the roll angle, and "Rp" is the pan angle. Also, (xrot, yrot, zrot) are the coordinates after rotation.

[0173] These coordinates (xrot, yrot, zrot) are used to calculate corresponding points on the celestial sphere in perspective projection. As shown in Figure 21B, the coordinate plane 131 is perspectively projected onto the surface of the celestial sphere (area 132). In other words, the point at which a straight line drawn from the coordinates to the center of the celestial sphere intersects with the sphere is found. Each coordinate is calculated as follows:

[0174]

number

[0175] In (Equation 12), xsph, ysph, and zsph are coordinates obtained by projecting the coordinates on the coordinate plane 131 onto the coordinates on the surface of the celestial sphere model MT. In this relationship, image data projected onto a plane is obtained.

[0176] For example, a cutout region for an image projected onto a plane by the above-mentioned method is set in step ST17 of FIG.

[0177] In step ST17, the clipping region information CRA in the current frame is set based on a tracking process using image analysis (object recognition) and clipping region instruction information CRC according to a user operation. For example, in FIGS. 22A and 22B, the cut-out region information CRA set for an image of a certain frame is shown in the form of a frame. Such cut-out region indication information CRC is set for each frame. The cropping area information CRA also reflects an aspect ratio of the image specified by a user or automatic control.

[0178] The cut-out region information CRA is reflected in the process of step ST18. That is, as described above, a region on the celestial sphere model MT according to the cut-out region information CRA is projected onto a plane to obtain an output image oPD.

[0179] The output image oPD thus obtained is image data that has been subjected to the wobble modification process in step ST16. By performing the process of Fig. 14 for each frame, when the output image oPD is played back and displayed, an image with a shaking effect added as a shaking effect is displayed. Therefore, when the user performs an operation to input the shaking change parameter PRM, the image is one to which a shaking effect according to the user's intention has been added. Such image data is displayed or saved as an image with a shaking effect.

[0180] In this embodiment, for example, associated recording / playback is performed as step ST20. For example, the CPU 71 can perform associated recording for each frame (for example, recording in the storage unit 79 or the removable recording medium 81 in FIG. 5), and can also play back / transfer and output them when necessary.

[0181] In the frame-by-frame association recording process, for example, image capture shake information and shake modification information SMI are associated with image data in units of frames and recorded.

[0182] First, in step ST12, the CPU 71 performs recording processing of image data iPD for which the internal correction of the imaging device has been cancelled as frame-by-frame image data. In other words, this is image data in which the shake elimination in the imaging device 1 has been cancelled and the effects of shake such as camera shake at the time of imaging are still present.

[0183] Furthermore, the CPU 71 performs a process of recording the quaternion QD, timing information TM, and camera parameters CP for that frame as imaging shake information associated with this image data iPD. FIG. 23 shows metadata MTD1 supplied from the image source VS to the image processing device TDx, and metadata MTD2 supplied from the image processing device TDx to the image processing device TDy. Information corresponding to this metadata MTD2 is associated with each frame of the image data iPD as imaging shake information.

[0184] Furthermore, the CPU 71 performs a recording process of the swing change parameters PRM as the swing change information SMI associated with the image data iPD. Alternatively, instead of the swing change parameters PRM, or together with the swing change parameters PRM, the adjusted quaternion eQD may be recorded.

[0185] The image data iPD of each frame, the imaging shake information (metadata MTD2), and the shake modification information SMI, which are recorded in this manner in association with each other, are transferred to an external device and made available for use. In Fig. 2, image data VD2, metadata MTD2, and shaking modification information SMI are transmitted from image processing device TDx to image processing device TDy, but in the case of the example in Fig. 14, the image data VD2 becomes image data composed of image data iPD. Also, the metadata MTD2 indicates shaking information at the time of imaging as shown in Fig. 23. The shaking modification information SMI becomes information indicating the amount of processing of shaking effect performed on the image data iPD.

[0186] FIG. 24 shows an example of processing by the image processing device TDy that has acquired such image data VD2, metadata MTD2, and fluctuation modification information SMI. In the following description, the same steps as those already described will be denoted by the same step numbers to avoid duplication of explanation. As in FIG. 14, step ST30 is a "swing change" in a broad sense, and the processes of steps ST13, ST14, ST15, and ST16 are performed.

[0187] FIG. 24 shows, for example, the routine processing performed by the image processing device TDy. In this case, the associated recording / playback process in step ST20 involves playing back information transmitted from the image processing device TDx. In other words, the association recording / playback process of step ST20 is a process in which, for example, information transmitted from the image processing device TDx (image data VD2, metadata MTD2, jiggle modification information SMI) is recorded in the memory unit 79 or removable recording medium 81 of the information processing device 70 of Figure 5 functioning as the image processing device TDy, and is sequentially read out from them.

[0188] The CPU 71 of the information processing device 70 functioning as the image processing device TDy reads out image data pPD for each frame as the process of step ST20. The image data pPD refers to the image data reproduced from the recording medium in step ST20. When the image processing device TDx performs the process of FIG. 14, the reproduced image data pPD becomes the image data iPD described with reference to FIG. Furthermore, in the process of step ST20, the CPU 71 reads out the camera parameters CP corresponding to the image data pPD of the current playback frame in order to paste it onto the celestial sphere model MT in step ST13. Furthermore, in the process of step ST20, the CPU 71 reads out the quaternion QD and timing information TM corresponding to the image data pPD of the current reproduction frame for the synchronization process of step ST14. Furthermore, in the process of step ST20, the CPU 71 reads out the fluctuation modification information SMI corresponding to the image data pPD of the current reproduction frame for the fluctuation information adjustment of step ST15.

[0189] In step ST13, the CPU 71 refers to the reconstructed camera parameters CP and pastes the reconstructed image data pPD onto the celestial sphere model.

[0190] In step ST14, the CPU 71 performs synchronization processing using the quaternion QD corresponding to the reproduced image data pPD and the timing information TM. In the process of step ST15, the CPU 71 adjusts the quaternion QD using the reproduced swing change information SMI and the input swing change parameter PRM, and generates an adjusted quaternion eQD.

[0191] The CPU 71 sets the swing change parameter PRM in step ST41 based on the UI processing in step ST40, so that the swing change parameter PRM is used in the swing information adjustment processing in step ST15.

[0192] In the process of step ST16, the CPU 71 uses the adjusted quaternion eQD to rotate the image data pPD pasted onto the celestial sphere model MT, thereby performing a swing change process. In step ST18, the CPU 71 cuts out the image whose shaking has been changed in accordance with the cut-out region information CRA, projects it onto a plane, and generates and outputs output image data oPD. The image processing device TDy that performs each of these processes is capable of the following operations.

[0193] -Reproduction of the shaking effect previously achieved with the image processing device TDx In step ST15, the adjusted quaternion eQD can be generated by reflecting the swing change information SMI (e.g., swing change parameters PRM) in the image processing device TDx. Therefore, in step ST16, the CPU 71 of the image processing device TDy can perform the same swing change processing as the swing change previously performed in the image processing device TDx, and obtain such output image data oPD, which can be displayed or recorded.

[0194] -Addition of new shaking effects The image data pPD (image data iPD) to be processed is image data that has not been subjected to swing modification in the image processing device TDx. Therefore, the swing modification parameter PRM is input in the image processing device TDy by user operation or automatic control, and the adjusted quaternion eQD corresponding to the swing modification parameter PRM is generated, so that in step ST16, swing modification processing can be performed only in the image processing device TDy that is not affected by past swing modifications, and an image based on such output image data oPD can be displayed or recorded.

[0195] -Adjustments and corrections to the shaking effects previously implemented with the image processing device TDx In step ST15, an adjusted quaternion eQD can be generated by reflecting both the swing change information SMI (eg, swing change parameters PRM) in the image processing device TDx and the swing change parameters PRM newly input by user operation or automatic control. Therefore, the CPU 71 of the image processing device TDy can change the degree of shaking of the image caused by the shaking change performed in the image processing device TDx, for example, in response to the user of the image processing device TDy adjusting or partially changing the shaking change parameter PRM, and can display or record an image based on such output image data oPD.

[0196] Furthermore, when a swing change is performed in the image processing device TDy, the information on the swing change may also be associated. That is, the shaking change information SMI associated with the image data VD2 (iPD) and the imaging time shaking information (metadata MTD2) of each frame is updated to the shaking change information SMI used this time. In this way, the other image processing device that has acquired the associated information group can reproduce or modify the swing change performed in the image processing device TDy.

[0197] It is also possible to associate and record information related to the cutout area information CRA in step ST17. For example, information on the aspect ratio may be associated with the information. This allows the aspect ratio designation information to be reflected when playing back images or adding a shaking effect. Of course, in the processing in the image processing device TDx in FIG. 14, the cut-out region information CRA such as the aspect ratio may also be recorded.

[0198] <5. User interface for changing sway> In the above, it has been explained that the swing change is performed in step ST30 in FIG. 14 or FIG. As for shaking modification, as described above, shaking removal can be achieved by changing the shaking state in a direction that eliminates shaking between frames (ideally making the shaking zero), or by changing the shaking state to add or reduce shaking as a shaking effect.

[0199] Here, from the viewpoint of a person watching a video, a certain amount of shaking may give a greater sense of impact and realism than an image without any shaking at all. Also, there are cases where a person wants to leave some shaking rather than completely remove it. In this embodiment, for example, in steps ST40 and ST41 in Figs. 14 and 24, the swing change parameter PRM is set in response to a user operation, so that the user operation is reflected in the swing change. Below, we will explain various examples of UI actions for changing the sway.

[0200] [5-1: UI operation of the first embodiment] Fig. 25 shows a swing range adjustment image 200 displayed as a UI image. This is an image for adjusting the swing range to be displayed on the display unit 77 in Fig. 5, for example, and also functions as the input unit 76 using a touch panel or the like. The sway width adjustment image 200 is provided with a yaw adjustment section 201, a roll adjustment section 202, and a pitch adjustment section 203.

[0201] Each of the yaw adjustment unit 201, roll adjustment unit 202, and pitch adjustment unit 203 is presented with an image of a slider for operating each band of sway frequencies, designated L (low range), M (mid range), and H (high range), and the user can adjust the sway amplitude by touching any of the sliders. For example, the user can use the shaking width adjustment image 200 to adjust the amount of shaking for each desired frequency band for the shaking in the yaw direction (horizontal shaking), pitch direction (vertical shaking), and roll direction (rotational shaking).

[0202] In the UI processing of step ST40 in Figures 14 and 24, this swing range adjustment image 200 is displayed, and when a user operation is detected, a swing change parameter PRM is set in step ST41 according to the operation information, and is reflected in the swing change processing of step ST16 via the processing of step ST15 as described above.

[0203] The sway range adjustment image 200 in Figure 25 is an example corresponding to an example of generating an adjusted quaternion eQD such as that shown in Figure 19, and specifically, the above-mentioned low-frequency gain LG, mid-frequency gain MG, high-frequency gain HG, yaw gain YG, pitch gain PG, and roll gain RG are set as the sway change parameter PRM in response to the operation of the sway range adjustment image 200.

[0204] In the wobble change process, the amount of rotation for removing the wobble is changed based on the operation information in response to the operation of the yaw, pitch, and roll directions of the image pasted on the celestial sphere model MT, or rotation is added to produce a wobble effect in response to the operation.

[0205] By providing such a swing range adjustment image 200, the user can adjust the swing state of each of the yaw axis component, pitch axis component, and roll axis component separately for each band, that is, low frequency, mid frequency, and high frequency. Therefore, the output image oPD can be an image with a variety of shake expressions, for example, an image in which the shake in the roll direction and yaw direction is suppressed but the shake in the pitch direction is left to a certain extent, or an image in which high-frequency shake is suppressed but large shake is left uncorrected. In other words, by adjusting the target and degree of shake correction, a more diverse range of moving image expressions becomes possible.

[0206] In Fig. 25, each of the shaking directions can be operated in three bands, but of course other examples are also possible. For example, there is a case where only the yaw direction, pitch direction, and roll direction can be operated and adjustment for each frequency band is not possible, a case where the direction cannot be specified and only adjustment is possible for each band such as low, mid, and high, a case where only one direction is adjustable, and a case where only one band is adjustable.

[0207] [5-2: UI operation of the second embodiment] FIG. 26 shows an example of a shake width adjusted image 200 according to the second embodiment. The yaw adjustment unit 201 is for adjusting lateral shaking, and is therefore provided with a slider that can be slid laterally. The roll adjustment unit 202 adjusts the swing in the rotational direction, and is therefore provided with a slider that can be slid in a circular direction. Since pitch adjustment section 203 adjusts the vertical swing, it is provided with a slider that can be slid vertically. This allows the user to easily understand the direction of the swing that can be adjusted, and allows the user to easily perform the desired operation.

[0208] [5-3: UI operation of the third embodiment] FIG. 27 shows an example in which the adjustment operation for the shaking state can be intuitively performed. A shaking width adjustment image 200 is displayed as shown in FIG. 27A, and preset buttons 210 are displayed as shown in FIG. 27B. The preset buttons 210 are prepared, for example, for a "slow walker (Slow)", a "walker (Walk)", a "runner (Fast)", etc. In addition, swing change parameters PRM corresponding to these preset buttons 210 are recorded in the ROM 72 or the storage unit 79. When the user presses a certain preset button 210, the swing change parameter PRM is read out. Also, an operation state according to the swing change parameter PRM is expressed. For example, as shown in Fig. 27C, the content of the preset data is presented to the user as the position of the slider of the swing width adjustment image 200.

[0209] In this way, the user can know the operation state of the slider according to the video from the viewpoint of a person walking slowly, a person walking, a person running, etc. Also, the swing change parameter PRM according to the state of the slider at that time is set and reflected in the swing change process, so that the user can check the video from the viewpoint of a person walking, a person running, etc. from the played image. In particular, adjusting the sliders to produce the desired effect is difficult for anyone other than an experienced user, but the use of such preset buttons 210 makes it easier for the user to perform operations to obtain the image they desire. Of course, after the slider position is changed based on the preset information, the user can make fine adjustments, making it easy to achieve the swing that the user desires.

[0210] 28 is an example in which only the preset buttons 210 are presented without using the shaking width adjustment image 200. Furthermore, this is also an example in which a variety of presets can be selected. First, as shown in FIG. 28A, preset buttons 210 corresponding to major categories such as "people," "animals," and "vehicles" are presented. If the user selects "Animals", then, as shown in Fig. 28B, preset buttons 211 for subcategories such as "Rabbit", "Cat", "Horse", etc. are presented. Swing change parameters PRM that match these preset buttons 211 are recorded in the ROM 72 or the storage unit 79.

[0211] If the user selects the "rabbit" preset button 211, preset data (sway width adjustment data) corresponding to the swaying of the rabbit when it is running is read out, and the swaying change parameter PRM is set based on the preset data. Then, the swaying change process is performed based on the swaying change parameter PRM.

[0212] For example, by presenting preset buttons 210, 210 hierarchically as in Figs. 28A and 28B, the user can select from a variety of types of shaking, and can easily instruct the desired shaking correction.

[0213] It should be noted that there are many more possible examples of user interfaces that display such controls as sliders and preset buttons.

[0214] [5-4: UI processing in the first, second and third embodiments] An example of UI processing for carrying out the UI processing in the above-mentioned first, second and third embodiments is shown in Fig. 29. This is processing executed by the CPU 71 using the functions of the UI processing unit 103 and the parameter setting unit 102.

[0215] In step S101, the CPU 71 performs processing for displaying a UI and detecting an operation. For example, the CPU 71 detects user operations on the preset buttons 210 and 211 shown in FIGS. 27 and 28 while controlling the display of the shaking width adjustment image 200 as shown in FIG. 25 or 26 and the display of the same. The CPU 71 also switches the display in response to an operation, for example, switching the slider position. When a hierarchical structure of the preset buttons 210, 211 is used, the CPU 71 operates, for example, a large category preset button 210 to display the next hierarchy, for example, a small category preset button 211.

[0216] When the CPU 71 detects the operation of the slider of the shake width adjustment image 200, the process proceeds from step S102 to step S103, where the CPU 71 acquires the operation information (operation value of the slider), and in step S104 sets the shake change parameter PRM according to the operation information and outputs it to the shake change process (step ST30 in FIG. 14).Then, the CPU 71 returns to step S101.

[0217] When the CPU 71 detects the operation of selecting preset data, the process proceeds from step S105 to step S106, and the CPU 71 reads out the preset data corresponding to the operation. Then, the CPU 71 performs display control so that the value of the preset data is reflected in the position of the slider as shown in FIG. 27C. In step S108, the CPU 71 sets the swing change parameter PRM based on the preset data, outputs it to the swing change process (step ST30 in FIG. 14), and returns to step S101.

[0218] By the above processing, the UI display and its operation correspondence as shown in FIG. 25 (or FIG. 26), FIG. 27, and FIG. 28 can be realized, and the shaking change processing according to the slider of the shaking width adjustment image 200 and the preset buttons 210, 211 can be executed.

[0219] [5-5: UI operation of the fourth embodiment] An example of a UI operation according to the fourth embodiment in which a timeline is displayed will be given. 30A is an example of a timeline 250 that the CPU 71 causes the display unit 77 to display. The timeline 250 shows a video from the beginning to the end, with the horizontal direction being the time axis direction. The timeline 250 is made up of an image timeline 251, a vibration timeline 252, and a sound timeline 253.

[0220] In the image timeline 251, thumbnail images of frames constituting a moving image are intermittently extracted and displayed in chronological order. In the vibration timeline 252, the vibration state is displayed, for example, by a waveform. Although one vibration waveform is shown here, it may be displayed separately, for example, as a yaw direction vibration waveform, a pitch direction vibration waveform, and a roll direction vibration waveform. Furthermore, the vibration waveform may be displayed separately for each frequency band. The audio signal waveform added to the video is displayed on the audio timeline 253. Of course, in the case of stereo audio of the L channel and the R channel, each audio signal waveform may be displayed.

[0221] For example, by displaying such a timeline 250, the user can check the shaking state in accordance with changes in the video scene and changes in the audio situation. This makes it convenient to create or remove shaking effects within a specified section.

[0222] The vibration information is associated with the image frames as IMU data or quaternion QD, for example, and the vibration timeline 252 displays a vibration waveform based on this vibration information.

[0223] It may be possible to record and read only the vibration information represented by this vibration waveform onto a recording medium. For example, by performing a specified operation such as placing the cursor on the shaking timeline 252 and right-clicking, a save window 260 for saving the shaking information is displayed as shown in FIG. 30B, allowing the user to select "Save shaking information" or "Open shaking information."

[0224] When the user clicks "Save vibration information", for example, a confirmation window 261 shown in Fig. 30C is displayed. The confirmation window 261 is provided with an OK button 262 and a cancel button 263 so that the user can confirm his / her intention. If only the shaking information is saved, that is, if the shaking information is saved separately from the image and audio, the synchronization information between the shaking information and the image will be lost. Therefore, the user is notified that the synchronization information will be lost and asked to confirm OK / cancel. When the user operates the OK button 262, the CPU 71 performs processing for saving the vibration information. If the user operates the cancel button 263, the CPU 71 does not perform the process of saving the shaking information, and returns to the display state of the timeline 250 in FIG. 30A, for example.

[0225] If the user clicks on "Open Shake Information", for example, the shake information may be enlarged to display an image that can be adjusted in detail, or a list of numerical values ​​as IMU data or quaternion QD may be displayed, allowing the user to check or edit in detail. By enlarging and displaying the shaking waveform on the time axis and making it possible to change the waveform itself (by changing the amplitude or frequency), it may be possible to adjust the amount of shaking in a certain section of the video or even eliminate the shaking.

[0226] [5-6: UI operation of the fifth embodiment] The fifth embodiment is an example in which preset data can be assigned on a timeline 250.

[0227] FIG. 31 shows a timeline 250A with the horizontal direction as the time axis direction. In this case, the timeline 250A is provided with an image timeline 251, a sound timeline 253, a vibration timeline 252, and an assignment frame 254. The assignment frame 254 is an area for the user to arbitrarily assign preset data.

[0228] For example, preset buttons 210 and 211 are displayed together with a timeline 250A on the screen of the display unit 77. The user can assign preset data by, for example, dragging and dropping an arbitrary preset button 211 into an assignment frame 254. For example, by performing a drag / drop operation DG1 on the rabbit button in preset buttons 211, preset data display 255 of the rabbit is displayed in assignment frame 254 as shown in FIG. 32A, indicating that the preset data of the rabbit shaking has been assigned to the corresponding section in the video. Similarly, by performing a drag / drop operation DG2 on the horse button, preset data display 256 of the horse is displayed in assignment frame 254 as shown in FIG. 32A, indicating that preset data for the horse's swaying has been assigned to the corresponding section in the video.

[0229] When preset data of a rabbit or preset data of a horse is assigned to a certain section as in FIG. 32A, this means that the swinging state of that section is changed to swinging like a rabbit or swinging like a horse. That is, in this section, the swing change parameters PRM corresponding to the rabbit preset data and the horse preset data are set and the swing change is performed. For this reason, in the section to which the preset data is assigned, the original vibration shown in the vibration timeline 252 is no longer reflected in the image. Therefore, the vibration waveform in the corresponding section is displayed in an inconspicuous manner as shown in the figure (shown by a dashed line in the figure), or is erased or displayed in gray, so that the user can understand this.

[0230] Depending on the drag / drop operation of the preset data, the preset data to be operated is assigned to a section of a fixed time length, for example, and then the user can adjust the section as desired within the assignment frame 254. For example, the section length of the rabbit preset data and the section length of the horse preset data can be widened or narrowed by performing an operation such as dragging on the edge of the preset data display 255, 256.

[0231] Also, in the figure, a cross-fade display 257 is performed at the boundary of the preset data (the boundary between the rabbit section and the horse section in FIG. 32A), and near this boundary, the swaying state is shown gradually changing from that of a rabbit to that of a horse. The user can also change each section by moving a portion of the cross-fade display 257 or by moving the ends, for example, from the state of FIG. 32A to the state of FIG. 32B.

[0232] By providing UI screens such as those shown in Figs. 31 and 32 above, it becomes very easy for the user to assign preset data to each scene in a video.

[0233] [5-7: UI operation of the sixth embodiment] The sixth embodiment is an example in which a timeline display and a playback image display enable an operation to be performed while checking a vibration waveform and a playback display to which vibration has been applied.

[0234] An example screen is shown in Fig. 33. On this screen, for example, a shaking width adjusted image 200, a monitor display 270, a playback display section 271, and a shaking timeline 252 are presented to the user. The shaking width adjustment image 200 allows the user to specify the amount of shaking in each direction by operating a slider, for example, as described with reference to FIG. 25 and FIG.

[0235] The waveform monitor 270 displays vibration waveforms in the vicinity of the current position (key frame) for each of the yaw, roll, and pitch directions, for example. In the figure, each shaking waveform is shown as “before adjustment”, which is the waveform before adjustment by operating the shaking width adjustment image 200 . The waveform monitor 270 is provided with a switching button 279, and when the user operates the switching button 279, the displayed vibration waveform is changed to the "adjusted" one. In other words, the waveform is switched to the one adjusted by operating the vibration width adjustment image 200 and displayed. By, for example, toggling the switching button 279, the user can alternate between viewing the vibration waveforms "after adjustment" and "before adjustment," and visually confirm the degree of adjustment. Of course, the vibration waveforms "after adjustment" and "before adjustment" may be displayed side by side.

[0236] The horizontal direction of the shaking timeline 252 is the time axis direction, and shows a shaking waveform 283. In this example, for example, the shaking waveform 283 from the beginning (0 minutes 00 seconds) to the end (10 minutes 00 seconds) of a 10-minute video is shown. The points that have been designated as key frames by user operations are displayed clearly as key frame displays 281. The user can set a frame at any point in time as a key frame by moving a key frame operator 282. In this example, key frames can be set on the vibration timeline 252, but key frames may also be set on the image timeline 251 or sound timeline 253 described above.

[0237] A playback image is displayed in the playback display portion 271. Near the playback display portion 271, a playback repetition period designation button 272, a repetition period length 273, and a playback operation portion 274 are displayed. A repeat button, a pause button, a play button, etc. are displayed on the playback operation section 274. The user can use these controls to turn repeat playback on / off, pause video playback, start playback, etc.

[0238] The playback repeat period designation button 272 is an operator for repeatedly playing back a relatively short section, and for example, the playback period length is changed in response to the operation. The playback period length specified by the operation is displayed as repeat period length 273. In the figure, it indicates that a section of 3 minutes and 0 seconds is to be repeated.

[0239] This repeated playback is performed within a section based on a key frame designated by the user on the swing timeline 252 . For example, the repeat playback start position is set to a key frame. In the case of the figure, when the user performs a playback operation, a 3-minute section from the position indicated by the key frame display 281 on the swing timeline 252 is repeatedly played back as a repeat playback period. If the user wishes to repeat playback of a shorter section, the user can shorten the repeat playback period by operating playback repetition period designation button 272. If the user wishes to play back a different section, the user can move key frame operator 282.

[0240] While watching the repeat playback, the user can adjust the state of shaking by using the shaking width adjustment image 200. This allows the user to adjust the shaking to suit the scene while actually watching the image. The image to be reproduced is assumed to be the output image data oPD of Fig. 14 or Fig. 24. That is, it is an image to which the shaking has been changed by the shaking change parameter PRM. Therefore, when the user performs an operation to adjust the shaking state using the shaking width adjustment image 200, the shaking is reflected in the reproduced image, so the user can operate it to any shaking state while viewing the actual shaking state.

[0241] The section to be repeatedly played back is based on a key frame, but the key frame need not necessarily be the beginning of the section to be played back. For example, a frame a predetermined time before the key frame (eg, 5 seconds before) may be set as the playback start frame. Alternatively, the playback start frame may be determined so that the key frame is located in the center of the section to be repeatedly played back.

[0242] Also, there are various possibilities as to which section of the image the vibration state adjusted by the vibration width adjustment image 200 while checking the repeat playback, that is, the vibration change parameter PRM set in response to the operation, should be applied to.

[0243] For example, the swing change parameter PRM according to the operation may be applied only to the section being repeatedly played back. Furthermore, regardless of the section being repeatedly played back, the shaking change parameter PRM according to the operation may be applied to the section from the key frame to the end of the image. Furthermore, regardless of the section being repeatedly played back, the shaking change parameter PRM according to the operation may be applied to the section from the beginning of the image to the key frame. Also, a swing change parameter PRM according to an operation may be applied to a predetermined time section before and after a designated key frame. Furthermore, regardless of the section being repeatedly played back, the shaking change parameter PRM according to the operation may be applied to the entire section of the video.

[0244] Also, a frame as a scene change point may be detected, and a swing change parameter PRM according to an operation may be applied to a section determined to be a scene including a key frame. Also, a frame as a scene change point may be detected, and a swing change parameter PRM according to an operation may be applied to the section from a key frame to the next scene change point, or to the section from the scene change point preceding the key frame to the key frame. If the main subject of the photograph can be determined by automatic scene recognition, it is also possible to automate the shake adjustment according to the movements of the main subject.

[0245] As an example of an application when you do not want to apply the swing change parameter PRM according to the operation before the key frame, you can create a new key frame at the last point where you do not want to apply it and input the default swing change parameter PRM. The default swing change parameter PRM is, for example, a value when all the gains specified by the sliders of the swing width adjustment image 200 are set to "1". As an example of limiting the application range of the swing change parameter PRM according to the operation, it is conceivable to perform the operation by using the areas before and after the point to be limited as key frames.

[0246] In the case of adjustments that remove all jitter, it is also possible to provide a check box for the entire track represented by the jitter timeline 252 so that jitter removal can be instructed.

[0247] For example, in the vibration timeline 252, the color of the vibration waveform 283 may be changed so that the user can determine whether or not adjustments (vibration changes) have been made, or a line graph-like line may be displayed superimposed on the vibration waveform 283 to connect the marks of the key frames.

[0248] [5-8: UI processing in the fourth, fifth and sixth embodiments] An example of UI processing for carrying out the UI processing in the above fourth, fifth and sixth embodiments will be described with reference to Fig. 34 and Fig. 35. This is processing executed by the CPU 71 using the functions of the UI processing unit 103 and the parameter setting unit 102. 34 and 35 show a series of processes divided into parts, and the connections in the flowcharts are indicated by "c1" and "c2". This process also assumes that in the display of Figure 33, the timeline 250A and preset buttons 210, 211 of Figures 31 and 32 are displayed instead of the shaking timeline 252, and that key frames are displayed in the shaking timeline 252 in timeline 250A as shown in Figure 33.

[0249] In step S200, the CPU 71 performs processing for displaying a UI and detecting an operation. For example, the CPU 71 controls the display unit 77 to display a shaking width adjustment image 200, a waveform monitor 270, a playback display unit 271, a playback repetition period designation button 272, a repetition period length 273, a playback operation unit 274, a switching button 279, a timeline 250A (image timeline 251, sound timeline 253, shaking timeline 252, assignment frame 254), and preset buttons 210 and 211. The CPU 71 also detects user operations on the screens displaying these. The CPU 71 also switches the display in response to the operation, for example, switching the slider position, etc. In response to the hierarchical structure of the preset buttons 210 and 211, the CPU 71 displays the next hierarchical level, for example, the sub-level preset buttons 211, in response to the operation of the major level preset button 210.

[0250] The CPU 71 branches the process in response to detection of a user operation at steps S201, S202, S203, S204, S205, S206, S207, S208, S209, and S210 in FIGS. 34 and 35.

[0251] When the CPU 71 detects a save operation by the user, the process proceeds from step S201 to step S215 in Fig. 34. The save operation is, for example, an operation of right-clicking on the shaking timeline 252 and selecting "save shaking information" in the save window 260 as described in Fig. 30. In this case, the CPU 71 performs a save process in step S215. For example, the CPU 71 displays a confirmation window 261 (see FIG. 30C), and if the user operates the cancel button 263, the process returns to step S200 without saving, and if the user operates the OK button, the process performs a process of saving the shaking information in the storage unit 79 or the like, and returns to step S200.

[0252] When the CPU 71 detects that the user has performed a key frame designation operation, the process proceeds from step S202 to step S230 in Fig. 34 to set a key frame. That is, the key frame display 281 is performed at a position on the shaking timeline 252 according to the operation of the key frame operator 282, and the frame number is specified to set it as a key frame. The frame number can be, for example, the hour, minute, and second frame number with the first frame of the video being 0:00:00 frame.

[0253] Next, the CPU 71 sets the target section in step S231. This is a process for setting a section to which the shaking change by the user operation is applied, based on the key frame. As described above, there are various methods for this setting, but for example, the section from the key frame to the end of the image is set as the application section. The CPU 71 returns to step S200 via the processes of steps S230 and S231 described above.

[0254] When the CPU 71 detects that the user has performed a monitor switching operation, i.e., operated the switching button 279, the process proceeds from step S203 to step S240, and switches the display on the waveform monitor 270. In this case, if the display on the waveform monitor 270 is before adjustment, the CPU 71 performs processing to switch to the waveform after adjustment, and if the display on the waveform monitor 270 is after adjustment, the CPU 71 performs processing to switch to the waveform before adjustment. Then, the process of the CPU 71 returns to step S200.

[0255] When the CPU 71 detects that the user has performed a shake width adjustment operation, that is, an operation of the slider of the shake width-adjusted image 200, the process proceeds from step S204 to step S220, and it is confirmed whether a target section in which the operation is reflected has been set. If the target section has not been set, that is, if the user has not set a key frame, the CPU 71 proceeds to step S222, and for example, sets the entire section of the video as the target section, and applies the swing change parameter PRM according to the current slider operation value to the entire section. Of course, setting the entire section as the target is just one example. On the other hand, if a target section has been set by the process of step S231 up to that point, the CPU 71 proceeds to step S221 and applies the swing change parameter PRM corresponding to the current slider operation value to the set target section. Then, the process of the CPU 71 returns to step S200.

[0256] If the CPU 71 detects a preset assignment operation by the user, the process proceeds from step S205 to step S234. The preset assignment operation is an operation for assigning certain preset data to a certain section, such as the drag / drop operations DG1 and DG2 in FIG. In this case, the CPU 71 sets the section to be assigned in step S234, and applies the swing change parameter PRM based on the assigned preset data to that section. Further, the CPU 71 causes the assignment frame 254 to display preset data 255, and also changes the display (for example, the dashed line display in FIG. 32A) on the vibration timeline 252 to indicate that the original vibration will no longer be applied. Then, the processing of the CPU 71 returns to step S200.

[0257] When the CPU 71 detects an operation by the user to change the preset assignment section, the process proceeds from step S206 to step S235, for example, when the user performs an operation to change the preset assignment section as described with reference to FIG. In this case, the CPU 71 sets a new section to be assigned in step S235, so that the section to which the perturbation change parameter PRM based on the preset data is applied is changed.Then, the process of the CPU 71 returns to step S200.

[0258] When the CPU 71 detects that the user has performed a playback operation, that is, operated the playback button of the playback operation unit 274, the process proceeds from step S207 to step S250 in FIG. 35, and it is checked whether a key frame has been set.

[0259] If the user has not set a key frame, the CPU 71 proceeds to step S251, where it sets the top frame of the moving image as the playback start frame. Then, in step S253, playback start control is performed, whereby the video is played back from the beginning on the playback display section 271. The section to be repeatedly played back is the section corresponding to the time displayed in the repeat period length 273 .

[0260] If the user has set a key frame, the CPU 71 proceeds from step S250 to step S251, and sets a playback start frame, which is the beginning of the section to be repeatedly played back, in accordance with the key frame setting. As described above, there are cases where the key frame itself is set as the playback start frame, and there are also cases where the playback start frame is set near the key frame. Then, in step S253, playback start control is performed, whereby the video is played back in the playback display section 271 from the frame corresponding to the key frame setting. The section to be repeatedly played back is the section corresponding to the time displayed in the repeat period length 273, starting from the frame corresponding to the key frame setting.

[0261] After the CPU 71 performs playback start control in step S253, the process returns to step S200 in FIG.

[0262] It should be noted that the section to be repeatedly played back, which is set in step S252, does not necessarily have to match the section to which the swing change by the user operation, which is set in the above-mentioned step S231, is applied. However, these may be matched and the swing change may be applied to the section that is repeatedly played back. In this case, the swing change is applied only to the section where the user can confirm the swing change, so that an interface that is easy for the user to understand can be provided. Conversely, if the user wants to check a portion of the video and apply the shaking to a wider range, it is desirable to apply the shaking change to a wider section than the section being played back repeatedly. Furthermore, when adding a shaking effect to a very short section and wanting to check the sections before and after it, it is desirable to apply a shaking change to a portion of the section being played back repeatedly.

[0263] When the CPU 71 detects that the user has performed a stop operation, that is, operated the stop button of the playback operation unit 274, the process proceeds from step S208 to step S260 in Fig. 35 to perform playback stop control, and then the process returns to step S200 in Fig. 34. In addition, playback stop and playback pause may be distinguished from each other. For example, when a playback pause operation different from a playback stop operation is detected, the CPU 71 performs a process of stopping the playback image at the frame at that time.

[0264] 35, that is, when the CPU 71 detects an operation of the playback period length, i.e., an operation of the playback repetition period designation button 272, the CPU 71 proceeds to step S265 and changes the playback period length. That is, the CPU 71 changes the setting of the playback period length and changes the display of the repetition period length 273 to the new time length. Then, the processing of the CPU 71 returns to step S200 in FIG. The playback period length updated in step S265 is reflected in the subsequent repeat playback or the currently ongoing repeat playback, allowing the user to adjust the section to be repeated.

[0265] When the CPU 71 detects in step S210 of FIG. 35 that the repeat playback has reached the playback end frame, the process proceeds to step S270 to perform repeat playback processing. The playback end frame is a frame located a set playback period from the playback start frame set in step S251 or step S252. The CPU 71 performs control to perform repeat playback processing by returning to the playback start frame, and then returns to step S200.

[0266] By the above-mentioned processing in FIGS. 34 and 35, the UI displays and their corresponding operations as shown in FIGS. 30, 31, 32, and 33 are realized.

[0267] <6. Summary and variations> The above embodiment provides the following advantages. The image processing devices TDx and TDy of the embodiment include a swing change unit 101 that performs swing change processing to obtain swing-changed image data by changing the swing state of input image data that constitutes a moving image, a UI processing unit 103 that presents operators related to swing change and obtains operation information by the operators, and a parameter setting unit 102 that sets a swing change parameter PRM based on the operation information obtained by the UI processing unit 103. This makes it possible to provide the user with an operating environment that allows image processing such as adding shaking to an image or suppressing shaking of an image. In particular, by making it possible to visually manipulate parameters for changing the swing on the UI screen, it is possible to provide a swing manipulation environment that is easy for users to understand.

[0268] In the embodiment, the UI processing unit 103 presents an operator for adjusting the amount of shaking, and the parameter setting unit 102 sets a parameter for adjusting the amount of shaking based on the operation information of the operator. For example, the process corresponds to the display of the shaking width adjustment image 200 shown in Fig. 25, Fig. 26, Fig. 27, and Fig. 33 and the operation of the image. This allows the user to adjust the amount of shaking of the image as desired.

[0269] In the embodiment, an example is given in which the UI processing unit 103 presents an operator for adjusting the amount of sway for one or more sway direction components, and the parameter setting unit 102 sets a parameter for adjusting the amount of sway for the sway direction component based on operation information for the amount of sway for the sway direction component. That is, the shaking appearing in the image contains a plurality of shaking direction components, but the amount of shaking can be adjusted for each of the shaking direction components or for each of the shaking direction components. This allows the user to adjust the amount of shaking for each shaking direction, for example, to adjust the shaking in a specific direction. Also, the amount of swing may be adjustable for a plurality of swing direction components. By adjusting the amount of swing for each directional component, a variety of swing changes can be realized. For example, the user can make adjustments such as reducing only the vertical shaking, or increasing the vertical shaking and decreasing the horizontal shaking.

[0270] In particular, in the embodiment, an example has been given in which the vibration direction components include all or part of the yaw direction vibration component, the roll direction vibration component, and the pitch direction vibration component. By being able to adjust all or part of the sway components in the yaw, roll, and pitch directions, the user can perform operations such as removing or increasing specific sway, or producing a variety of sway, using an easy-to-understand interface.

[0271] In the embodiment, an example is given in which the shaking change unit 101 performs gain processing based on shaking change parameters for each shaking direction on shaking information for adding shaking based on shaking information at the time of imaging (e.g., quaternion QDs for shaking), and generates adjusted shaking information. As in the processing in Figures 18 and 19, gain processing is performed for each direction of shaking (yaw, pitch, roll), and by mixing, an adjusted quaternion eQD can be generated in which the amount of shaking is adjusted for each direction of shaking. In this way, it is possible to perform arbitrary adjustments for each direction of shaking and create a variety of shaking effects. For example, it is easily possible to create a shaking effect in which vertical shaking is added but horizontal shaking is not added.

[0272] In the embodiment, an example is given in which the UI processing unit 103 presents an operator for adjusting the amount of vibration for one or more vibration frequency bands, and the parameter setting unit 102 sets a parameter for adjusting the amount of vibration for the vibration frequency band based on operation information for the amount of vibration for the frequency band. The shaking that appears in an image contains vibration components of multiple frequencies. Therefore, it is possible to adjust the amount of shaking for each frequency band or for multiple frequency bands. This allows the user to adjust the shaking based on the type of shaking, such as large shaking or small shaking. For example, adjustments can be made to remove or add only small fluctuations. In addition, by making it possible to adjust the amount of vibration for vibration components in multiple frequency bands, it is possible to achieve a variety of vibration adjustments. For example, the user can adjust the vibration to reduce large vibrations while emphasizing small vibrations.

[0273] In particular, in the embodiment, an example is given in which operators are presented to adjust the amount of swing in at least the low and high frequency bands as frequency bands, and the parameter setting unit 102 sets a parameter for adjusting the amount of swing in the low frequency band and a parameter for adjusting the amount of swing in the high frequency band. For example, the vibration can be adjusted in two bands, a low frequency band and a high frequency band, or it can be adjusted in three or more bands, a low frequency band, one or more intermediate frequency bands, and a high frequency band. This allows the user to adjust the amount of shaking for each band, making it possible to produce an image with any degree of shaking. By providing an interface that allows adjustment for each band, it becomes easier for the user to predict the degree of shaking that will be achieved in response to the adjustment operation, improving operability.

[0274] In the embodiment, an example is given in which the shaking change unit 101 performs gain processing based on the shaking change parameter PRM for each shaking frequency band on shaking information for adding shaking based on shaking information at the time of imaging (e.g., quaternions QDs for shaking), to generate adjusted shaking information. As in the processing of Figures 17 and 19, bands are divided, gain processing is performed, and then mixing is performed to generate an adjusted quaternion eQD in which the amount of shaking is adjusted for each frequency band of shaking. In this way, it is possible to perform arbitrary adjustments for each frequency band of shaking to produce a variety of shaking effects. For example, it is easy to create a shaking effect in which large shaking is suppressed while small shaking is increased.

[0275] In the embodiment, the UI processing unit 103 presents operators (preset buttons 210, 211) for selecting preset data of a shaking waveform to enable a selection operation of preset data, and the parameter setting unit 102 sets parameters for adjusting the shaking of an image in response to the operation of selecting preset data (see Figs. 27, 28, and 31). This allows the user to very easily change the vibration by specifying and selecting the prepared preset data. For example, by preparing one or more preset data for a shaking pattern in which changes in the amount of shaking, frequency components, shaking direction components, etc. are set in advance, the user can change the shaking by simply selecting the preset data. Also, a plurality of preset data may be prepared, and the user may select the desired swing.

[0276] In the above example, preset data is provided for the swaying of people, animals, vehicles, etc., so that the user can intuitively perform operations on the video to change the swaying in accordance with the image content and editing intent. In particular, with regard to swaying, even if you do not have detailed knowledge of frequency components or directional components such as yaw, roll, and pitch, you can imagine the swaying of an animal or a vehicle and specify it, making operation very easy.

[0277] In addition, for example, it is expected that preset data will be prepared by further subdividing the swaying of a person into swaying when walking slowly, swaying when walking normally, swaying when running, swaying when running vigorously, etc. For animal movements, if you prepare preset data for various animal types such as rabbits, cats, horses, dogs, frogs, etc., you can edit a variety of movements. For vehicle swaying, if you prepare preset data for various vehicles such as cars, trucks, trains, and airplanes, you can edit a variety of swaying effects. Furthermore, it is also possible to prepare preset data according to the type of road surface condition, such as asphalt roads, dirt roads, sand, ice, etc., and to prepare preset data according to sea conditions, such as calm, light winds, and strong winds on a ship.

[0278] In the embodiment, an example has been given in which the UI processing unit 103 executes the presentation of the timelines 250 and 250A indicating the shaking component in the time axis direction of the image (see FIGS. 30, 31, 32, and 33). This allows the user to easily check the state of shaking in each section (each scene) of a video. This is very suitable for users to check the overall tendency and state of shaking in a video, or to check sections where they want to perform shaking effects or remove shaking.

[0279] In the embodiment, an example has been given in which the UI processing unit 103 allows a user to specify a key frame on the timeline 250A (shake timeline 252) through a user operation (see steps S202, S230, S231, etc. in FIGS. 33, 34, and 35). This allows the user to arbitrarily specify a certain point (section, scene) in a video. This is a convenient function for specifying a playback section or a jitter editing point.

[0280] In the embodiment, an example has been given in which the UI processing unit 103 performs processing to play back an image based on a key frame specified by a user operation on the timeline 250A (shaking timeline 252) (see steps S202, S230, S207, S250, S252, S253, etc. in Figures 33, 34, and 35). This allows the user to select a desired portion of the video and instruct it to be played back, thereby enabling the user to check the state of shaking by viewing the playback image of the desired portion.

[0281] In the embodiment, an example has been given in which the UI processing unit 103 performs processing to repeatedly play back an image section determined based on a key frame specified by a user operation on the timeline 250A (see steps S202, S230, S207, S250, S252, S253, S210, S270, etc. in Figures 33, 34 and 35). The user can easily understand the shaking situation by watching a certain section (scene) repeatedly. Also, if shaking effects are added, the user can easily judge whether the shaking effects are appropriate by watching them repeatedly. Therefore, repeat playback is an ideal way to present information for the user to check the shaking.

[0282] In the embodiment, an example has been given in which the UI processing unit 103 presents operators (preset buttons 210, 211) for selecting preset data of a vibration waveform, and performs processing for assigning the selected preset data to the timeline 250A (see FIGS. 31 and 32). This allows the user to perform operations such as assigning any preset data to any section on the timeline 250A. This is an intuitive operation using preset data, and allows the user to visually confirm on the timeline the section to which the preset data is to be applied.

[0283] In the embodiment, the UI processing unit 103 performs processing to present the state of the shaking that changes according to the assignment of preset data to the timeline 250A. That is, as shown in Figures 32A and 32B, in the section in the shaking timeline 252 to which the preset data is assigned, the display of the shaking waveform is made to be in a different state, and the display of a rabbit or a horse is placed on the timeline 250A to indicate that the shaking is added (see steps S234 and S235 in Figure 34). This allows the user to confirm that by assigning preset data on the timeline 250A, the state of shaking changes in that section. In Figures 32A and 32B, it is clearly indicated that the original waveform will be invalidated and that the oscillation will resemble that of a rabbit or a horse. However, it is also possible to display an oscillation waveform that resembles that of a rabbit or a horse, for example.

[0284] In the embodiment, an example has been described in which the UI processing unit 103 reflects an operation for changing shaking in a section set based on a key frame (see steps S202, S230, S231, S204, S220, and S221 in FIGS. 34 and 35). This allows the user to specify a key frame, so that the section to which the slider operation or the like is applied can be appropriately set, and the user can perform swing change operations in that section. This improves operability. There are various ways to set the intervals according to the key frames, as in the above example.

[0285] In the embodiment, an example has been given in which the UI processing unit 103 performs processing to present the state before and after the shaking of an image caused by a shaking change operation on a timeline (see waveform monitor 270 in FIG. 33 and steps S203 and S240 in FIG. 34). This allows the user to confirm how the state of sway changes in response to the sway change operation, making it easier for the user to understand the operational status of the sway change. In addition, the vibration waveforms before and after the vibration change may be displayed side by side at the same time.

[0286] In the embodiment, an example has been given in which the jitter change unit 101 performs jitter change processing to add jitter to input image data that is not jittered or to input image data that has been subjected to jitter reduction processing (image data PD or image data PD(NS)) (see Figures 7, 8, and 9). For example, the shake correction is performed on an image captured by a fixed camera and originally free of camera shake or an image from which camera shake or the like has been removed by the imaging device. That is, with this configuration, processing can be performed to intentionally add shaking to an image that has no shaking or only a small amount of shaking, thereby widening the range of image expression.

[0287] In the embodiment, an example has been given in which the wobble modification unit 101 performs wobble modification processing on input image data (image data PD(S)) that includes a wobble component (see FIGS. 10 and 11). For example, an image in which camera shake or the like has not been removed is input, and shake modification is performed on this image. In this case, the shaking can be removed or reduced, or the shaking can be increased or added as desired. The shaking change process can remove the shaking that is originally in the input image data, or increase the shaking. It can also add new shaking after removing the shaking that is originally in the input image data. This can improve the quality of the image and expand the range of expression.

[0288] In the embodiment, the UI processing unit 103 performs processing for storing the shaking information in response to a save operation by the user (see step S215 in FIG. 30 and FIG. 34). This allows information such as changes in shaking caused by operations arbitrarily performed by the user to be saved, and the information can be reflected in subsequent playback, used for further editing, or used to remove shaking.

[0289] The program of the embodiment is a program that causes, for example, a CPU, a DSP, or a device including these to execute the above-mentioned user interface processing in the processing explained with reference to FIG. 14 and FIG. That is, the program of the embodiment causes the information processing device to execute a swing change process (ST30) for obtaining swing-changed image data by changing the swing state of input image data constituting a moving image.Furthermore, the program causes the information processing device to execute a UI process for presenting an operator related to swing change and acquiring operation information by the operator, and a parameter setting process for setting a swing change process parameter PRM based on the operation information acquired by the UI process, that is, the process of Fig. 29 and the processes of Fig. 24 and Fig. 35.

[0290] By using such a program, the above-mentioned image processing devices TDx and TDy can be realized in devices such as the mobile terminal 2, the personal computer 3, or the imaging device 1.

[0291] A program for implementing such image processing devices TDx and TDy can be recorded in advance in a HDD serving as a recording medium built into a device such as a computer device, or in a ROM or the like within a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magnet optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0292] In addition, such a program is suitable for providing the image processing devices TDx and TDy of the embodiments in a wide range of applications. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the personal computer or the like can function as the image processing device of the present disclosure.

[0293] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0294] The present technology can also be configured as follows. (1) a swing change unit that performs swing change processing to obtain swing-changed image data by changing a swing state of input image data constituting a moving image; A user interface processing unit that acquires operation information related to a swing change; a parameter setting unit that sets parameters for the swing change processing based on the operation information acquired by the user interface processing unit; Image processing device. (2) the user interface processing unit causes a control for adjusting an amount of shaking to be presented; The parameter setting section sets a parameter for adjusting an amount of shaking based on operation information of the operator. The image processing device according to (1) above. (3) the user interface processing unit causes a control for adjusting a swing amount with respect to one or more swing direction components to be presented; The parameter setting unit sets a parameter for adjusting the amount of shaking of the shaking direction component based on operation information of the amount of shaking of the shaking direction component. The image processing device according to (1) or (2) above. (4) The sway direction component includes at least one of a yaw direction sway component, a roll direction sway component, and a pitch direction sway component. The image processing device according to (3) above. (5) the user interface processing unit causes a control for adjusting the amount of vibration for one or more vibration frequency bands to be presented; The parameter setting unit sets a parameter for adjusting the amount of fluctuation of the frequency band based on operation information of the amount of fluctuation of the frequency band. An image processing device according to any one of (1) to (4) above. (6) the user interface processor causes a control to be presented for adjusting the amount of vibration of at least a low frequency band and a high frequency band as the frequency band; The parameter setting unit sets a parameter for adjusting the amount of vibration in the low frequency band and a parameter for adjusting the amount of vibration in the high frequency band. The image processing device according to (5) above. (7) The user interface processing unit enables a selection operation of preset data of a vibration waveform, The parameter setting unit sets a parameter for adjusting image shaking in response to a selection operation of preset data. An image processing device according to any one of (1) to (6) above. (8) The user interface processing unit executes presentation of a timeline showing a shaking component in a time axis direction of an image. An image processing device according to any one of (1) to (7) above. (9) The user interface processing unit: Allowing the user to specify key frames on the timeline The image processing device according to (8) above. (10) The user interface processing unit: A key frame can be specified on the timeline by a user operation, A process of playing back images based on a key frame designated by a user operation on the timeline is performed. The image processing device according to (8) or (9) above. (11) The user interface processing unit: A process of repeatedly playing back an image section determined based on a key frame designated by a user operation on the timeline The image processing device according to (10) above. (12) The user interface processing unit: A process of assigning preset data selected by an operation to the timeline is performed. An image processing device according to any one of (8) to (11) above. (13) The user interface processing unit: A process of presenting a state of shaking that changes according to an assignment of preset data to the timeline is performed. The image processing device according to (12) above. (14) The user interface processing unit enables a user to specify a key frame on the timeline through a user operation, The operation to change the swing is reflected in the section set based on the key frame. The image processing device according to claim 8. An image processing device according to any one of (8) to (13) above. (15) The user interface processing unit: A process is performed to display the state before and after the image shake is changed according to the shake change operation. An image processing device according to any one of (1) to (14) above. (16) The sway change unit is A shaking modification process is performed to add shaking to input image data that is not subjected to shaking or to input image data that has been subjected to shaking reduction processing. An image processing device according to any one of (1) to (15) above. (17) The sway change unit is Perform shake modification processing on input image data that includes a shake component An image processing device according to any one of (1) to (16) above. (18) The user interface processing unit: A process for storing the vibration information is performed in response to a user's operation for saving the vibration information. An image processing device according to any one of (1) to (17) above. (19) The image processing device A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; An image processing method. (20) A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; A program for causing an information processing device to execute the above. [Explanation of symbols]

[0295] 1 imaging device, 2 mobile terminal, 3 personal computer, 4 server, 5 recording medium, 11 lens system, 12 imaging element section, 13 camera signal processing section, 14 recording control section, 15 display section, 16 output section, 17 operation section, 18 camera control section, 19 memory section, 22 driver section, 23 sensor section, 41 low pass filter, 42 mid pass filter, 43 high pass filter, 44, 45, 46 gain calculation section, 47, 57, 61 synthesis section, 51 yaw component extraction section, 52 pitch component extraction section, 53 roll component extraction section, 54, 55, 56 gain calculation section, 58, 59, 60 direction-specific processing section, 70 information processing device, 71 CPU, 72 ROM, 73 RAM, 74 bus, 75 input / output interface, 76 input section, 77 display section, 78 audio output section, 79 storage section, 80 Communication unit, 81 removable recording medium, 82 drive, 101 shaking change unit, 101a shaking removal unit, 101b shaking addition unit, 102 parameter setting unit, 103 UI processing unit, 104 correction cancellation unit, 110 image before lens distortion correction, 111 image after lens distortion correction, 120 exposure timing, 131 coordinate plane, 132 area, 200 shaking width adjusted image, 210 preset button, 211 preset button, 250, 250A timeline, 251 image timeline, 252 shaking timeline, 253 sound timeline, 254, 256 preset data display, 257 crossfade display, 260 save window, 261 confirmation window, 262 OK button, 263 cancel button, 270 waveform monitor, 271 playback display unit, 272 playback repeat period designation button, 273 Repeat period length, 274 playback operation section, 279 switching button, 281 key frame display, 282 key frame operation device, 283 vibration waveform, VS image source, TDx image processing device, TDy image processing device

Claims

1. a swing change unit that performs swing change processing to obtain swing-changed image data by changing a swing state of input image data constituting a moving image; A user interface processing unit that acquires operation information related to a swing change; a parameter setting unit that sets a parameter of the swing change process based on the operation information acquired by the user interface processing unit, the user interface processing unit causes a control for adjusting the amount of vibration for one or a plurality of vibration frequency bands to be presented; The parameter setting unit sets a parameter for adjusting the amount of fluctuation of the frequency band based on operation information of the amount of fluctuation of the frequency band. Image processing device.

2. the user interface processor causes a control to be presented for adjusting the amount of vibration of at least a low frequency band and a high frequency band as the frequency band; The parameter setting unit sets a parameter for adjusting the amount of vibration in the low frequency band and a parameter for adjusting the amount of vibration in the high frequency band. The image processing device according to claim 1 .

3. the user interface processing unit causes a control for adjusting a swing amount with respect to one or a plurality of swing direction components to be presented; The parameter setting unit sets a parameter for adjusting the amount of shaking of the shaking direction component based on operation information of the amount of shaking of the shaking direction component.

3. The image processing device according to claim 1 or 2.

4. The sway direction component includes at least one of a yaw direction sway component, a roll direction sway component, and a pitch direction sway component. The image processing device according to claim 3 .

5. A swing change unit that performs swing change processing to obtain swing-changed image data by changing the swing state of input image data that constitutes a moving image; A user interface processing unit that acquires operation information related to a swing change; a parameter setting unit that sets a parameter of the swing change process based on the operation information acquired by the user interface processing unit, The user interface processing unit enables a selection operation of preset data of a vibration waveform, The parameter setting unit sets a parameter for adjusting image shaking in response to a selection operation of preset data. Image processing device.

6. The user interface processing unit executes presentation of a timeline showing a shaking component in a time axis direction of an image.

6. The image processing device according to claim 1.

7. The user interface processing unit: Allowing the user to specify key frames on the timeline The image processing device according to claim 6.

8. The user interface processing unit: A key frame can be specified on the timeline by a user operation, A process of playing back images based on a key frame designated by a user operation on the timeline is performed.

8. The image processing device according to claim 6 or 7.

9. The user interface processing unit: A process of repeatedly playing back an image section determined based on a key frame designated by a user operation on the timeline The image processing device according to claim 8.

10. A swing change unit that performs swing change processing to obtain swing-changed image data by changing a swing state of input image data that constitutes a moving image; A user interface processing unit that acquires operation information related to a swing change; a parameter setting unit that sets a parameter of the swing change process based on the operation information acquired by the user interface processing unit, The user interface processing unit: It allows the selection of preset data for vibration waveforms, A timeline showing the shaking components in the time axis direction of the image is displayed. performing a process of assigning the selected preset data to the timeline; The parameter setting unit is Set parameters to adjust image shaking according to the preset data assigned to the timeline Image processing device.

11. The user interface processing unit: A process of presenting a state of shaking that changes according to an assignment of preset data to the timeline is performed. The image processing device according to claim 10.

12. The user interface processing unit enables a user to specify a key frame on the timeline through a user operation, The operation to change the swing is reflected in the section set based on the key frame. The image processing device according to any one of claims 6 to 11.

13. The user interface processing unit: A process is performed to display the state before and after the image shake is changed according to the shake change operation. The image processing device according to any one of claims 1 to 12.

14. The sway change unit is A shaking modification process is performed to add shaking to input image data that is not subjected to shaking or to input image data that has been subjected to shaking reduction processing. The image processing device according to any one of claims 1 to 13.

15. The sway change unit is Perform shake modification processing on input image data that includes a shake component The image processing device according to any one of claims 1 to 13.

16. The user interface processing unit: A process for storing the vibration information is performed in response to a user's operation for saving the vibration information. The image processing device according to any one of claims 1 to 15.

17. The image processing device A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; Do the following: In the user interface processing, an operator for adjusting the amount of vibration for one or more vibration frequency bands is presented, In the parameter setting process, a parameter for adjusting the amount of fluctuation of the frequency band is set based on operation information of the amount of fluctuation of the frequency band. Image processing methods.

18. An image processing device comprising: A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; Do the following: In the user interface processing, a selection operation of preset data of vibration waveforms is enabled, In the parameter setting process, a parameter for adjusting image shaking is set in response to a selection operation of preset data. Image processing methods.

19. An information processing device comprising: A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; Run the command, In the user interface processing, an operator for adjusting the amount of vibration for one or more vibration frequency bands is presented, In the parameter setting process, a parameter for adjusting the amount of fluctuation of the frequency band is set based on operation information of the amount of fluctuation of the frequency band. program.

20. An information processing device comprising: A vibration modification process for obtaining vibration-modified image data by modifying a vibration state of input image data constituting a moving image; A user interface process for acquiring operation information related to a swing change; a parameter setting process for setting parameters for the swing change process based on operation information acquired in the user interface process; Run the command, In the user interface processing, a selection operation of preset data of vibration waveforms is enabled, In the parameter setting process, a parameter for adjusting image shaking is set in response to a selection operation of preset data. program.

Citation Information

Patent Citations

  • Camera, photographed image selection / recording method and program

    JP2006180471A

  • Image display method, image pick-up method, and image synthesizing method

    JP2006203463A

  • Image processing apparatus and method, and computer program

    JP2011097512A

  • Image processing program, image processing apparatus, and image processing method

    JP2014153778A

  • Imaging device, control method of the same, and information processing system

    JP2015216510A