Image processing device, image processing method and program

Abstract

To solve a problem in which a processing time for vibration reduction processing is longer.SOLUTION: An imaging device includes image blur amount calculation means (205) that calculates a first blur amount per frame on the basis of the amount of blur between consecutive images captured by the imaging device that are separated by a predetermined number of frames, inertial blur amount calculation means (202) that calculates a second blur amount per frame on the basis of inertial information for a plurality of frames detected in response to shaking of the imaging device, and blur amount synthesis means (206) that calculates a third blur amount used to stabilize the consecutive images captured by the imaging device by subtracting the difference obtained by subtracting the first blur amount from the second blur amount from the inertial information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a video shake correction function. [Background technology]

[0002] Many anti-shake techniques have been proposed for images captured by an imaging device. For example, there is an anti-shake technique that performs anti-shake processing on images using the amount of shake calculated based on inertial information representing the amount of rotation of the imaging device obtained by a gyro sensor. There is also an anti-shake technique that performs anti-shake processing on images using the amount of shake calculated based on images captured by the imaging device.

[0003] Anti-shake technology using gyro sensors has the advantages of being highly robust and requiring little calculation, but also has the disadvantage that the inertial information contains an offset component specific to the gyro sensor, which does not contain the translational component of the imaging device.Anti-shake technology using captured video also has the advantage of including not only the rotational component of the imaging device but also the translational component, but also has the disadvantage of lacking robustness and including time-consuming processing such as processing the feature points of the image for each frame to calculate the amount of shake.

[0004] As a method to compensate for the shortcomings of both methods, the technology proposed in Patent Document 1 is proposed. In Patent Document 1, accuracy is improved by correcting using not only the amount of shake based on inertial information but also the amount of shake based on an image captured by an imaging device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-205551 Summary of the Invention [Problem to be solved by the invention]

[0006] In the above-mentioned Patent Document 1, the amount of blur based on inertial information and the amount of blur of an image based on an imaging device are combined on a frame-by-frame basis, which means that the amount of blur based on the captured image needs to be calculated for each frame, which poses a problem of lengthening the processing time for image stabilization. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present disclosure is characterized by comprising: an image blur amount calculation means for calculating a first blur amount per frame based on the amount of blur between consecutive images captured by an imaging device that are separated by a predetermined number of frames; an inertial blur amount calculation means for calculating a second blur amount per frame based on inertial information for a plurality of frames detected in response to shaking of the imaging device; and a blur amount synthesis means for calculating a third blur amount used to stabilize the consecutive images captured by the imaging device by subtracting the difference obtained by subtracting the first blur amount from the second blur amount from the inertial information. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to maintain accuracy and increase speed in image stabilization processing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a physical configuration of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing the logical configuration of the image processing device in FIG. 1. [Figure 3] 1 is a graph outlining the present disclosure. [Figure 4] FIG. 10 is a diagram in which both the amount of gyro shake and the amount of image shake are calculated for each frame. [Figure 5] This is a diagram in which the amount of gyro shake is calculated for each frame, and the amount of image shake is calculated for every five frames. [Figure 6] 6 is a conceptual diagram illustrating the calculation of the difference between the amount of gyro shake every five frames and the amount of image shake in FIG. 5. [Figure 7]FIG. 7 is a conceptual diagram illustrating calculation of a composite blur amount for every other frame based on the difference in FIG. 6. [Figure 8] FIG. 10 is a diagram illustrating inertial information included in a video file. [Figure 9] FIG. 2 is a diagram illustrating a GUI (Graphical User Interface) displayed on the display of FIG. [Figure 10] 10 is a flowchart illustrating an example of operation of the first embodiment. [Figure 11] FIG. 11 is a diagram showing a first shake amount in S1006 of FIG. [Figure 12] FIG. 11 is a diagram illustrating a second amount of shaking in step S1007 of FIG. [Figure 13] FIG. 11 is a diagram schematically illustrating the difference in S1008 of FIG. [Figure 14] FIG. 11 is a diagram illustrating a third shake amount in step S1009 of FIG. [Figure 15] FIG. 10 is a diagram illustrating the relationship between the amount of blur and an image. [Figure 16] FIG. 10 is a block diagram showing a logical configuration of a second embodiment. [Figure 17] 10 is a schematic diagram showing a case in which the frame interval of the image blur amount is changed between a low frequency part and a high frequency part in the second embodiment. FIG. [Figure 18] 10A and 10B are diagrams illustrating the analysis results of inertia information in the second embodiment. [Figure 19] 10 is a graph for explaining an outline of Example 2. [Figure 20] 10 is a flowchart illustrating an example of operation of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present disclosure, and not all combinations of features described in the following embodiments are necessarily essential to the solutions of the present disclosure. Note that the same reference numerals are used to designate the same components.

[0011] [Example 1] <Summary> Many anti-shake techniques for images captured by an imaging device have been proposed. Among these, there is a technique that performs anti-shake processing on images using the amount of shake calculated based on inertial information acquired by a gyro sensor. Anti-shake techniques using a gyro sensor have the advantages of being highly robust and requiring little computational effort, but have the disadvantage that the inertial information contains an offset component specific to the gyro sensor, but does not include the translational component of the imaging device. Furthermore, among many anti-shake techniques, there is a technique that performs anti-shake processing on images using the amount of shake calculated based on images captured by an imaging device. Anti-shake techniques using captured images have the advantage of including not only the rotational component but also the translational component of the imaging device, but also have the disadvantage of lacking robustness and involving time-consuming processing, such as processing feature points of the image for each frame. Therefore, in the present disclosure, the amount of shake based on images captured by an imaging device is used at intervals of several frames, and for each frame during the several frames, the amount of shake calculated based on the inertial information acquired by a gyro sensor is used. By performing such an operation, the present disclosure can remove the offset component of the inertial information while maintaining the accuracy of the amount of blur in the image, and can also achieve high-speed image stabilization processing. The present disclosure will be described in detail below.

[0012] Example 1 <Configuration of image processing device> <Physical configuration> 1 is a diagram illustrating the physical configuration of the present disclosure. The image processing device 100 includes a CPU 101, a RAM 102, a ROM 103, a SATA (Serial ATA) I / F (Interface) 104, a VC (Video Card) 105, and a general-purpose I / F 106. The CPU 101, RAM 102, ROM 103, SATA I / F 104, VC 105, and general-purpose I / F 106 are connected to each other via a system bus 107.

[0013] The CPU 101 uses the RAM 102 as a work memory and executes an operating system (OS) and various programs stored in the ROM 103, the external storage device 111, etc. Note that the OS and various programs may be stored in an internal storage device. The CPU 101 also controls each component via a system bus 107. Note that the processing according to the flowcharts described below is performed by the CPU 101 after program code stored in the ROM 103, the external storage device 111, etc. is loaded into the RAM 102. The external storage device 111 is connected to the SATA I / F 104 via a serial bus 108. The external storage device 111 is an HDD (Hard Disc Drive) or an SSD (Solid State Drive). The VC 105 is connected to a display 112 via a serial bus 109. The general-purpose I / F 106 is connected to an input device 113, such as a mouse or a keyboard, via a serial bus 110. The CPU 101 displays a GUI (Graphical User Interface) provided by a program on the display 112 and receives input information representing user instructions obtained via the input device 113. The image processing device 100 is realized, for example, by a desktop PC (Personal Computer). Alternatively, it may be realized by a notebook PC or tablet PC integrated with the display 112. The external storage device 111 may also be realized by a medium (recording medium) and an external storage drive for accessing the medium. The medium may be a flexible disk (FD), CD-ROM, DVD, USB memory, MO, flash memory, or the like. For example, it is assumed that a video file 1000 (described with reference to FIG. 8) of video captured by an imaging device is stored in a flash memory such as an SD card. FIG. 8 is a diagram illustrating inertial information included in the video file 1000. FIG. 8(a) is a diagram illustrating an example of the logical configuration of the video file 1000 recorded by the imaging device. The video file 1000 includes a header 1001 , metadata 1002 , and image data 1003 .The logical area of ​​the header 1001 stores information such as the number of frames in the video file 1000 and the compression format. Each logical area of ​​the metadata 1002 and image data 1003 is configured for each frame. For simplicity's sake, the example in FIG. 8( a) shows the metadata 1002 and image data 1003 for frames 0 to 2. In practice, the logical areas for the metadata 1002 and the logical areas for the image data 1003 are alternately allocated, corresponding to the number of frames stored in the logical area of ​​the header 1001. FIG. 8( b) shows an example of the logical configuration of the metadata 1002 included in the video file 1000. The logical area of ​​the metadata 1002 is allocated with a logical area for inertial information 1004 and a logical area for other metadata 1005. The logical area of ​​the inertial information 1004 stores information indicating pan, tilt, and roll for each frame. The gyro sensor detects the amount of rotation as angular velocity at a frequency ranging from several hundred Hz to several kHz. In the example of FIG. 8(b), a use case is assumed in which the angular velocity obtained by the gyro sensor is 600 Hz and the frame rate of the video file 1000 is 30 fps. Therefore, as shown in FIG. 8(b), 600÷30=20 pan, tilt, and roll values ​​are included in the inertial information 1004. Note that the logical area for the other metadata 1005 stores imaging information such as the exposure and focal length of the imaging device. The other metadata 1005 is not used in this embodiment. Therefore, a detailed description of the other metadata 1005 is omitted. Assuming that such a video file 1000 is stored in a flash memory, the image processing device 100 can acquire the video file 1000 via the flash memory. The image processing device 100 can acquire inertial information, etc., by referencing the acquired video file 1000.

[0014] <Logical configuration> Fig. 2 is a block diagram showing the logical configuration of the image processing device 100 in Fig. 1. Fig. 2(a) is a diagram showing an example of a logical configuration that functions when the CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory. Note that not all of the processes shown below necessarily need to be executed by the CPU 101, and the image processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.

[0015] The image processing device 100 includes a logical configuration of an inertial information acquisition unit 201, an inertial shake amount calculation unit 202, an interval control unit 203, an image acquisition unit 204, an image blur amount calculation unit 205, a shake amount synthesis unit 206, and an image stabilization processing unit 207. The inertial information acquisition unit 201 acquires inertial information for each time from a video file. The inertial information is generated by a gyro sensor incorporated in the imaging device and recorded in the video file 1000 as described above with reference to FIG. 8. Note that in addition to the gyro sensor, an acceleration sensor, a Global Navigation Satellite System (GNSS), and the like may also be combined. In this way, information other than inertial information including the three elements of pan, tilt, and roll may be recorded as supplementary information. FIG. 2(b) is a diagram showing an overview of the three elements of pan, tilt, and roll. As shown in Figure 2(b), in the mutually orthogonal XYZ axes, the rotational angular velocity of the X axis (roll axis) is called roll, the rotational angular velocity of the Y axis (tilt axis) is called tilt, and the rotational angular velocity of the Z axis (pan axis) is called pan. The rotational angular velocity of pan, tilt, and roll is detected by a gyro sensor built into the imaging device.

[0016] The inertial shake amount calculation unit 202 converts the inertial information into the gyro shake amount V gyro The data obtained from the gyro sensor as inertial information is expressed in angular velocity [degree / second], which indicates the amount of rotation per second. Therefore, by dividing the data obtained from the gyro sensor by the frame rate and converting it into units of [degree / frame], the inertial information is converted into the amount of gyro shake V gyroThe data obtained from the gyro sensor contains three elements: pan, tilt, and roll, so the amount of gyro shake V gyro can be expressed as a vector including three elements: pan, tilt, and roll. gyro For example, the second shake amount V2 is calculated based on the gyro shake amounts V of pan, tilt, and roll for a predetermined number of frames (k) from frame i+1 to frame i+k. gyro By this averaging, a second shake amount V2 including the three elements of pan, tilt, and roll is calculated as the average rotation amount over k frames. The averaging calculation is, for example, an arithmetic mean, but is not limited to this. For example, a weighted mean, a geometric mean, or a harmonic mean may be used.

[0017] The interval control unit 203 determines the frame interval of the images selected by the image blur amount calculation unit 205. The frame interval is longer than the interval between frames captured by the imaging device. For example, a predetermined number of frames (k) is set as the frame interval. The image acquisition unit 204 acquires an image of frame i and an image of frame i+k. The image blur amount calculation unit 205 calculates the image blur amount V from the image of frame i and the image of frame i+k. img Calculate the amount of image blur V img Specifically, the image blur amount calculation unit 205 calculates a motion vector from the image of frame i and the image of frame i+k, and calculates the image blur amount V based on the calculated motion vector. img More specifically, the amount of image blur V is calculated by optimizing the amount of rotation in the rotation process so that the squared error between the coordinates obtained by performing three-dimensional rotation on the coordinates of the feature points detected in the image of frame i and the coordinates of the feature points in frame i+k is minimized. img is calculated. img As mentioned above, the blur amount between images V img Therefore, in the following, the image blur amount V img The blur amount between images is V img It is also called.

[0018] The shake amount synthesis unit 206 synthesizes the first shake amount V1 and the second shake amount V2 to calculate a third shake amount V3. For example, the gyro shake amounts V from frame i+1 to frame i+k are gyro The third blur amount V3 from frame i+1 to frame i+k is V3=V gyro -D, where D is the difference between the first blur amount V1 and the second blur amount V2, and is calculated by subtracting the second blur amount V2 from the first blur amount V1. D is also the average value over k frames.

[0019] The stabilization processor 207 performs stabilization processing on the video based on the third blur amount V3. The third blur amount V3 indicates a rotation vector from the previous frame. Therefore, by rotating the current frame in the opposite direction, a current frame that is not rotated (not blurred) relative to the previous frame can be obtained.

[0020] <Outline of the Disclosure> FIG. 3 is a graph outlining the present disclosure. The horizontal axis of FIG. 3 is set to frames. On the other hand, the vertical axis of FIG. 3 first represents the value of the gyro sensor in angular velocity [degree / second], and the amount of shake used in image stabilization processing is expressed by converting the value of the gyro sensor into the amount of angle change per frame [degree / frame]. However, in FIG. 3, in order to clearly show that the difference with the amount of image blur increases as the drift component included in the amount of gyro shake accumulates, the vertical axis of FIG. 3 is set to the direction [degree] obtained by integrating the amount of angle change per frame. In the graphs described below, if the vertical axis is [degree], the graph takes into account the accumulation of drift components, and if the vertical axis is [degree / frame], the graph takes into account the amount of shake according to the present disclosure. FIG. 3 shows the integral value of the first shake amount V1, the integral value of the second shake amount V2, the integral value of the third shake amount V3, the integral value of the gyro shake amount V gyro The integral value is shown.

[0021] Gyro shake amount V gyroThe integral value is indicated by the points marked with ◇ and the lines connecting the points marked with ◇. For example, point 801 indicates the gyro shake amount V obtained from the gyro sensor mounted on the imaging device. gyro is the value obtained by integrating from the 0th frame to each frame. gyro contains an offset component. Therefore, the amount of gyro vibration V gyro If we continue to integrate, the offset component will accumulate. Therefore, the amount of gyro vibration V gyro The integral value gradually increases as the frames pass. In other words, due to a slight error in the gyro sensor, the amount of gyro shake V gyro A drift occurs in which the integral value increases.

[0022] The integral value of the first blur amount V1 is indicated by a square point and a thick line connecting the square points. For example, point 802 indicates the image blur amount V calculated from the motion vector of the image between frames in the video captured by the imaging device. img The first blur amount V1 converted into a value per frame is integrated from the 0th frame to each frame. img The calculation of the image blur amount V is performed every few frames, not every frame. img is calculated every five frames. Therefore, the integrated value of the first blur amount V1 exists only every five frames. Therefore, within the section indicated by the thick line connecting the square points, it is not possible to determine how the imaging device is moving from the first blur amount V1 alone.

[0023] The integral value of the second shake amount V2 is indicated by a circle and a line connecting the circle points. For example, point 803 is an integral value of the gyro shake amount V2 at the same interval as the first shake amount V1. gyro 3, the interval between the integral values ​​of the second shake amount V2 is 5 frames. Therefore, the points 803 are arranged at 5 frame intervals.

[0024] The integrated value of the third shake amount V3 is indicated by triangle points and the lines connecting the triangle points. For example, point 804 is the integrated value of the third shake amount V3. The integrated value of the third shake amount V3 will be described using FIGS. 4 to 7 and 9. FIG. 4 is a diagram showing both the amount of gyro shake and the amount of image blur calculated for each frame. The amount of gyro shake is not affected by the subject. Therefore, image stabilization processing using the amount of gyro shake is highly robust. Furthermore, the amount of gyro shake can be obtained simply by converting the unit of inertial information, so the amount of calculation is small. This enables high-speed processing. However, the amount of gyro shake includes an offset component generated by the temperature characteristics of the gyro sensor. Therefore, unless the offset component is removed, the integrated value of the gyro shake amount will experience a phenomenon called drift, in which integral errors gradually accumulate, as shown in FIG. 4. Therefore, using the integrated value of the gyro shake amount as is will not result in accurate image stabilization processing. Furthermore, since inertial information only contains rotational shake components of the imaging device, such as pan, tilt, and roll, it does not contain translational components in the pan, tilt, and roll axes. Therefore, the accuracy of stabilization processing using only inertial information decreases when the imaging device is subject to translational shake. On the other hand, the amount of image blur is calculated by calculating the amount of movement between frames from the video. Therefore, the amount of image blur includes not only the rotational component of the imaging device but also the translational component. Therefore, the accuracy of stabilization processing using image blur amount does not decrease even when the imaging device is subject to translational shake. However, if a moving object is present in the subject, changes in the image due to the movement of the moving object may be mistaken for movement of the imaging device, and the amount of image blur may not be calculated correctly. Therefore, stabilization processing using image blur amount lacks robustness when a moving object is present in the subject. Furthermore, the process of calculating the amount of image blur from the video includes time-consuming processes such as feature point detection, feature point matching, and optimization. Therefore, the process of calculating the amount of image blur from the video is slower than the process of calculating the amount of gyroscopic shake from inertial information. Therefore, first, the amount of image blur is calculated every predetermined number of frames. Fig. 5 shows the amount of gyro blur calculated every frame and the amount of image blur calculated every five frames.As shown in Fig. 5, for example, the amount of image blur between the image of frame 20 and the image of frame 25 is calculated. In this way, when the predetermined number of frames is five, the amount of image blur between images separated by a frame interval of five frames is calculated. Here, an example of setting the frame interval will be described with reference to Fig. 9.

[0025] FIG. 9 is a diagram illustrating a GUI displayed on the display 112 of FIG. 1. FIG. 9(a) is a diagram illustrating an example of a GUI displayed on the display 112 of FIG. 1. An application 1100 is displayed as a GUI. In the application 1100, an application name "video stabilization app" is displayed on the GUI. On the GUI of the application 1100, a file load button 1101, a preview area 1102, a stabilization process start button 1103, and a shooting situation selection option 1104 are arranged as GUI elements. The file load button 1101, the stabilization process start button 1103, and the shooting situation selection option 1104 are GUI elements intended to be selected by the user using the input device 113 of FIG. 1.

[0026] For example, when the user selects the file load button 1101, a GUI for specifying a video file is displayed. Since GUIs for specifying files are also common in PCs, smartphones, and the like, a description thereof will be omitted. When a video file is specified, the contents of the specified video file are displayed in a preview area 1102. When the user selects the shake correction process start button 1103, the operation of the image processing device 100 is initiated, and a video file that has undergone shake correction processing is generated. The shooting condition options 1104 display multiple items for inputting instructions regarding the status of the imaging device at the time of capturing the video captured by the imaging device. When an item is selected by the user via the input device 113 in FIG. 1, the circle (single circle) to the left of the item changes to a double circle (double circle), and the previous circle changes to a circle. These items are associated with a value of the frame interval k. Examples of frame interval k corresponding to the items in the shooting condition options 1104 are shown in the table in FIG. 9(b). FIG. 9(b) is a diagram showing an example of the relationship between the items in the shooting condition options 1104 and the frame interval k. As the situation becomes more dynamic, k is set to a smaller value. Furthermore, when the user selects the image stabilization process start button 1103 in FIG. 9(a) to start image stabilization process, a value corresponding to the item selected in the shooting condition options 1104 is input to the interval control unit 203 as the frame interval k.

[0027] Next, the difference between the amount of gyro shake and the amount of image blur is calculated every five frames. Fig. 6 is a conceptual diagram of the calculation of the difference between the amount of gyro shake every five frames and the amount of image blur in Fig. 5. Conceptually, the amount of image blur is subtracted from the amount of gyro shake every five frames, but as described above with reference to Fig. 2, in this embodiment, it is actually done as follows. That is, a process is performed in which a first amount of blur converted from the amount of image blur per frame is subtracted from a second amount of blur calculated by averaging the amount of gyro shake for a plurality of frames.

[0028] Next, the composite blur amount is calculated from the gyro shake amount based on the difference. FIG. 7 is a conceptual diagram of the calculation of the composite blur amount for every other frame based on the difference in FIG. 6. As shown in FIG. 7, the difference every five frames is converted to a difference every other frame, and the converted difference is subtracted from the gyro shake amount for every other frame to calculate the composite blur amount. Return to FIG. 3. In short, first, the frame interval of the second blur amount integrated value corresponding to the gyro shake amount and the frame interval of the first blur amount integrated value corresponding to the image blur amount are matched. Then, the third blur amount integrated value at point 804 (the triangle point and the line connecting it) is calculated based on the integrated value of the gyro shake amount for the intervening one-frame interval. This allows the blur amount to be calculated for every frame even if the calculation of the image blur amount is omitted every few frames, thereby speeding up the image stabilization process. <Example of operation> FIG. 10 is a flowchart illustrating an example of the operation of the first embodiment. The processing illustrated in FIG. 10 is implemented by the CPU 101 executing a program stored in the RAM 102 or the ROM 103. The processing illustrated in FIG. 10 is executed when the image stabilization processing start button 1103 in FIG. 9(a) is operated. Note that some or all of the functions of the steps in FIG. 10 may be implemented by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. Furthermore, the processing illustrated in FIG. 10 can also be implemented as a cloud computing configuration in which one function is shared among multiple resources via the Internet and processed collaboratively, as long as it realizes various functions of the image processing device 100. Note that in the following description, the processing of S1001 to S1012 in FIG. 10 will be described assuming that they are executed sequentially, but the order is not particularly limited to this. For example, since it is sufficient that the first shake amount V1 and the second shake amount V2 have been calculated by the time of the processing of S1008, the processing of S1006 may be executed after the processing of S1007.

[0029] In S1001, the CPU 101 acquires an image. In S1002, the CPU 101 acquires inertial information. It is assumed that the inertial information is generated by a gyro sensor or the like incorporated in the imaging device and recorded in a video file 1000 as shown in FIG. 8. In S1002, the CPU 101 acquires all of the inertial information 1004 for each frame recorded in the video file 1000.

[0030] In step S1003, the CPU 101 converts the inertial information into the gyro shake amount V gyro Specifically, for each frame, the pan, tilt, and roll, each expressed in angular velocity [degree / second], are divided by the frame rate to obtain the gyro shake amount V converted into units of [degree / frame]. gyro For example, in Figure 3, the slope of the thin line connecting the points 801 is the amount of gyro vibration V gyro The method for calculating the amount of gyro shake for one frame is not limited to this. Other known methods may be used. gyro FIG. 15 is a diagram illustrating the relationship between the amount of shake and an image. As shown in FIG. 15, the amount of gyro shake V gyro [i+1] is the difference between the angle of the image capture device when frame i+1 was captured and the angle of the image capture device when frame i was captured.

[0031] In S1004, the CPU 101 determines the frame interval k of the image. As described above with reference to Figures 9(a) and 9(b), in this embodiment, k is determined by the user's selection.

[0032] In S1005, CPU 101 sets frame index i to the index of the first frame. As a result, frame i references the first frame, and the image of frame i references the image of the first frame. Therefore, the image of frame i+k references the image of the kth frame from the first frame.

[0033] In S1006, the CPU 101 calculates the image blur amount V from the image of frame i and the image of frame i+k. img Calculate the amount of image blur V img is calculated based on the motion vectors calculated from the two images. Specifically, the coordinates of the feature points detected in the image of frame i are subjected to a three-dimensional rotation process, and the amount of rotation that minimizes the squared error between the rotated coordinates and the coordinates of the feature points in frame i+k is calculated by an optimization process. The amount of rotation calculated in this way is used as the image blur amount V img The method for calculating the rotation amount from the motion vector is not limited to the above, and other known methods may be used. img Figure 15 shows the relationship when k=5.

[0034] Here, the image blur amount V in FIG. img is calculated from the image of frame i and the image of frame i+k. img is the rotation amount for k frames. Image blur amount V img is expressed as a vector with three elements: pan (yaw), tilt (pitch), and roll. img The pan, tilt, and roll in step S1004 are each divided by k to obtain the amount of shake per frame, which is defined as the first shake amount V1. FIG. 11 is a diagram schematically illustrating the first shake amount V1 in step S1006 in FIG. 10. As shown in FIG. 11, the first shake amount V1 is converted into a value for every five frames. Therefore, the first shake amount V1 indicates the average amount of rotation over k frames. For example, in FIG. 3, the slope of the thick line connecting the squares at points 802 is the first shake amount V1. Note that while the graph in FIG. 3 shows only one of the pan, tilt, and roll elements, there are actually three similar graphs for pan, tilt, and roll. However, since the processing for each element is the same, only one element will be used for the explanation.

[0035] In S1007, the CPU 101 calculates the gyro shake amount V from frame i+1 to frame i+k. gyroare averaged to calculate the second shake amount V2. Data obtained from the gyro sensor is also expressed as a vector having three elements: pan, tilt, and roll. Therefore, the second shake amount V2 is also expressed as a vector having three elements. The second shake amount V2 indicates the average amount of rotation over k frames. FIG. 12 is a diagram schematically showing the second shake amount V2 in S1007 of FIG. 10. As shown in FIG. 12, the second shake amount V2 is calculated by averaging the gyro shake amount V gyro This is the average of 5 frames.

[0036] In S1008, the CPU 101 calculates the difference D between the first shake amount V1 and the second shake amount V2 by subtracting the first shake amount V1 from the second shake amount V2. The difference D is the average value over k frames of the difference between the second shake amount V2 calculated using inertial information and the first shake amount V1 calculated using video. FIG. 13 is a diagram schematically showing the difference in S1008 of FIG. 10. As shown in FIG. 13, the difference D is calculated every five frames, and the first shake amount V1, which is the image blur amount converted into one frame, is multiplied by the gyro shake amount V gyro The second blur amount V2 is calculated by subtracting the second blur amount V1 from the average of the first five frames.

[0037] In S1009, the CPU 101 calculates the gyro shake amount V from frame i+1 to frame i+k. gyro The third blur amount V3 from frame i+1 to frame i+k is V3=V gyro -D. Figure 14 is a diagram schematically showing the third blur amount V3 of S1009 in Figure 10. Comparing the image blur amount calculated every other frame in Figure 11 with the third blur amount V3 in Figure 14, the third blur amount V3 is approximately the same value as the image blur amount calculated every frame assuming k is 1 (the image blur amount calculated every other frame in Figure 11). In other words, by using the third blur amount V3, similar image stabilization processing can be performed without having to calculate the image blur amount every frame, which takes time.

[0038] In S1010, CPU 101 advances frame index i by k frames. In S1011, CPU 101 performs loop processing until all frames have been scanned. If i exceeds N, which is the final frame of the video, the processing ends. If it does not exceed N, the processing returns to S1006 and repeats. In S1012, CPU 101 performs image stabilization processing using third shake amount V3. Since third shake amount V3 indicates a rotation vector from the previous frame, rotating the current frame in the opposite direction can obtain a current frame that is not rotated (unblurred) relative to the previous frame. Methods for performing image stabilization processing using third shake amount V3 are well known, so detailed description will be omitted.

[0039] As described above, according to this embodiment, the inertial information acquired from the gyro sensor and the amount of blur calculated at predetermined intervals from successive images captured by the imaging device are combined, thereby maintaining the accuracy of the amount of blur in the image, eliminating the offset component of the inertial information, and realizing faster image stabilization processing.

[0040] [Example 2] In the first embodiment, a method for calculating the amount of blur at a fixed frame interval k corresponding to the shooting conditions selected by the user was described. However, in reality, it is difficult to uniquely predict the frequency of blur from the shooting conditions specified in the instruction input. If the blur is of low frequency relative to the set frame interval k, the interval for calculating the image blur may become excessively narrow, and performing multiple calculations of the amount of blur may result in unnecessary processing. Furthermore, if the blur is of high frequency relative to the set frame interval k, the amount of image blur may be aliased if the interval for calculating the image blur amount is too long, and translational blur may not be accurately represented.

[0041] Therefore, in this embodiment, a method will be described in which the frame interval k is changed to a more appropriate value depending on the situation by analyzing inertial information obtained from a gyro sensor, thereby appropriately controlling the calculation interval for the amount of image blur, thereby achieving higher accuracy and faster processing.

[0042] In reality, analyzing inertial information that does not include translational blur components makes it impossible to determine the frequency of translational blur. However, it is rare for rotational and translational blur to have different frequencies. Specifically, to rotate an imaging device without translating it, it must be rotated around a point called the entrance pupil (nodal point) of the imaging device's lens, but it is rare for the center of rotation and the entrance pupil to perfectly coincide. Therefore, the rotational period and translational period of the imaging device's movement are almost always the same. Therefore, the frequency of rotational blur and the frequency of translational blur are considered to be the same. Therefore, by analyzing inertial information obtained from a gyro sensor, it is possible to predict the frequency of translational blur in image blur.

[0043] <Logical configuration> FIG. 16 is a block diagram showing the logical configuration of the second embodiment. The image processing device 100 in this embodiment is composed of an inertial information acquisition unit 201, an inertial blur amount calculation unit 202, an interval control unit 403, an image acquisition unit 204, an image blur amount calculation unit 205, a blur amount synthesis unit 206, an image stabilization processing unit 207, and an inertial information analysis unit 408. The inertial information acquisition unit 201, the inertial blur amount calculation unit 202, the image acquisition unit 204, the image blur amount calculation unit 205, and the blur amount synthesis unit 206 in FIG. 15 are the same as those in the first embodiment, so their description will be omitted. The inertial information analysis unit 1608 acquires inertial information from the inertial information acquisition unit 201 and analyzes the vibration component of the inertial information. The interval control unit 1603 sets an appropriate frame interval k depending on the situation based on the vibration component. FIG. 17 is a schematic diagram showing how the frame interval k of the image blur amount is changed between the low-frequency and high-frequency components in the second embodiment. As shown in FIG. 17, the frame interval k for calculating the amount of image blur is increased in the low-frequency range and decreased in the high-frequency range. This reduces the frequency of time-consuming image blur calculations in the low-frequency range, thereby enabling further speedup. The change in frame interval k is determined based on the analysis results of the inertial information. FIG. 18 is a diagram illustrating the analysis results of the inertial information in Example 2. In FIG. 18, if F is the sampling frequency, F / 2 is the Nyquist frequency. Therefore, frequencies higher than F / 2 are ignored. The threshold t indicates the criterion for noticeable blur. In FIG. 18, the highest frequency of frequencies f [F / frame] that is equal to or less than F / 2 and whose intensity [degree] exceeds the threshold t is frequency 3. Therefore, the period p = F ÷ f = 32 ÷ 3 = 10.7 [frames]. Here, the half period k' = p ÷ 2 = 10.7 ÷ 2 = 5.3 [frames]. The half period k' is calculated here so that one rising or falling edge of the blur change can be reproduced. If k' (=5.3) frames or more have passed since frame i, for which the amount of blur was calculated the previous time, the half period k' is advanced to set the frame interval k = 6, and the amount of image blur between frame i and frame i+k is calculated. Note that threshold value t may be a function of sampling frequency F. For example, threshold value t may be determined by g and F, with g being a counter that scans F. FIG. 19 is a graph outlining the second embodiment.Point 801 in Figure 19 is similar to point 801 in Figure 3, so its description will be omitted. Point 902, represented by a square dot and a thick line connecting them, is the integrated value of the amount of image blur calculated from the motion vector of the image. The frame interval k was determined by analyzing the integrated value of the amount of gyro shake at point 801. As a result, no high-frequency blur was observed in section 905, so frame interval k was set to 10. As a result, point 902, which represents the integrated value of the amount of image blur, is calculated at intervals of 10 frames in section 905, which enables faster image stabilization processing than processing at a fixed frame interval. On the other hand, high-frequency blur was observed in section 906, so frame interval k was set to 3. As a result, point 904, which represents the third integrated value of the amount of blur in section 906, can be calculated with appropriate precision.

[0044] <Example of operation> FIG. 20 is a flowchart illustrating an example of operation of the second embodiment. The processing shown in FIG. 20 is realized by the CPU 101 executing a program stored in the RAM 102 or the ROM 103. The processing shown in FIG. 20 is executed when the image stabilization processing start button 1103 in FIG. 9(a) receives an operation. Note that some or all of the functions of the steps in FIG. 20 may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of each process indicates the step in the flowchart. Furthermore, the processing shown in FIG. 20 can also be realized as a cloud computing configuration in which one function is shared among multiple resources and processed collaboratively via the Internet, as long as it realizes various functions of the image processing device 100.

[0045] The processes of S1001, S1002, and S1003 are the same as those in the first embodiment, and therefore a detailed description thereof will be omitted. In S2001, the CPU 101 sets the frame index j to the index of the first frame. In S2002, the CPU 101 analyzes the inertia information in the section from frame j to F frames to generate analysis information. First, the gyro shake amount V gyrois numerically integrated to calculate the integral value. The integral value is subjected to FFT (Fast Fourier Transform) analysis in the section from frame j to frame F to generate frequency components. These frequency components are used as analysis information. For example, the frequency components are represented by the graph in FIG. 18 mentioned above.

[0046] In S2003, CPU 101 calculates frame interval k in the section from frame j to frame F. For the FFT results, the half period k' = F÷f÷2 is calculated for the highest frequency f whose intensity exceeds threshold t [degree], which indicates the standard for noticeable blur. The unit of frequency f is [F / frame]. The unit of half period k' is [frame]. The unit of intensity is [degree]. Furthermore, when searching for the highest frequency f whose intensity exceeds threshold t, the portion exceeding F / 2, the Nyquist frequency, is not scanned. For example, using the graph in FIG. 18, when threshold t = 0.5, the intensity of frequency 3 [F / frame] is the highest frequency below F / 2 that exceeds threshold t, so f is 3. The half period k' for this f is F÷f÷2 = 5.3. Here, threshold t is empirically set to t = 0.5 [degree]. Furthermore, because the degree of noticeable blur varies depending on the frequency, t may be a function of frequency. For example, it can be determined as t = 0.1 ÷ g × F / 2 (where g is a counter that scans the frequency). The half period k' is rounded up to the frame interval k.

[0047] In S2004, the CPU 101 sets the frame index i to j. This index i is the same as that used in the first embodiment. S1006, S1007, S1008, S1009, and S1010 are the same processes as in the first embodiment, and therefore a description of these processes will be omitted. In S2005, the CPU 101 determines whether the third blur amount has been calculated for all frames from frame k to j+F, which defines the frame interval k. If i is less than j+F, scanning of all frames has not yet been completed, and the process returns to S1006. If i is greater than or equal to j+F, scanning of all frames has been completed, and the process proceeds to S2006. In S2006, the CPU 101 sets the current i to j. Since F is not necessarily a multiple of k, i in S2005 does not necessarily match j+F. In that case, i may exceed j+F, and the third blur amount may be calculated beyond the period of F frames for which analysis information was generated. For this reason, in S2006, ik may be set to j, and processing may proceed to the next F frame interval after rolling back. In S2007, CPU 101 determines whether j has finished scanning all frames. If j is less than the total number of frames N, the process returns to S2002, and if j is N or greater, the process proceeds to S1012. S1012 is the same as in the first embodiment, and performs image stabilization processing on the image based on the third blur amount, and then ends processing.

[0048] By determining the frame interval k for calculating the amount of blur based on image information from the analysis results of inertial information as described above, when the imaging device is vibrating at a low frequency, the image blur calculation interval can be increased, thereby speeding up the image stabilization process. On the other hand, when the imaging device is vibrating at a high frequency, the image blur calculation interval can be set to a short interval, allowing the amount of blur, including translational blur components, to be accurately calculated, thereby maintaining the accuracy of the image stabilization process. For example, as shown in FIG. 19 , the frame interval k was determined by analyzing the integral value of the gyroscopic blur amount at point 801. Since no high-frequency blur was observed in section 905, the frame interval k was set to 10. As a result, the image blur amount integral value 902 (□) was calculated at intervals of 10 frames in section 905, thereby speeding up the image stabilization process compared to processing at a fixed frame interval. On the other hand, since high-frequency blur was observed in section 906, the frame interval k was set to 3. As a result, the blended blur amount integral value 904 (△) in section 906 was calculated with appropriate accuracy.

[0049] <Effects of the Invention> As described above, according to the present disclosure, the inertial information acquired from the gyro sensor and the amount of blur calculated at predetermined intervals from successive images captured by the imaging device are combined, thereby maintaining the accuracy of the amount of blur in the images, eliminating the offset component of the inertial information, and realizing faster image stabilization processing.

[0050] [Other embodiments] Although various examples and embodiments of the present disclosure have been shown and described above, the spirit and scope of the present disclosure are not limited to the specific descriptions in this specification. The present disclosure is not limited to the above-described embodiments, and various modifications may be made. In addition, the present disclosure may be realized by appropriately combining parts of the above-described embodiments.

[0051] In addition, in the present embodiment, an example has been described in which the image processing device 100 acquires the video file 1000 via a flash memory, but this is not limiting. For example, the video file 1000 may be acquired wirelessly. In this case, the image processing device 100 only needs to have the functionality of a wireless communication unit (not shown).

[0052] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The program may also be provided by recording it on a computer-readable storage medium.

[0053] The disclosure of the present embodiment includes configurations typified by the following image processing device, image processing method, and program.

[0054] <Configuration 1> an image blur amount calculation means for calculating a first blur amount per frame based on the amount of blur between successive images captured by the imaging device that are separated by a predetermined number of frames; an inertial blur amount calculation means for calculating the second blur amount per frame based on inertial information for a plurality of frames detected in response to shaking of the imaging device; a blur amount combining means for calculating a third blur amount used for image stabilization of successive images captured by the imaging device by subtracting a difference obtained by subtracting the first blur amount from the second blur amount from the inertial information; An image processing device comprising:

[0055] <Configuration 2> 2. The image processing device according to claim 1, wherein the inertial shake amount calculation means calculates the second shake amount per frame by averaging gyro shake amounts obtained by converting the inertial information for the plurality of frames into a value per frame.

[0056] <Configuration 3> 3. The image processing device according to configuration 2, wherein the inertial information is calculated for each frame based on the amount of inertial rotation about each of three axes orthogonal to one another.

[0057] <Configuration 4> The three axes are the pan axis, the tilt axis, and the roll axis, 4. The image processing device according to configuration 3, wherein the amount of inertial rotation is expressed by the angular velocity of each of the pan axis, the tilt axis, and the roll axis.

[0058] <Configuration 5> 5. The image processing device according to configuration 4, wherein the inertial shake amount calculation means converts the inertial shake amount into the gyroscopic shake amount based on the angular velocities and the frame rate of the imaging device.

[0059] <Configuration 6> The image processing device according to configuration 1, wherein the amount of blur between the images is an amount of image rotation that minimizes an error between rotated coordinates obtained by performing a three-dimensional rotation process on coordinates of feature points detected in one of the images separated by the predetermined number of frames, and coordinates of feature points detected in the other of the images separated by the predetermined number of frames.

[0060] <Configuration 7> 7. The image processing device according to configuration 6, wherein the image rotation amount is the rotation amount about each of three mutually orthogonal axes, and is expressed by the vectors of the pan axis, tilt axis, and roll axis.

[0061] <Configuration 8> 4. The image processing device according to configuration 3, wherein the predetermined number of frames is set to be smaller as the amount of inertial rotation increases.

[0062] <Configuration 9> 2. The image processing device according to configuration 1, further comprising analysis means for determining the predetermined number of frames based on analysis information obtained by analyzing the inertial information.

[0063] <Configuration 10> 10. The image processing device according to configuration 9, wherein the analysis means uses frequency components generated from the inertial information as the analysis information.

[0064] <Configuration 11> The image processing device according to configuration 10, further comprising an interval control means for selecting the frequency with the highest intensity from among the frequencies included in the frequency components and having intensities exceeding a preset threshold value, and setting the predetermined number of frames to a natural number equal to or greater than half a period of the period corresponding to the selected frequency.

[0065] <Configuration 12> 12. The image processing device according to claim 11, wherein the interval control means extracts frequencies having an intensity exceeding the preset threshold from frequencies equal to or lower than half of the frequencies included in the frequency components.

[0066] <Configuration 13> 11. The image processing device according to configuration 10, further comprising an interval control means for controlling to increase the predetermined number of frames as the frequency components become lower, and for controlling to decrease the predetermined number of frames as the frequency components become higher.

[0067] <Configuration 14> an image blur amount calculation step of calculating a first blur amount per frame based on the amount of blur between images separated by a predetermined number of frames among successive images captured by the imaging device; an inertial blur amount calculation step of calculating the second blur amount per frame based on inertial information for a plurality of frames detected in response to shaking of the imaging device; a blur amount combining step of calculating a third blur amount used for image stabilization of successive images captured by the imaging device by subtracting a difference obtained by subtracting the first blur amount from the second blur amount from the inertial information; An image processing method comprising:

[0068] <Configuration 15> 15. A program for causing a computer to execute each step of the image processing method according to claim 14. [Explanation of symbols]

[0069] 100 Image processing device 101 CPU 202 Inertial shake calculation unit 205 Image blur amount calculation unit 206 Shake amount synthesis section

Claims

1. an image blur amount calculation means for calculating a first blur amount per frame based on the amount of blur between successive images captured by the imaging device that are separated by a predetermined number of frames; an inertial blur amount calculation means for calculating the second blur amount per frame based on inertial information for a plurality of frames detected in response to shaking of the imaging device; a blur amount combining means for calculating a third blur amount used for image stabilization of successive images captured by the imaging device by subtracting a difference obtained by subtracting the first blur amount from the second blur amount from the inertial information; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the inertial blur amount calculation means calculates the second blur amount per frame by averaging gyro blur amounts obtained by converting the inertial information for the plurality of frames into a value per frame.

3. 3. The image processing apparatus according to claim 2, wherein the inertial information is calculated for each frame based on the amount of inertial rotation about each of three axes orthogonal to one another.

4. The three axes are a pan axis, a tilt axis, and a roll axis, 4. The image processing device according to claim 3, wherein the amount of inertial rotation is expressed by the angular velocity of each of the pan axis, the tilt axis, and the roll axis.

5. 5. The image processing device according to claim 4, wherein the inertial shake amount calculation means converts the inertial shake amount into the gyroscopic shake amount based on the angular velocities and the frame rate of the imaging device.

6. 2. The image processing device according to claim 1, wherein the amount of blur between the images is an amount of image rotation that minimizes an error between rotated coordinates obtained by performing a three-dimensional rotation process on coordinates of feature points detected in one of the images separated by the predetermined number of frames, and coordinates of feature points detected in the other of the images separated by the predetermined number of frames.

7. 7. The image processing device according to claim 6, wherein the image rotation amount is a rotation amount about three mutually orthogonal axes, and is expressed by a vector of each of a pan axis, a tilt axis, and a roll axis.

8. 4. The image processing device according to claim 3, wherein the predetermined number of frames is set to be smaller as the amount of inertial rotation increases.

9. 2. The image processing apparatus according to claim 1, further comprising: an analysis unit that determines the predetermined number of frames based on analysis information obtained by analyzing the inertial information.

10. 10. The image processing apparatus according to claim 9, wherein the analyzing means uses frequency components generated from the inertial information as the analysis information.

11. The image processing device according to claim 10, further comprising an interval control means for selecting the frequency with the highest intensity from among the frequencies included in the frequency components and having an intensity exceeding a predetermined threshold, and setting the predetermined number of frames to a natural number equal to or greater than half a period of the period corresponding to the selected frequency.

12. 12. The image processing device according to claim 11, wherein the interval control means extracts frequencies having intensities exceeding the preset threshold from frequencies equal to or lower than half of the frequencies included in the frequency components.

13. 11. The image processing device according to claim 10, further comprising an interval control means for controlling to increase the predetermined number of frames as the frequency component becomes lower and for controlling to decrease the predetermined number of frames as the frequency component becomes higher.

14. an image blur amount calculation step of calculating a first blur amount per frame based on the amount of blur between consecutive images captured by the imaging device that are separated by a predetermined number of frames; an inertial blur amount calculation step of calculating the second blur amount per frame based on inertial information for a plurality of frames detected in response to shaking of the imaging device; a blur amount combining step of calculating a third blur amount used for image stabilization of successive images captured by the imaging device by subtracting a difference obtained by subtracting the first blur amount from the second blur amount from the inertial information; An image processing method comprising:

15. A program for causing a computer to execute each step of the image processing method according to claim 14.

Images