Image correction device, image correction method, and image correction program

Abstract

Provided are an image correction device, an image correction method, and an image correction program that make it possible to suppress occurrence of image blur while reducing calculation cost. The image correction device includes an angle calculation unit, a drift correction unit, a pixel movement amount calculation unit, and an image correction unit, and moves pixels constituting a captured image captured by an imaging device to thereby correct the captured image. The angle calculation unit calculates the angle of the imaging device on the basis of an angular velocity detected by a gyro sensor. The drift correction unit removes a drift component of the gyro sensor from the calculated angle to thereby correct the angle. The pixel movement amount calculation unit calculates the movement amount of pixels for each line of the captured image corresponding to a line of imaging elements included in the imaging device on the basis of the corrected angle. The image correction unit corrects the captured image according to the movement amount of the pixels calculated for each line of the captured image.

Description

Image correction device, image correction method, and image correction program

[0001] The present invention relates to an image correction device, an image correction method, and an image correction program.

[0002] For example, when capturing images of the entire operating room using an imaging device (surgical field camera) installed on the ceiling of an operating room, vibrations (regular vibrations with a small swing angle) caused by a motor in an air conditioner or the like may be transmitted to the imaging device, causing the imaging device to vibrate. In this case, the vibration components are carried over into the captured image, causing the captured image to blur.

[0003] Patent Document 1 discloses a technology for effectively correcting camera shake and lens distortion. In the correction method described in Patent Document 1, a Kalman filter performs signal processing to remove error components due to camera shake and gyro drift based on the acceleration and angular velocity of a camera module output from a motion sensor.

[0004] However, executing signal processing using a Kalman filter based on the acceleration and angular velocity of the camera module increases the calculation cost, and therefore the technology disclosed in Patent Document 1 cannot be directly applied to the problem of image blurring in captured images.

[0005] International Publication No. 2017 / 014071

[0006] An object of the present invention is to provide an image correction device, an image correction method, and an image correction program that are capable of reducing calculation costs and suppressing the occurrence of image blurring.

[0007] According to the present invention, there is provided an image correction device having the following configuration: [1] An image correction device that corrects a captured image captured by an imaging device by moving pixels that constitute the image, the image correction device comprising an angle calculation unit, a drift correction unit, a pixel movement amount calculation unit, and an image correction unit, wherein the angle calculation unit calculates an angle of the imaging device based on an angular velocity of the imaging device detected by a gyro sensor, the drift correction unit corrects the angle by removing a drift component of the gyro sensor from the calculated angle, the pixel movement amount calculation unit calculates an amount of movement of the pixels for each line of the captured image that corresponds to a line of an imaging element of the imaging device based on the corrected angle, and the image correction unit corrects the captured image in accordance with the amount of movement of the pixels calculated for each line of the captured image.

[0008] According to the present invention, the angle of the imaging device calculated based on the angular velocity detected by the gyro sensor is corrected by removing the drift component of the gyro sensor from the angle, the amount of pixel movement is calculated for each line of the captured image based on the corrected angle, and the captured image is corrected according to the calculated amount of pixel movement. As a result, it is possible to correct the captured image without using acceleration and without the need to perform signal processing using a Kalman filter, which increases calculation costs, and as a result, it is possible to reduce calculation costs and suppress the occurrence of image blur.

[0009] Various embodiments of the present invention are described below. The embodiments described below can be combined with each other. [2] The image correction device according to [1], further comprising a bias removal unit, which removes a bias component included in the angular velocity detected by the gyro sensor. [3] The image correction device according to [1] or [2], further comprising an angular velocity conversion unit, which converts the angular velocity detected by the gyro sensor so that a three-dimensional coordinate system coincides between the imaging device and the gyro sensor. [4] The image correction device according to any of [1] to [3], further comprising an angle change amount calculation unit, which calculates an angle change amount of the imaging device based on the corrected angle, and the pixel movement amount calculation unit calculates a movement amount of the pixel for each line of the captured image based on the calculated angle change amount. [5] The image correction device according to any one of [1] to [4], wherein the imaging device vibrates at a vibration frequency of 5 Hz or more and a vibration angle of 0.15° or less. [6] An image correction method for correcting an image captured by an imaging device by moving pixels constituting the image, the image correction method comprising: an angle calculation step, a drift correction step, a pixel movement amount calculation step, and an image correction step, wherein the angle calculation step calculates an angle of the imaging device based on an angular velocity of the imaging device detected by a gyro sensor; the drift correction step corrects the angle by removing a drift component of the gyro sensor from the calculated angle; the pixel movement amount calculation step calculates a pixel movement amount for each line of the captured image corresponding to a line of an imaging element of the imaging device based on the corrected angle; and the image correction step corrects the captured image for each line of the captured image according to the calculated pixel movement amount. [7] An image correction program that causes a processor to execute the image correction method according to [6].

[0010] According to the present invention, there is no need to perform signal processing using a Kalman filter, which increases calculation costs, and it is possible to reduce calculation costs and suppress the occurrence of image blurring.

[0011] 1 is a block diagram showing an example of the configuration of an image processing system 1. FIG. 2A is a diagram showing an example of a captured image, and FIG. 2B is a diagram showing an example of a corrected captured image. It is a diagram explaining each line of a captured image 30 corresponding to each line of an image sensor 12 of an imaging device 10. It is a block diagram showing an example of the functional configuration of a calculation unit 23 (image processing device 20). It is a diagram showing an example of coordinate transformation for matching three-dimensional coordinate systems between an imaging head of an imaging device 10 and a gyro sensor 11. It is an example of a graph showing an error between an angle corrected by a drift correction unit 23d and an actual angle of the imaging device 10, and an error between an angle calculated by a conventional technique and the actual angle of the imaging device 10. FIGS. 7A and 7B are diagrams explaining various parameters used in the calculation process of an angle change amount calculation unit 23e and a pixel movement amount calculation unit 23f. It is an example of a graph showing an error in pixel movement amount Δy for each incident angle α. It is a diagram explaining a process for calculating a pixel movement amount in a line 50 of an Nth row of a captured image 30. It is a flowchart showing an example of an image correction process performed by an image processing device 20.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The features shown in the following embodiments can be combined with each other. Furthermore, each feature can be an invention independently.

[0013] FIG. 1 is a block diagram showing an example of the configuration of an image processing system 1 according to an embodiment of the present invention. As shown in FIG. 1, the image processing system 1 includes an imaging device 10 and an image processing device 20. The imaging device 10 and the image processing device 20 are connected via a communication cable such as a LAN (Local Area Network) cable. Note that the communication cable may be any cable that allows mutual data communication, and is not limited to the present embodiment. Note that the image processing device 20 functions as the "image correction device" of the present invention.

[0014] The imaging device 10 is, for example, a camera (surgical field camera) that captures an overall image of a surgical procedure being performed by a doctor, and is installed on a pillar, wall, or the like in an operating room. The imaging device 10 captures an image of a subject (e.g., a surgical procedure) and generates a moving image by capturing the image. The moving image includes multiple frames of captured images obtained by capturing the image. The imaging device 10 transmits the captured moving image to the image processing device 20 via a communication cable. In this embodiment, the imaging device 10 is a rolling shutter type (line exposure sequential readout type) imaging device that exposes pixels in an image sensor 12 (described later) in sequence, starting from the top row, and reads out the electrons converted into electric charges in sequence, starting from the top row.

[0015] 1, an imaging device 10 includes a gyro sensor 11 and an image sensor 12. The image sensor 12 functions as the "imaging element" of the present invention.

[0016] The gyro sensor 11 is a vibration sensor that detects vibrations of the imaging device 10, and detects angular velocities around the X-axis (roll axis), Y-axis (pitch axis), and Z-axis (yaw axis) as vibrations of the imaging device 10. The gyro sensor 11 then transmits data indicating the angular velocities around the X-axis, Y-axis, and Z-axis as angular velocity data to the image processing device 20 via a communication cable.

[0017] Vibrations of the imaging device 10 occur as follows. That is, for example, when an imaging device 10 installed on the ceiling of an operating room is used to capture images of the entire operating room, vibrations (regular vibrations with a small swing angle) caused by a motor in an air conditioning unit (air conditioner) or the like may be transmitted to the imaging device 10, causing the imaging device 10 to vibrate. In this case, vibration components are carried over into the images captured by the imaging device 10, causing the captured images to blur. The imaging device 10 may vibrate, for example, when the imaging device 10 vibrates at a vibration frequency of 5 Hz or more and a vibration angle of 0.15° or less. Note that this is not limited to cases where an imaging device 10 installed on the ceiling of an operating room captures images of the entire operating room. For example, when the imaging device 10 is suspended near an air conditioning unit for monitoring the engine room of a ship or the inside of a building, vibrations caused by a motor in the air conditioning unit or the like may be transmitted to the imaging device 10, causing the imaging device 10 to vibrate.

[0018] The image sensor 12 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures an image of an imaging range including a subject. When an imaging lens 13 (see FIG. 7 ) is attached to the main body of the imaging device 10, subject light representing the subject passes through the imaging lens 13 and is focused on the image sensor 12, and captured image data representing an image of the subject (captured image) is generated by the image sensor 12. The image sensor 12 then transmits the generated captured image data to the image processing device 20 via a communication cable.

[0019] The image processing device 20 displays a captured image based on the captured image data transmitted from the imaging device 10 (image sensor 12) on the display screen of the display unit 26. Furthermore, the image processing device 20 corrects the captured image by moving pixels that make up the captured image captured by the imaging device 10 (in other words, changing the arrangement of pixels) based on the angular velocity data and captured image data transmitted from the imaging device 10 (gyro sensor 11 and image sensor 12).

[0020] Fig. 2A is a diagram showing an example of a captured image, and Fig. 2B is a diagram showing an example of a corrected captured image. As shown in Fig. 2A, when an imaging device 10 employing a rolling shutter system in which the readout timing is different between the upper and lower parts of a pixel array vibrates, the captured image captured by the imaging device 10 contains vibration components, causing image blur (a diagonal distortion phenomenon) in the captured image. In the captured image shown in Fig. 2B, the captured image shown in Fig. 2A has been corrected by the image processing device 20, and the image blur that occurred in the captured image has been eliminated.

[0021] As shown in FIG. 1, the image processing device 20 includes a captured image acquisition unit 21, an angular velocity acquisition unit 22, a calculation unit 23, an image correction unit 24, a display control unit 25, a display unit 26, and a storage unit 27.

[0022] Each component of the image processing device 20 may be implemented by software or hardware. When implemented by software, various functions can be realized by a CPU executing a computer program. The program may be stored in an internal storage unit or a computer-readable non-transitory recording medium. Alternatively, the program may be read from an external storage unit and implemented using so-called cloud computing. When implemented by hardware, the functions can be realized by various circuits such as an ASIC, an FPGA, or a DRP (Dynamically Reconfigurable Processor). This embodiment deals with various pieces of information and concepts that encompass them. These are represented by high or low signal values ​​as a binary bit group consisting of 0s or 1s, and communication and calculations can be performed using the above software or hardware aspects.

[0023] The functional configuration of the image processing device 20 will be described below.

[0024] The captured image acquisition unit 21 receives and acquires captured image data transmitted from the image sensor 12 of the imaging device 10. Then, the captured image acquisition unit 21 outputs the acquired captured image data to the image correction unit 24.

[0025] The angular velocity acquisition unit 22 receives and acquires angular velocity data (data indicating the angular velocity around the X-axis, the Y-axis, and the Z-axis) transmitted from the gyro sensor 11 of the imaging device 10. Then, the angular velocity acquisition unit 22 outputs the acquired angular velocity data to the calculation unit 23.

[0026] Based on the angular velocity data output from the angular velocity acquisition unit 22, the calculation unit 23 calculates the amount of pixel movement for each line of the captured image corresponding to the lines of the image sensor 12 of the imaging device 10. The purpose of calculating the amount of pixel movement is to correct the captured image in the image correction unit 24 by moving the pixels that make up the captured image captured by the imaging device 10. The calculation unit 23 outputs movement amount data indicating the amount of pixel movement calculated for each line of the captured image to the image correction unit 24.

[0027] 3 is a diagram illustrating each line of a captured image 30 corresponding to each line of the image sensor 12 of the imaging device 10. As shown in FIG. 3, the captured image 30 has a plurality of lines (1st row to N+2th row, N: natural number) of the image sensor 12, each of which corresponds to a plurality of lines (1st row to N+2th row) of the image sensor 12 and is made up of a plurality of pixels in the row direction (left-right direction in the figure). In this embodiment, the calculation unit 23 calculates, based on the angular velocity data output from the angular velocity acquisition unit 22, the amount of movement of a plurality of pixels constituting each of the plurality of lines (1st row to N+2th row) of the captured image 30 corresponding to the lines of the image sensor 12 of the imaging device 10.

[0028] Fig. 4 is a block diagram showing an example of the functional configuration of the calculation unit 23. As shown in Fig. 4, the calculation unit 23 includes, as its functional configuration, a bias removal unit 23a, an angular velocity conversion unit 23b, an angle calculation unit 23c, a drift correction unit 23d, an angle change amount calculation unit 23e, and a pixel movement amount calculation unit 23f.

[0029] Based on the angular velocity data output from the angular velocity acquisition unit 22, the bias removal unit 23a removes bias components that are included in the angular velocity indicated by the angular velocity data (angular velocity detected by the gyro sensor 11) and that arise due to, for example, temperature conditions.

[0030] For example, the angular velocity around each axis (X-axis, Y-axis, and Z-axis) detected as the n-th detection result by the gyro sensor 11 is expressed as Gyro ｉｎ (n), the angular velocity Gyro as a result of removing the bias component from the angular velocity ｏｕｔ (n) can be expressed by the following equation (1): where p is the angular velocity Gyro used to calculate the average value of the angular velocity in the second term on the right side of equation (1). ｉｎ is the number of

[0031]

[0032] When the three-dimensional coordinate systems of the imaging device 10 (e.g., a left-handed coordinate system) and the gyro sensor 11 (e.g., a right-handed coordinate system) differ, the angular velocity conversion unit 23b converts the angular velocity indicated by the angular velocity data output from the angular velocity acquisition unit 22 (the angular velocity detected by the gyro sensor 11) so that the three-dimensional coordinate systems of the imaging device 10 and the gyro sensor 11 coincide with each other. FIG. 5 is a diagram showing an example of coordinate conversion that matches the three-dimensional coordinate systems of the imaging head of the imaging device 10, which is a left-handed coordinate system, and the gyro sensor 11, which is a right-handed coordinate system. As shown in FIG. 5, the coordinate conversion is performed by converting the coordinate systems between the imaging head of the imaging device 10 and the gyro sensor 11, and then aligning the coordinate axes. Note that if the coordinate systems of the imaging head of the imaging device 10 and the gyro sensor 11 coincide with each other, no coordinate system conversion is performed, and only the coordinate axes are aligned. Furthermore, the alignment of the coordinate axes is performed using a rotation matrix, but this is not necessary if the coordinate axes of the imaging head of the imaging device 10 and the gyro sensor 11 are aligned.

[0033] The angular velocity conversion unit 23b converts the angular velocity Gyro from which the bias component has been removed by the bias removal unit 23a. ｏｕｔ When the three-dimensional coordinate system (gyro sensor 11) is a right-handed coordinate system, the angular velocity Gyro ｏｕｔ The conversion formula in this case can be expressed by the following formula (2). ＸＬ , Gyro ＹＬ , Gyro ＺＬare the angular velocities around each axis (X-axis, Y-axis, and Z-axis) in the left-handed coordinate system, and Gyro ｏｕｔＸ , Gyro ｏｕｔＹ , Gyro ｏｕｔＺ are the angular velocities around each axis (X-axis, Y-axis, and Z-axis) in the right-handed coordinate system.

[0034]

[0035] Gyro after aligning the coordinate axes between the imaging head (left-handed coordinate system) of the imaging device 10 and the gyro sensor 11 (right-handed coordinate system) Ｒｏｌｌ , Gyro Ｐｉｔｃｈ , Gyro Ｙａｗ is the rotation angle (θ) around each axis (X-axis, Y-axis, and Z-axis) between the gyro sensor 11 and the imaging head (substrate). Ｘ , θ Ｙ , θ Ｚ ), it can be expressed by Equation (4) using the rotation matrix expressed by Equation (3) below.

[0036]

[0037]

[0038] The angle calculation unit 23c calculates the angle (posture angle) of the imaging device 10 based on the angular velocity obtained by converting the three-dimensional coordinate system by the angular velocity conversion unit 23b.

[0039] First, the angle calculation unit 23c calculates the angular velocity (angular velocity vector: Gyro Ｒｏｌｌ , Gyro Ｐｉｔｃｈ , Gyro Ｙａｗ ) is converted into a quaternion form that is easy to rotate in three-dimensional space. The converted quaternion q Ｇｙｒｏ can be expressed by the following equation (5).

[0040]

[0041] Next, the angle calculation unit 23c calculates the variation of the quaternion due to the angular velocity vector (instantaneous rotation amount). Specifically, the angle calculation unit 23c calculates the variation of the quaternion due to the angular velocity vector using a past quaternion (normalized) corresponding to the attitude of the image capture device 10. The past quaternion corresponding to the attitude of the image capture device 10 is denoted by q ｐｒｅｖ In this case, the variation of the quaternion q ｄ can be expressed by the following formula (6). Here, × in formula (6) represents the product of quaternions. For example, the quaternion q L and a quaternion q expressed by the following equation (8): R The product of the angle and the quaternion q corresponding to the orientation of the image capture device 10 can be expressed by the following equation (9). ｐｒｅｖ does not exist, the quaternion q ｐｒｅｖ Using the above, the variation of the quaternion q ｄ Calculate.

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] Next, the angle calculation unit 23c calculates the fluctuation q of the quaternion obtained by the formula (6). ｄ and the past quaternion q ｐｒｅｖ The current attitude (quaternion) q of the image capture device 10 is calculated by the following equation (11) using ｐｏｓｅ Here, data_interval is the time from when the previous detection result of the angular velocity by the gyro sensor 11 is obtained until when the current (present) detection result of the angular velocity by the gyro sensor 11 is obtained.

[0048]

[0049] Next, the angle calculation unit 23c calculates the orientation (quaternion) q of the imaging device 10 at the current time as expressed by the following equation (12) in order to convert it into Euler angles (to be described later) and calculate the orientation (quaternion) of the imaging device 10 at the next time. ｐｏｓｅ Normalize the normalized posture (quaternion) q ｃｕｒｒ is expressed by the following equation (13), where q is the past quaternion (normalized) corresponding to the orientation of the image capture device 10. ｐｒｅｖ Update as.

[0050]

[0051]

[0052] Here, norm(q ｐｏｓｅ ) is the orientation (quaternion) q expressed by the following equation (14): ｐｏｓｅ This shows the norm calculation of norm(q ｐｏｓｅ ) is the conjugate q expressed by the following formula (15): ＊ ｐｏｓｅ is expressed as a scalar value by the following equation (16).

[0053]

[0054]

[0055]

[0056] Next, the angle calculation unit 23c calculates the normalized attitude (quaternion) q ｃｕｒｒ is converted into Euler angles with a rotation order of Z axis → Y axis → X axis. The normalized orientation (quaternion) q ｃｕｒｒ is expressed by the following equation (17), the current Euler angle Angle is expressed by the following equation (18): ｃｕｒｒ，ｉｎ can be expressed by the following equations (19) to (21).

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] The drift correction unit 23d corrects the angle (Euler angle) of the imaging device 10 calculated (converted) by the angle calculation unit 23c by removing the drift component (continuous fluctuation (deviation)) of the gyro sensor 11 from the angle. The drift correction unit 23d corrects the angle of the imaging device 10 using a high-pass filter or a band-pass filter. In this embodiment, the drift correction unit 23d corrects the angle (Euler angle) of the imaging device 10 using a one-dimensional high-pass Butterworth filter (high-pass filter). The drift correction unit 23d corrects the angle of the imaging device 10 by setting the cutoff frequency of the high-pass filter or band-pass filter to a desired frequency. This can improve the accuracy of vibration frequencies of 5 Hz or higher, for example. The Euler angle including the drift component based on the previous angular velocity detection result by the gyro sensor 11 is converted into the Angle. ｐｒｅｖ，ｉｎ The corrected Euler angle based on the previous angular velocity detection result is set as Angle ｐｒｅｖ，ｒｅｓ The Euler angle including the drift component based on the current angular velocity detection result by the gyro sensor 11 is defined as Angle ｃｕｒｒ，ｉｎ If so, the corrected Euler angle Angle based on the current angular velocity detection result is ｃｕｒｒ，ｒｅｓ can be expressed by the following equation (22): Ｂ , C Ａ are the coefficients of a one-dimensional high-pass Butterworth filter.

[0063]

[0064] 6 is an example of a graph showing the error (represented by curve L1) between the angle (angle of imaging device 10) corrected by drift correction unit 23d and the actual angle of imaging device 10, and the error (represented by curve L2) between the angle (angle of imaging device 10) calculated by conventional technology (executing signal processing using a Kalman filter based on the acceleration and angular velocity of the camera module) and the actual angle of imaging device 10, when the vibration angle of imaging device 10 is fixed at 0.05 degrees and the vibration frequency is changed in 3 Hz increments in the range of 3 to 30 Hz. As shown in FIG. 6, in the range where the vibration frequency of imaging device 10 is 5 Hz or higher, the error between the angle (angle of imaging device 10) corrected by drift correction unit 23d and the actual angle of imaging device 10 is smaller than the error between the angle (angle of imaging device 10) calculated by conventional technology and the actual angle of imaging device 10. The drift correction unit 23d corrects the angle of the imaging device 10 by using a high-pass filter or a band-pass filter whose cutoff frequency is set to a desired value, thereby improving the accuracy of vibration frequencies of 5 Hz or higher.

[0065] The angle change amount calculation unit 23e calculates the angle change amount of the imaging device 10 (the vibration amount of the imaging device 10) based on the angle corrected by the drift correction unit 23d. FIGS. 7A and 7B are diagrams explaining various parameters used in the calculation process of the angle change amount calculation unit 23e and the pixel movement amount calculation unit 23f. In FIGS. 7A and 7B, d is the focal length from the center of the imaging lens 13 to the focal point. α is the angle of incidence when light passing through coordinate h on the Z axis of the image sensor 12 is incident on the focal point. β is the angle of incidence when light passing through coordinate w on the Y axis of the image sensor 12 is incident on the focal point. The distance between coordinate y and coordinate y-1 on the Z axis of the image sensor 12 is calculated based on the sensor size μ on the Z axis per pixel of the image sensor 12. ｙ The distance between the coordinate x and the coordinate x−1 on the Y axis of the image sensor 12 corresponds to the sensor size μ ｘ is equivalent to

[0066] Angle change amount of the imaging device 10 ｄｉｆｆis the angle corrected last time by the drift correction unit 23d. ｐｒｅｖ (see equation (23) below), and the angle corrected this time by the drift correction unit 23d is defined as Angle ｃｕｒｒ (see equation (24) below), it can be calculated by equation (25) below.

[0067]

[0068]

[0069]

[0070] The angle change amount calculation unit 23e uses the normalized angle change amount when calculating the pixel movement amount described below, and therefore calculates the normalized angle change amount Anglediff_normal (see equation (26) below) using equation (27) below.

[0071]

[0072]

[0073] Based on the normalized angle change calculated by the angle change calculation unit 23e, the pixel movement amount calculation unit 23f calculates the pixel movement amount for each line of the captured image corresponding to the line of the image sensor 12 (image pickup element) of the imaging device 10. If the number of vertical and horizontal pixels of the image sensor 12 is (Height, Width), the movement amount Δx(h, w) on the X axis of a pixel corresponding to coordinate h on the Z axis and coordinate w on the Y axis in the image sensor 12 can be expressed by the following equation (28). Also, the movement amount Δy(h, w) on the Y axis of a pixel corresponding to coordinate h on the Z axis and coordinate w on the Y axis in the image sensor 12 can be expressed by the following equation (29).

[0074]

[0075]

[0076] Here, the normalized angle change amount Angle calculated by the angle change amount calculation unit 23e is Ｙ and Angle ｚis a sufficiently small value (for example, less than 0.15°), the first term on the right-hand side of equations (28) and (29) can be omitted. In this embodiment, the captured image is corrected line by line (row by row), i.e., there is no need to correct the captured image column by column. Therefore, the incident angle β may be interpreted as 0, and the first and second terms on the left-hand side can be simplified as the first term on the right-hand side, as expressed by the following equation (30). In addition, the normalized angle change amount Angle Ｙ is assumed to be a sufficiently small value (for example, less than 0.15°), and it is experimentally determined that the error in the amount of pixel movement between the first and second terms on the left side of equation (31) and the first term on the right side of equation (31) below is less than one pixel, the first and second terms on the left side of equation (31) can be simplified as the first term on the right side.

[0077]

[0078]

[0079] Based on the above, the pixel movement amount calculation unit 23f calculates the movement amounts Δx and Δy at each pixel position using the following equations (32) and (33), based on the normalized angle change amount calculated by the angle change amount calculation unit 23e.

[0080]

[0081]

[0082] FIG. 8 shows the relationship between d / μ ｙ 8 is an example of a graph showing an error in pixel movement amount Δy for each incident angle α (see FIG. 7A) when the angle α is 3000 and the vertical angle of view is 35 degrees. In FIG. 8, a curve L1 indicates the normalized angle change amount Angle calculated by the angle change amount calculation unit 23e. Ｙ is 0.15 [deg], the error in the amount of movement of pixels between the first and second terms on the left side of the equation (31) and the first term on the right side. The curve L2 indicates the error in the amount of movement of pixels between the first and second terms on the left side of the equation (31) and the first term on the right side. Ｙis 0.10 [deg], the error in the amount of movement of pixels between the first and second terms on the left side of the equation (31) and the first term on the right side. The curve L3 indicates the error in the amount of movement of pixels between the first and second terms on the left side of the equation (31) and the first term on the right side. Ｙ is 0.05 [deg], this indicates the error in the amount of pixel movement between the first and second terms on the left side of the equation (31) and the first term on the right side. Ｙ is less than 0.15 [deg], it can be seen that the error in the amount of pixel movement between the first and second terms on the left side and the first term on the right side of equation (31) above is less than one pixel.

[0083] The movement amounts Δx and Δy at each pixel position calculated by the pixel movement amount calculation unit 23f are synchronized with the output timing (time) of the angular velocity detection result by the gyro sensor 11, but are not synchronized with each line (row 1 to row N+2, where N is a natural number) of the captured image 30 (see FIG. 9 ) corresponding to each line of the image sensor 12 of the imaging device 10. Therefore, the pixel movement amount calculation unit 23f uses time information such as a timestamp to calculate the pixel movement amount for each line of the captured image 30. For example, the pixel movement amount calculation unit 23f calculates the pixel movement amount on the target line for which the pixel movement amount is to be calculated based on timestamps for two pixels close to the target line (the output timing of the angular velocity detection result by the gyro sensor 11 corresponding to the pixels), the movement amounts of the two pixels, and a timestamp for the target line (the output timing of the angular velocity detection result by the gyro sensor 11 corresponding to the line). Then, the pixel movement amount calculation unit 23f outputs movement amount data indicating the calculated movement amount of pixels on each line to the image correction unit 24.

[0084] 9 is a diagram illustrating a process for calculating the amount of movement of pixels in the Nth line 50 of the captured image 30. First, the pixel movement amount calculation unit 23f calculates a timestamp for the line 50 (Nth line) for which the amount of movement of pixels is to be calculated.Ｎ can be calculated using the following equation (34) with the exposure timing of the image capture device 10 as a reference. In equation (34), FR is the frame rate of the image capture device 10, and ExpTime ｌｉｎｅ is the exposure time for one line of the image capturing device 10. ｓｔａｒｔ is the time from when the VD (vertical drive signal) is generated until the exposure of the first row starts, and ExpOffset ｅｎｄ is the time from when the next VD (vertical drive signal) is generated until the exposure of the (N+2)th row (last row) is completed.

[0085]

[0086] Next, the pixel movement amount calculation unit 23f calculates the time stamp Timestamp Ｎ The movement amount of the pixel in the target line 50 is calculated based on the timestamps (output timings of the angular velocity detection results by the gyro sensor 11 corresponding to the pixels 40 and 41, respectively) of the two pixels 40 and 41 (see FIG. 9) exposed before and after the target line 50 (Nth row) for which the movement amount of the pixel is to be calculated, and the movement amount calculated for the two pixels 40 and 41. The movement amount of the pixel 40 calculated based on the angular velocity detected by the gyro sensor 11 chronologically before the pixel 41 is calculated as (Δx ｂｅｆｏｒｅ , Δy ｂｅｆｏｒｅ ) and the timestamp for pixel 40 is Timestamp ｂｅｆｏｒｅ The movement amount of pixel 41 calculated based on the angular velocity detected by the gyro sensor 11 after pixel 40 in the time series is (Δx ａｆｔｅｒ , Δy ａｆｔｅｒ ), and the timestamp for pixel 41 is Timestamp ａｆｔｅｒ In this case, the pixel movement amount (Δx Ｎ , Δy Ｎ ) can be calculated by the following equations (35) and (36), respectively.

[0087]

[0088]

[0089] The image correction unit 24 corrects the captured image (see, for example, FIG. 2A ) indicated by the captured image data output from the captured image acquisition unit 21, in accordance with the movement amount (the movement amount of pixels for each line of the captured image 30) indicated by the movement amount data output from the pixel movement amount calculation unit 23f of the calculation unit 23. Then, the image correction unit 24 outputs corrected image data indicating the corrected captured image (see, for example, FIG. 2B ) to the display control unit 25.

[0090] The display control unit 25 outputs the corrected image data output from the image correction unit 24 to the display unit 26, and controls the display unit 26 to display the captured image indicated by the corrected image data on the display unit 26.

[0091] The display unit 26 is, for example, a liquid crystal display device having a display panel including a plurality of pixels arranged on a substrate, and under the control of the display control unit 25, displays the captured image represented by the corrected image data output from the display control unit 25.

[0092] The storage unit 27 is configured with, for example, a RAM (Random Access Memory) or a DRAM (Dynamic Random Access Memory), and is used as a work area when the image processing device 20 executes processes based on various programs.

[0093] A part of the storage unit 27 is, for example, a non-volatile memory such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the image processing device 20. The storage unit 27 can hold a database including one or more tables for recording various information, processing results, and the like.

[0094] The programs stored in the memory unit 27 include, for example, an OS (Operating System) for realizing the basic functions of the image processing device 20, drivers for controlling various hardware, programs for realizing various functions, etc., and include a program that functions as the ``image correction program'' of the present invention.

[0095] 10 is a flowchart showing an example of image correction processing (corresponding to the "image correction method" of the present invention) performed by the image processing device 20 in this embodiment. The image correction processing performed by the image processing device 20 is executed every time an image capturing process is performed by the image capturing device 10.

[0096] First, the captured image acquisition unit 21 receives and acquires captured image data transmitted from the image sensor 12 of the imaging device 10 (step S100). Then, the captured image acquisition unit 21 outputs the acquired captured image data to the image correction unit 24.

[0097] Next, the angular velocity acquisition unit 22 receives and acquires the angular velocity data (data indicating the angular velocity around the X-axis, the Y-axis, and the Z-axis) transmitted from the gyro sensor 11 of the imaging device 10 (step S110). Then, the angular velocity acquisition unit 22 outputs the acquired angular velocity data to the calculation unit 23. The angular velocity is acquired.

[0098] Next, the bias removal unit 23a of the calculation unit 23 removes the bias component contained in the angular velocity indicated by the angular velocity data (angular velocity detected by the gyro sensor 11) based on the angular velocity data output from the angular velocity acquisition unit 22 (step S120).

[0099] Next, the angular velocity conversion unit 23b of the calculation unit 23 converts the angular velocity from which the bias component has been removed in step S120 so that the three-dimensional coordinate systems between the imaging device 10 (e.g., a left-handed coordinate system) and the gyro sensor 11 (e.g., a right-handed coordinate system) coincide (step S130).

[0100] Next, the angle calculation unit 23c of the calculation unit 23 calculates the angle (posture angle) of the imaging device 10 based on the angular velocity obtained by converting the three-dimensional coordinate system in step S130 (step S140).

[0101] Next, the drift correction unit 23d of the calculation unit 23 corrects the angle of the imaging device 10 calculated in step S140 by removing the drift component of the gyro sensor 11 from the angle (step S150).

[0102] Next, the angle change amount calculation unit 23e of the calculation unit 23 calculates the angle change amount of the imaging device 10 (the vibration amount of the imaging device 10) based on the angle corrected in step S150 (step S160).

[0103] Next, the pixel movement amount calculation unit 23f of the calculation unit 23 calculates the pixel movement amount for each line of the captured image corresponding to the line of the image sensor 12 of the imaging device 10, based on the angle change amount calculated in step S160 (step S170). Then, the pixel movement amount calculation unit 23f outputs movement amount data indicating the calculated pixel movement amount to the image correction unit 24.

[0104] Finally, the image correction unit 24 corrects the captured image (see, for example, FIG. 2A ) represented by the captured image data output from the captured image acquisition unit 21 in accordance with the movement amount represented by the movement amount data output from the pixel movement amount calculation unit 23 f (the movement amount of pixels for each line of the captured image 30) (step S180). When the process of step S180 is completed, the image processing device 20 ends the image correction process shown in FIG.

[0105] As described above in detail, in this embodiment, the image processing device 20 (image correction device) is an image correction device that corrects a captured image by moving pixels that constitute the captured image captured by the imaging device 10, and includes an angle calculation unit 23c, a drift correction unit 23d, a pixel movement amount calculation unit 23f, and an image correction unit 24. The angle calculation unit 23c calculates the angle (attitude angle) of the imaging device 10 based on the angular velocity detected by the gyro sensor 11. The drift correction unit 23d corrects the angle by removing the drift component of the gyro sensor 11 from the calculated angle. The pixel movement amount calculation unit 23f calculates the pixel movement amount for each line of the captured image that corresponds to the line of the imaging element (image sensor 12) of the imaging device 10 based on the corrected angle. The image correction unit 24 corrects the captured image in accordance with the calculated pixel movement amount.

[0106] According to this embodiment configured as described above, the angle of the imaging device 10 calculated based on the angular velocity detected by the gyro sensor 11 is corrected by removing the drift component of the gyro sensor 11 from the angle, the amount of pixel movement is calculated for each line of the captured image based on the corrected angle, and the captured image is corrected according to the calculated amount of pixel movement. Therefore, it is possible to correct the captured image without using acceleration and without the need to perform signal processing using a Kalman filter, which increases calculation costs, and as a result, it is possible to reduce calculation costs and suppress the occurrence of image blur.

[0107] Moreover, in this embodiment, the image processing device 20 further includes a bias removal unit 23a. The bias removal unit 23a removes bias components (due to temperature conditions, etc.) included in the angular velocities detected by the gyro sensor 11. According to this embodiment configured as described above, the bias components included in the angular velocities detected by the gyro sensor 11 are removed, and therefore the angle calculation unit 23c can calculate the angle (attitude angle) of the image capture device 10 based on the angular velocities with higher accuracy than when the bias components are not removed.

[0108] In this embodiment, the image processing device 20 further includes an angular velocity conversion unit 23b. The angular velocity conversion unit 23b converts the angular velocity detected by the gyro sensor 11 so that the three-dimensional coordinate systems of the image capture device 10 (left-handed coordinate system) and the gyro sensor 11 (right-handed coordinate system) match. According to this embodiment configured as described above, even if the three-dimensional coordinate systems of the image capture device 10 and the gyro sensor 11 differ, it is possible to determine an angular velocity that matches the three-dimensional coordinate systems of the image capture device 10 and the gyro sensor 11.

[0109] In the above embodiment, the calculation unit 23 may first calculate the angle (posture angle) of the imaging device 10 corresponding to the N-th line 50 of the captured image 30, and then calculate the amount of movement of the pixel on the N-th line 50 from the calculated angle. For example, referring to FIG. 9 , the angle of the imaging device 10 corresponding to the pixel 40 calculated based on the angular velocity detected by the gyro sensor 11 before the pixel 41 in the time series may be called Angle ｂｅｆｏｒｅand the timestamp for the pixel 40 (the output timing of the angular velocity detection result by the gyro sensor 11 corresponding to the pixel 40) is ｂｅｆｏｒｅ The angle of the imaging device 10 corresponding to the pixel 41 calculated based on the angular velocity detected by the gyro sensor 11 after the pixel 40 in the time series is defined as Angle ａｆｔｅｒ and the timestamp for the pixel 41 (the output timing of the angular velocity detection result by the gyro sensor 11 corresponding to the pixel 41) is ａｆｔｅｒ In this case, the angle Angle of the imaging device 10 corresponding to the N-th line 50 of the captured image 30 is Ｎ can be calculated by the following equation (37).

[0110]

[0111] The pixel movement amount (Δx Ｎ , Δy Ｎ ) is the angle Angle of the imaging device 10 calculated corresponding to the N-th line 50 of the captured image 30. Ｎ can be calculated using the following equations (38) to (40): where d is the focal length from the center of the imaging lens 13 to the focal point, as described with reference to FIGS. 7A and 7B. ｘ is the sensor size on the Y axis per pixel in the image sensor 12, and μ ｙ is the sensor size on the Z axis per pixel in the image sensor 12.

[0112]

[0113]

[0114]

[0115] Furthermore, in the above embodiment, an example has been described in which the image processing device 20 includes the bias removal unit 23a, but the present invention is not limited to this, and the image processing device 20 does not need to include the bias removal unit 23a. However, from the viewpoint of accurately calculating the angle (attitude angle) of the imaging device 10 based on the angular velocity detected by the gyro sensor 11, it is preferable that the image processing device 20 includes the bias removal unit 23a that removes the bias component included in the angular velocity.

[0116] In the above embodiment, the image processing device 20 includes the angular velocity conversion unit 23b, but the present invention is not limited to this. For example, if the three-dimensional coordinate systems of the imaging device 10 and the gyro sensor 11 are the same, the image processing device 20 does not need to include the angular velocity conversion unit 23b.

[0117] Various embodiments of the present invention have been described above, but these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0118] 1: image processing system, 10: imaging device, 11: gyro sensor, 12: image sensor, 13: imaging lens, 20: image processing device, 21: captured image acquisition unit, 22: angular velocity acquisition unit, 23: calculation unit, 23a: bias removal unit, 23b: angular velocity conversion unit, 23c: angle calculation unit, 23d: drift correction unit, 23e: angle change amount calculation unit, 23f: pixel movement amount calculation unit, 24: image correction unit, 25: display control unit, 26: display unit, 27: memory unit, 30: captured image, 40, 41: pixel, 50: line

Claims

1. An image correction device that corrects a captured image by moving pixels that constitute the image captured by an imaging device, comprising: an angle calculation unit, a drift correction unit, a pixel movement amount calculation unit, and an image correction unit, wherein the angle calculation unit calculates the angle of the imaging device based on the angular velocity of the imaging device detected by a gyro sensor, the drift correction unit corrects the angle by removing a drift component of the gyro sensor from the calculated angle, the pixel movement amount calculation unit calculates the amount of movement of the pixels for each line of the captured image that corresponds to a line of an imaging element of the imaging device based on the corrected angle, and the image correction unit corrects the captured image in accordance with the amount of movement of the pixels calculated for each line of the captured image.

2. An image correction device according to claim 1, further comprising a bias removal unit, wherein the bias removal unit removes a bias component contained in the angular velocity detected by the gyro sensor.

3. An image correction device according to claim 1, further comprising an angular velocity conversion unit, wherein the angular velocity conversion unit converts the angular velocity detected by the gyro sensor so that three-dimensional coordinate systems coincide between the imaging device and the gyro sensor.

4. An image correction device according to claim 1, further comprising an angle change amount calculation unit, wherein the angle change amount calculation unit calculates the angle change amount of the imaging device based on the corrected angle, and the pixel movement amount calculation unit calculates the movement amount of the pixel for each line of the captured image based on the calculated angle change amount.

5. An image correction device according to claim 1, wherein the imaging device vibrates at a vibration frequency of 5 Hz or more and a vibration angle of 0.15° or less.

6. An image correction method for correcting a captured image captured by an imaging device by moving pixels that constitute the image, the image correction method comprising: an angle calculation step, a drift correction step, a pixel movement amount calculation step, and an image correction step; in the angle calculation step, an angle of the imaging device is calculated based on an angular velocity of the imaging device detected by a gyro sensor; in the drift correction step, the angle is corrected by removing a drift component of the gyro sensor from the calculated angle; in the pixel movement amount calculation step, an amount of movement of the pixels for each line of the captured image that corresponds to a line of an imaging element of the imaging device is calculated based on the corrected angle; and in the image correction step, the captured image is corrected in accordance with the amount of movement of the pixels calculated for each line of the captured image.

7. An image correction program that causes a processor to execute the image correction method according to claim 6.

Images