Image processing device, imaging device, image processing method, and computer program

Abstract

To provide a technology for obtaining a blur amount in a stereo image with higher accuracy.SOLUTION: Based on a start point of a motion vector in one image of a stereo image and a point corresponding to the start point in the other image of the stereo image, a first three-dimensional point corresponding to the start point is obtained. Based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image, a second three-dimensional point corresponding to the end point is obtained. Based on the first three-dimensional point and the second three-dimensional point, a third three-dimensional point is obtained by removing a motion component of a camera work from the second three-dimensional point, and based on the third three-dimensional point and the first three-dimensional point, a blur amount in the stereo image is obtained.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for determining the amount of blur in a stereo image. [Background technology]

[0002] In the image processing of moving images, image stabilization (shake correction) is an important element for improving image quality. Image stabilization includes shake detection and image correction. In the former, the amount of shake is estimated from the measurement results of a gyro sensor or the like, or from the optical flow of the image. In the latter, a shake-corrected image is obtained by physically moving the lens or sensor to suppress shake so that an image with shake canceled is captured based on the acquired amount of shake, or by digitally geometrically correcting an image that has already been captured to cancel the shake. Image stabilization is often performed within the camera when capturing an image, but digital geometric correction can also be performed as one of the processes in the process of editing and processing content recorded on a PC (personal computer) or the like.

[0003] Optical flow is a vector field obtained by detecting motion vectors (amount of pixel movement) from temporally consecutive image frames. A motion vector is expressed by the coordinates of a start point and an end point, or by the coordinates of a start point and a two-dimensional vector. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-318568 A Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, when detecting optical flow from an image and estimating blur due to vibration, if there is camera work such as moving forward, it is difficult to separate the camera work and blur from the optical flow, and the accuracy of estimating the blur may decrease. In this invention, we provide a technology for calculating the amount of blur in a stereo image with higher accuracy. [Means for solving the problem]

[0006] One aspect of the present invention is characterized in that it comprises a first acquisition means for obtaining a first three-dimensional point corresponding to a starting point based on a starting point of a motion vector in one image of a stereo image and a point corresponding to the starting point in the other image of the stereo image; a second acquisition means for obtaining a second three-dimensional point corresponding to an end point based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; and a third acquisition means for obtaining a third three-dimensional point by removing a movement component of camerawork from the second three-dimensional point based on the first three-dimensional point and the second three-dimensional point, and for obtaining an amount of blur in the stereo image based on the third three-dimensional point and the first three-dimensional point. Effect of the Invention

[0007] According to the present invention, the amount of blur in a stereo image can be calculated with higher accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Diagram 2] 4 is a flowchart of a process performed by the image processing device to obtain a blur amount in a stereo image. [Diagram 3] 11 is a flowchart showing details of the process in step S2010. [Figure 4] 11 is a flowchart showing details of the process in step S2050. [Diagram 5] 13 is a flowchart of the process of step S2050. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] [First embodiment] First, an example of the hardware configuration of an image processing device according to this embodiment will be described with reference to the block diagram of Fig. 1. The image processing device according to this embodiment may be a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device. The image processing device according to this embodiment may also be an embedded circuit in an imaging device having an imaging unit that captures a video in which each frame is a stereo image (hereinafter referred to as a stereo video). The image processing device according to this embodiment may also be an embedded circuit in a device that requires a function of calculating the amount of blur in a stereo video.

[0011] The CPU 105 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 105 controls the operation of the entire image processing device, and executes or controls various processes that will be described as processes performed by the image processing device.

[0012] The RAM 102 has an area for storing computer programs and data loaded from the external storage 107, and an area for storing data received from the outside via the input / output I / F 108. The RAM 102 also has a work area used when the CPU 105 and the graphic processor 103 execute various processes. In this way, the RAM 102 can provide various areas as needed.

[0013] The external storage 107 is a non-volatile large-capacity information storage device such as a hard disk drive or an SSD (solid state drive using flash memory). The external storage 107 stores an OS (operating system), computer programs and data for causing the CPU 105 and the graphic processor 103 to execute or control various processes described as processes performed by the image processing device. The computer programs and data stored in the external storage 107 are loaded into the RAM 102 as appropriate under the control of the CPU 105, and become targets for processing by the CPU 105.

[0014] The input / output I / F 108 is an interface for performing data communication (transmission and reception of data) with an external device. For example, the input / output I / F 108 may be an interface for connecting to a wireless / wired network such as a LAN or the Internet, or may be an interface for connecting to a device capable of supplying moving images / still images.

[0015] The graphic processor 103 is a processor that executes various processes for displaying images and characters on the display 104. For example, the graphic processor 103 decodes encoded images and characters and displays them on the display 104, and performs image processing and other processes on the images and characters to be displayed.

[0016] Display 104 is a display device having a liquid crystal screen, and displays the results of processing by CPU 105 and graphic processor 103 as images and characters. User I / F 106 is a user interface such as a keyboard, mouse, touch panel, etc., and allows a user to input various instructions to CPU 105 by operating it. User I / F 106 may also be integrated with display 104 to form a touch panel screen.

[0017] The RAM 102 , the external storage 107 , the input / output I / F 108 , the graphic processor 103 , the CPU 105 , and the user I / F 106 are all connected to the bus 101 .

[0018] The hardware configuration shown in Fig. 1 is merely an example of a hardware configuration applicable to the image processing device according to this embodiment, and can be modified / altered as appropriate. For example, although the number of CPUs 105 is one in Fig. 1, multiple CPUs 105 may be provided. In this case, the image processing device enables parallel operation by multi-thread processing.

[0019] The image processing device according to this embodiment obtains the amount of blur in a stereo image, which is each frame in a stereo video. The stereo image is a set of one (left) captured image (left image) and the other (right) captured image (right image), for example, a stereo fisheye image. In this embodiment, a case will be described in which the stereo image is a set of a "fisheye image with a 180-degree angle of view by equidistant projection" on the left side and a "fisheye image with a 180-degree angle of view by equidistant projection" on the right side. In this embodiment, both the left and right images are RGB images in which each pixel has an RGB value, but are not limited to this, and may be, for example, a YUV image or a monochrome luminance image. In the following, the image for which a motion vector is obtained is a luminance image in which each pixel has a luminance value, and if the image is not a luminance image, the image processing device converts the image into a luminance image and then uses it for calculation of a motion vector, etc. In addition, if the stereo video is encoded, the image processing device decodes the encoded stereo video using the graphic processor 103, and the decoded stereo video is the subject of the following processing.

[0020] The process performed by the image processing device according to this embodiment to obtain the amount of blur in a stereo image will be described with reference to the flowchart in Fig. 2. Before starting the process of step S2010, CPU 105 acquires stereo video in RAM 102. The method of acquiring stereo video in RAM 102 is not limited to a specific acquisition method. For example, CPU 105 may load stereo video stored in external storage 107 into RAM 102. CPU 105 may also acquire stereo images of each frame transmitted from an "imaging device that captures stereo video" connected to input / output I / F 108 into RAM 102 via input / output I / F 108.

[0021] In step S2010, CPU 105 obtains a calibration value for correcting vertical deviation (deviation in the vertical direction) from the stereo images. In this embodiment, CPU 105 obtains the calibration value using the first stereo image in the stereo video sequence. The process in step S2010 will be described in detail later.

[0022] In step S2020, CPU 105 initializes the value of variable n, which indicates the frame number in the stereo video (the frame number of the first frame is 0), to 0. In step S2030, CPU 105 determines whether the value of variable n is smaller than the "total number of frames in the stereo video". If the result of this determination is that the value of variable n is smaller than the "total number of frames in the stereo video", processing proceeds to step S2040. On the other hand, if the value of variable n is equal to the "total number of frames in the stereo video", processing according to the flowchart in FIG. 2 ends.

[0023] In step S2040, CPU 105 acquires the optical flow of stereo image [n]. "Stereo image [n]" represents the stereo image of the n-th frame in the stereo video sequence.

[0024] First, the CPU 105 performs feature point detection and matching processing by the ORB (Oriented FAST and Rotated BRIEF) method on the left image (left image [n]) in the stereo image [n] and the left image (left image [n+1]) in the stereo image [n+1].

[0025] As a result, CPU 105 detects multiple feature points from each of left image [n] and left image [n+1], and finds a pair of feature points that minimizes the Hamming distance of the ORB feature between the multiple feature points detected from left image [n] and the multiple feature points detected from left image [n+1]. That is, CPU 105 can determine, for each pair, a vector whose start point is the "feature point detected from left image [n]" in the pair and whose end point is the "feature point detected from left image [n+1]" in the pair, as a "motion vector corresponding to the pair." CPU 105 then obtains a set of motion vectors (motion vector group) determined for all pairs as the optical flow of stereo image [n].

[0026] In step S2050, CPU 105 obtains the amount of blur in stereo image [n] (amount of blur [n]) from the "optical flow of stereo image [n]" acquired in step S2040. Details of the process in step S2050 will be described later. In step S2060, CPU 105 increments the value of variable n by 1. Then, the process proceeds to step S2030.

[0027] Next, details of the process in step S2010 above will be described with reference to the flowchart in Fig. 3. In step S3010, CPU 105 performs feature point detection using the ORB method on each of the left image (first left image) and right image (first right image) of the stereo image of the first frame in the stereo video, and detects multiple feature points from each of the first left image and the first right image.

[0028] In step S3020, CPU 105 pairs corresponding feature points between the multiple feature points detected from the initial left image and the multiple feature points detected from the initial right image. For example, CPU 105 pairs a feature point of interest in the initial left image with a feature point that has the smallest Hamming distance between the feature point of interest and the multiple feature points detected from the initial right image.

[0029] In step S3030, CPU 105 sets a provisional calibration value. In this embodiment, the elements of the calibration value are the horizontal and vertical inclinations of the left and right optical axes, and the horizontal and vertical offsets. Each time step S3030 is executed, different provisional calibration values ​​are set one after another so that the calibration value converges in the form of so-called iterative calculation. Methods for converging values ​​by iterative calculation include the steepest descent method, Newton's method, and Levenberg-Macquart method, and the Levenberg-Macquart method is used. Note that CPU 105 may generate a random value each time step S3030 is executed and set it as the provisional calibration value. In this way, the method for setting the provisional calibration value in step S3030 is not limited to a specific setting method.

[0030] In step S3040, CPU 105 uses the provisional calibration values ​​set in step S3030 to identify the ray direction from the coordinates of the feature points in the first left image and the coordinates of the feature points in the first right image, calculates the closest distance between the two rays, and obtains the sum of squares of the closest distances for all feature point pairs. The XY coordinate system of an ideal equidistant projection fisheye image that is free of distortion and does not require calibration can be easily converted to polar coordinates. In this embodiment, the XY coordinate system is converted to polar coordinates by removing distortion and adding the inclination of the optical axis and offset deviation, which are provisional calibration values. The polar coordinates directly correspond to the ray direction. If the calibration values ​​are correct, the rays of the corresponding feature point pairs will intersect (come as close as possible).

[0031] In step S3050, if the sum of squares calculated in this step S3040 is smaller than the minimum value Smin of the sums of squares calculated in the previous steps S3040, CPU 105 updates the calibration value with the provisional calibration value set in this step S3030, and updates the minimum value Smin with the sum of squares calculated in the current step S3040. On the other hand, if the sum of squares calculated in this step S3040 is equal to or larger than the minimum value Smin of the sums of squares calculated in the previous steps S3040, CPU 105 does not update the calibration value and minimum value Smin.

[0032] In step S3060, CPU 105 determines whether or not the calibration value has converged. There are various methods for determining whether or not the calibration value has converged, and the method is not limited to a specific method.

[0033] If it is determined that the calibration values ​​have converged as a result of this determination, the process proceeds to step S2020. On the other hand, if it is determined that the calibration values ​​have not converged, the process proceeds to step S3030.

[0034] The calibration value at the time when the process of step S2010 (the process according to the flowchart in FIG. 3) is completed is used in the subsequent processes. By projecting the coordinates and images using such calibration values, the vertical shift of the same subject in the left and right images is reduced, and the effect of stereo rectification is obtained.

[0035] Next, the details of the process in step S2050 will be described with reference to the flowchart in FIG. 4. In step S4010, CPU 105 sets a patch pt1 centered on a position in the "left image in stereo image [n]" that corresponds to the "position of the starting point pi of the i-th motion vector in the optical flow (a set of motion vectors) of stereo image [n]". CPU 105 then calculates a BRIEF feature from the set patch pt1, and searches the right image in stereo image [n] for "patch pt2 having a BRIEF feature most similar to the BRIEF feature" along the horizontal direction of the image. The search is performed in a space that reflects the calibration value. CPU 105 then obtains the horizontal deviation between the center position of patch pt1 and the center position of the searched patch pt2 as parallax. The number of motion vectors included in the optical flow (a set of motion vectors) is count. At this time, the CPU 105 performs such processing for i=1 to count, thereby making it possible to obtain the disparity of the starting point for each motion vector included in the optical flow (a set of motion vectors).

[0036] In step S4020, CPU 105 sets patch pt3 centered on a position in "the left image in stereo image [n]" corresponding to "the position of the end point qi of the i-th motion vector in the optical flow (a set of motion vectors) of stereo image [n]". CPU 105 then calculates a BRIEF feature from the set patch pt3, and searches "a patch pt4 having a BRIEF feature most similar to the BRIEF feature from the right image in stereo image [n]" along the horizontal direction of the image. This search is performed in a space reflecting the calibration value. CPU 105 then obtains the horizontal deviation between the center position of patch pt3 and the center position of the searched patch pt4 as the parallax. CPU 105 performs such processing for i=1 to count, thereby obtaining the parallax of the end point for each motion vector included in the optical flow (a set of motion vectors).

[0037] In step S4030, the CPU 105 calculates a projection start point p by projecting the start point pi onto a three-dimensional space using the start point pi, the parallax of the start point pi, and the calibration value. 3 i (three-dimensional point). Projection from the XY coordinates of stereo images to three-dimensional space coordinates is a known technique, so a description thereof will be omitted. CPU 105 performs such processing for i=1 to count to obtain the projection start points corresponding to the respective start points.

[0038] In step S4040, the CPU 105 calculates a projected end point qi by projecting the end point qi onto a three-dimensional space using the end point qi, the parallax of the end point qi, and the calibration value. 3 CPU 105 performs such processing for i=1 to count to obtain the projection end points corresponding to the respective end points.

[0039] In step S4050, the CPU 105 determines the projection start point p 3 i and the projection end point q 3 Using i, the camerawork movement component w (camerawork of the camera that captured the stereo video) is calculated. In this embodiment, the optical axis direction is the z-axis, the image vertical direction is the y-axis, and the image horizontal direction is the x-axis, the camerawork movement component is w, and the camerawork is only the forward component in the optical axis direction. In this case, CPU 105 calculates the camerawork movement component w according to the following equations (1) to (3).

[0040]

number

[0041] In step S4060, the CPU 105 calculates the projection end point q 3 The corrected projection end point q' obtained by removing the camerawork movement component w from i 3 Find i.

[0042]

number

[0043] CPU 105 performs the processes of steps S4050 and S4060 for i=1 to count, thereby obtaining corrected projection end points corresponding to the respective projection end points.

[0044] In step S4070, the CPU 105 calculates the projection start point p 3 i, correction projection end point q' 3 Normalize each of i to a vector with a magnitude of 1. x , p y , p z} is normalized according to the following formula (5).

[0045]

number

[0046] Here, norm(p) is a function that projects (converts into polar coordinates) the orthogonal three-dimensional coordinates (x, y, z) of a three-dimensional point onto coordinates on a sphere of radius 1. CPU 105 also sets values ​​for pan (p), tilt (t), and roll (r), which are components of the amount of shake (shake components), and configures a matrix (parameter) M shown in the following formula (6) using the pan (p), tilt (t), and roll (r) for which values ​​have been set.

[0047]

number

[0048] CPU 105 generates a plurality of matrices M while changing the values ​​to be set for each of pan (p), tilt (t), and roll (r). The values ​​to be set for each of pan (p), tilt (t), and roll (r) may be set randomly for each matrix M, or values ​​registered for each matrix M in a list created in advance may be read out in sequence and set in the matrix M, and the setting method is not limited to a specific one.

[0049] Then, the CPU 105 selects the "projection start point p" from among the generated matrices M according to the formula (7).3 The normalized result of i and the corrected projection end point q' 3 Matrix M that minimizes the "sum of the difference between the normalization result of i converted by matrix M and the normalization result for i=1 to count" (total value) is output as matrix e. In this way, CPU 105 can specify matrix M that minimizes the total value as matrix e. As a result, CPU 105 can obtain pan (p), tilt (t), and roll (r) set to configure the specified matrix M (matrix e) as components of the amount of blur in stereo image [n] (pan, tilt, and roll movements due to camera vibration from stereo image [n] to stereo image [n+1]).

[0050] Specifically, the maximum values ​​of the assumed pan (p), tilt (t), and roll (r) are set, and the values ​​are obtained by iterative calculation. As explained in the calibration value calculation, the iterative calculation can be performed using the steepest descent method, Newton's method, Levenberg-McQuart method, etc., but in this embodiment, the Levenberg-McQuart method is used.

[0051] In this way, according to this embodiment, the parallax between the left and right images is calculated for each of the start and end points of the motion vector, the start and end points are projected onto a three-dimensional space using the parallax, and the camerawork component is removed from the end point coordinates in the three-dimensional space coordinate system, allowing efficient calculation of the blur component that is not affected by the camerawork.

[0052] In addition, even if misalignment occurs in the optical system due to heat or deterioration over time, the input image is used to calculate the calibration value each time, enabling more accurate projection into 3D space and preventing deterioration in the accuracy of blur detection. This means that the user no longer needs to take a picture of the calibration chart and manually calculate the calibration value each time.

[0053] In the present embodiment, the calibration value is used for correction during coordinate conversion, but the image itself may be corrected. By detecting the motion vector from the corrected image, the detection accuracy of the motion vector is improved.

[0054] The image processing device can also create an anti-shake image by calculating a moving average for each blur component of the blur amount and performing geometric correction on a stereo image based on the difference between the blur component of the blur amount calculated for the current frame and the corresponding moving average.

[0055] In addition, since a fisheye image is input in this embodiment, an image for VR (Virtual Reality) display can be generated by converting the left and right images into equirectangular images with vibration isolation. If stereo display is not required, for example, only the left image may be output.

[0056] In addition, in the present embodiment, the image is described as a fisheye image (equidistant projection), but it is not limited to this, and may be, for example, a normal lens image (central projection image). In addition, in the present embodiment, the motion vector is obtained using the ORB method, but the method for obtaining the motion vector is not limited to a specific method, and other feature point detection and matching methods such as the SIFT method may be applied, or template matching may be used. In the present embodiment, an example is described in which fisheye images with different distortions are input, so the ORB method, which uses feature amount calculation robust to distortion, is useful, but if it is a central projection image, the distortion is small, so template matching is sufficient.

[0057] In addition, in this embodiment, the start and end points of the motion vectors in the left and right images are matched using BRIEF features, but this is not limited to this, and other features such as SIFT may be used, or template matching may be used.

[0058] In addition, in this embodiment, the calibration value is calculated so that the sum of squares of the closest distances of two rays is minimized. However, the present invention is not limited to this, and it is also possible to project feature points of the left image onto the right image via the midpoint of the closest points, and search for a calibration value that minimizes the sum of squared errors of all feature point pairs by taking the difference between the corresponding feature point coordinates of the right image and the projected point as the error.

[0059] In addition, in this embodiment, the left and right optical axis inclination and offset deviation are estimated as the calibration values, but this is not limited to this. For example, one side may be fixed and the other optical axis inclination and offset deviation may be estimated, or if the magnification or focal length is deviated, these parameters may be further estimated.

[0060] In this embodiment, the camera work is limited to the forward movement component, but it is not limited to this, and for example, horizontal and vertical movement components may also be estimated and removed from the three-dimensional coordinates. In this embodiment, the rotation calculation is performed using a matrix, but it is not limited to this, and for example, a quaternion may be used.

[0061] [Second embodiment] In the following embodiments including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiments are the same as the first embodiment. In this embodiment, a three-dimensional point (projection start point / projection end point) is obtained based on a parallax that is equal to or smaller than a threshold value. The process of step S2050 according to this embodiment will be described with reference to the flowchart in FIG.

[0062] In step S5010, the CPU 105 performs the same process as in step S4010 described above to obtain the disparity of the starting point for each motion vector included in the optical flow (a set of motion vectors).

[0063] In step S5020, the CPU 105 performs the same process as in step S4020 described above to obtain the disparity of the end point for each motion vector included in the optical flow (a set of motion vectors).

[0064] In step S5030, the CPU 105 acquires, as the target start point and the target end point, the start point and the end point of the motion vectors included in the optical flow (a set of motion vectors) whose start point disparity and end point disparity are both less than a threshold. In this embodiment, the threshold is one pixel, but is not limited to this. In this embodiment, when the disparity is less than the threshold, it is determined that "the disparity is 0".

[0065] Then, the CPU 105 projects the coordinates of each of the target start point and the target end point onto coordinates on a sphere of radius 1 (converts them into polar coordinates). Then, the CPU 105 generates a plurality of matrices M in the same manner as in the first embodiment, and calculates norm(p 3 i) with the target starting point converted to polar coordinates, and norm(q' 3 From the equation obtained by replacing i) with "the target end point converted into polar coordinates", the matrix e is obtained in the same manner as in the first embodiment.

[0066] In the first embodiment, the feature point coordinates are projected into a three-dimensional space. However, if the baseline length is unknown, it is not possible to project into a three-dimensional space. In this embodiment, a feature point pair that corresponds to a parallax of 0, that is, that can be considered to exist at infinity, is extracted, and the blur component is calculated only from the movement of those feature points, making it possible to calculate a blur that is not influenced by a forward component. This is because it can be considered that there is almost no forward component at infinity. In this way, by matching the left and right images, it is possible to accurately calculate the blur component without projection into a three-dimensional space.

[0067] [Third embodiment] In the first and second embodiments, the processing according to the flowcharts has been described as being executed by the CPU 105, but part or all of the processing may be executed by the graphic processor 103. The number of CPUs 105 and graphic processors 103 is not limited to one, and the image processing device may have other processors. For example, the image processing device may have hardware that performs part of the processing according to the flowcharts.

[0068] In other words, in an image processing device, the processing subject is not limited to a specific subject, the number of processing subjects is not limited to one, and the execution form of the processing, such as whether the processing is executed in a single thread or multiple threads, is not limited to a specific execution form.

[0069] Also, the method of detecting feature points used in the first and second embodiments is just an example, and other detection methods may be adopted. Also, in the first and second embodiments, a calibration value is calculated in step S2010, and the amount of blur of the stereo image is calculated with high accuracy using the calculated calibration value. However, a calibration value calculated in advance may be used, and in that case, the calculated calibration value is acquired in step S2010.

[0070] Also, if the vertical deviation between the left and right images in the stereo image is small enough to be ignored, the process of calculating / obtaining the calibration value (i.e., the process of step S2010) may be omitted. In this case, the subsequent processes will be performed without using the calibration value.

[0071] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0072] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0073] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0074] The invention of this specification includes the following image processing device, imaging device, image processing method, and computer program. (Item 1) a first acquisition means for acquiring a first three-dimensional point corresponding to a start point of a motion vector in one image of the stereo images, based on the start point of the motion vector in the one image of the stereo images and a point corresponding to the start point in the other image of the stereo images; a second acquisition means for acquiring a second three-dimensional point corresponding to the end point based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; a third acquisition means for determining a third three-dimensional point by removing a movement component of camerawork from the second three-dimensional point based on the first three-dimensional point and the second three-dimensional point, and for determining an amount of blur in the stereo image based on the third three-dimensional point and the first three-dimensional point; An image processing device comprising: (Item 2) The image processing device described in item 1, characterized in that the first acquisition means calculates a disparity based on a starting point of a motion vector in the one image and a point corresponding to the starting point in the other image, and calculates the first three-dimensional point based on the starting point and the disparity. (Item 3) The image processing device described in item 1 or 2, characterized in that the first acquisition means calculates a disparity based on an end point of a motion vector in the one image and a point corresponding to the end point in the other image, and calculates the second three-dimensional point based on the end point and the disparity. (Item 4) The third acquisition means is 4. The image processing device according to any one of items 1 to 3, characterized in that a movement component of camerawork is calculated based on a difference between the first three-dimensional point and the second three-dimensional point. (Item 5) moreover, 5. The image processing device according to any one of items 1 to 4, further comprising a fourth acquisition means for obtaining a calibration value for correcting a vertical misalignment between the one image and the other image based on a plurality of feature points in the one image and a plurality of feature points in the other image. (Item 6) 6. The image processing device according to item 5, wherein the first acquisition means further obtains the first three-dimensional point by using the calibration value. (Item 7) 7. The image processing device according to item 5 or 6, wherein the second acquisition means further obtains the second three-dimensional point by using the calibration value. (Item 8) The image processing device described in any one of items 1 to 7, characterized in that the third acquisition means obtains, as the amount of blur, a parameter that minimizes a value based on a difference between a three-dimensional point obtained by normalizing the first three-dimensional point and a three-dimensional point obtained by converting the three-dimensional point obtained by normalizing the third three-dimensional point using a parameter. (Item 9) a first acquisition means for acquiring a start point and an end point to be used based on a first parallax based on a start point of a motion vector in one image of stereo images and a point corresponding to the start point in the other image of the stereo images, and a second parallax based on an end point of a motion vector in the one image and a point corresponding to the end point in the other image; a second acquisition means for determining an amount of blur in the stereo image based on the start point and the end point acquired by the first acquisition means; An image processing device comprising: (Item 10) 10. The image processing device according to item 9, wherein the first acquisition means acquires a start point and an end point at which the first disparity and the second disparity are less than a threshold value. (Item 11) 11. The image processing device according to item 9 or 10, wherein the second acquisition means obtains, as the amount of blur, a parameter that minimizes a value based on a difference between a three-dimensional point corresponding to a start point acquired by the first acquisition means and a three-dimensional point obtained by converting a three-dimensional point corresponding to an end point acquired by the first acquisition means using a parameter. (Item 12) 12. The image processing device according to any one of items 1 to 11, wherein the stereo images are images of each frame of a moving image. (Item 13) An imaging unit that captures a stereo image; An image processing device according to any one of items 1 to 12, An imaging device comprising: (Item 14) An image processing method performed by an image processing device, comprising: a first acquisition step in which a first acquisition means of the image processing device obtains a first three-dimensional point corresponding to a starting point of a motion vector in one image of the stereo images, based on the starting point of the motion vector in the one image of the stereo images and a point corresponding to the starting point in the other image of the stereo images; a second acquisition step in which a second acquisition means of the image processing device obtains a second three-dimensional point corresponding to the end point based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; a third acquisition step in which a third acquisition means of the image processing device acquires a third three-dimensional point by removing a movement component of camerawork from the second three-dimensional point based on the first three-dimensional point and the second three-dimensional point, and acquires an amount of blur in the stereo image based on the third three-dimensional point and the first three-dimensional point; An image processing method comprising: (Item 15) An image processing method performed by an image processing device, comprising: a first acquisition step in which a first acquisition means of the image processing device acquires a start point and an end point to be used based on a first parallax based on a start point of a motion vector in one image of stereo images and a point corresponding to the start point in the other image of the stereo images, and a second parallax based on an end point of a motion vector in the one image and a point corresponding to the end point in the other image; a second acquisition step in which a second acquisition means of the image processing device calculates an amount of blur in the stereo image based on the start point and the end point acquired in the first acquisition step; An image processing method comprising: (Item 16) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 12.

[0075] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] 101: Bus 102: RAM 103: Graphics processor 104: Display 105: CPU 106: User I / F 107: External storage 108: Input / Output I / F

Claims

1. a first obtaining means for obtaining a first three-dimensional point corresponding to a starting point of a motion vector in one image of the stereo images, based on the starting point of the motion vector in the other image of the stereo images and a point corresponding to the starting point; a second obtaining means for obtaining a second three-dimensional point corresponding to the end point based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; a third acquisition means for determining a third three-dimensional point by removing a movement component of camerawork from the second three-dimensional point based on the first three-dimensional point and the second three-dimensional point, and for determining a blur amount in the stereo image based on the third three-dimensional point and the first three-dimensional point; a fourth acquisition means for obtaining a calibration value for correcting a vertical misalignment between the one image and the other image based on a plurality of feature points in the one image and a plurality of feature points in the other image; Equipped with The first acquisition means further determines the first three-dimensional point using the calibration value; the first obtaining means obtains a parallax based on a start point of a motion vector in the one image and a point in the other image corresponding to the start point, and obtains the first three-dimensional point based on the start point and the parallax; the first obtaining means obtains a parallax based on an end point of the motion vector in the one image and a point in the other image corresponding to the end point, and obtains the second three-dimensional point based on the end point and the parallax; The second acquisition means further obtains the second three-dimensional point using the calibration value; The third obtaining means obtains, as the amount of blur, a parameter that minimizes a value based on a difference between a three-dimensional point obtained by normalizing the first three-dimensional point and a three-dimensional point obtained by converting the three-dimensional point obtained by normalizing the third three-dimensional point using a parameter.

1. An image processing device comprising:

2. The third acquisition means 2. The image processing device according to claim 1, wherein a movement component of camerawork is calculated based on a difference between the first three-dimensional point and the second three-dimensional point.

3. a first obtaining means for obtaining a final start point and a final end point based on a first parallax based on a start point of a motion vector in one image of stereo images and a point corresponding to the start point in the other image of the stereo images, and a second parallax based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; a second obtaining means for obtaining a blur amount in the stereo image based on the final start point and the final end point; a third obtaining means for obtaining a calibration value for correcting a vertical misalignment between the one image and the other image based on a plurality of feature points in the one image and a plurality of feature points in the other image; Equipped with The first acquisition means further determines a first three-dimensional point using the calibration value; the first obtaining means obtains a parallax based on a start point of a motion vector in the one image and a point in the other image corresponding to the start point, and obtains the first three-dimensional point based on the start point and the parallax; the first obtaining means obtains a parallax based on an end point of the motion vector in the one image and a point in the other image corresponding to the end point, and obtains a second three-dimensional point based on the end point and the parallax; The second acquisition means further obtains the second three-dimensional point using the calibration value; The second acquisition means obtains, as the amount of blur, a parameter that minimizes a value based on a difference between a three-dimensional point corresponding to the start point acquired by the first acquisition means and a three-dimensional point obtained by converting the three-dimensional point corresponding to the end point acquired by the first acquisition means using the parameter.

1. An image processing device comprising:

4. The image processing device according to claim 3 , wherein the first acquisition means acquires a start point and an end point at which the first parallax and the second parallax are less than a threshold value.

5. 2. The image processing device according to claim 1, wherein the stereo images are images of each frame of a moving image.

6. Furthermore, an imaging unit that captures the stereo image, The first acquisition means acquires the stereo image using the imaging unit.

2. The image processing device according to claim 1, wherein:

7. An image processing method performed by an image processing device, a first acquisition step in which a first acquisition means of the image processing device obtains a first three-dimensional point corresponding to a starting point of a motion vector in one image of the stereo images, based on the starting point of the motion vector in the other image of the stereo images and a point corresponding to the starting point; a second acquisition step in which a second acquisition means of the image processing device obtains a second three-dimensional point corresponding to the end point based on the end point of the motion vector in the one image and a point corresponding to the end point in the other image; a third acquisition step in which a third acquisition means of the image processing device determines a third three-dimensional point by removing a movement component of camerawork from the second three-dimensional point based on the first three-dimensional point and the second three-dimensional point, and determines a blur amount in the stereo image based on the third three-dimensional point and the first three-dimensional point; a fourth acquisition step in which a fourth acquisition means of the image processing device calculates a calibration value for correcting a vertical deviation between the one image and the other image based on a plurality of feature points in the one image and a plurality of feature points in the other image; Equipped with In the first obtaining step, the first three-dimensional point is obtained by using the calibration value; In the first obtaining step, a parallax is obtained based on a start point of a motion vector in the one image and a point in the other image corresponding to the start point, and the first three-dimensional point is obtained based on the start point and the parallax; In the first obtaining step, a parallax is obtained based on an end point of the motion vector in the one image and a point in the other image corresponding to the end point, and the second three-dimensional point is obtained based on the end point and the parallax; In the second obtaining step, the second three-dimensional point is obtained by using the calibration value, In the third obtaining step, a parameter that minimizes a value based on a difference between a three-dimensional point obtained by normalizing the first three-dimensional point and a three-dimensional point obtained by converting the three-dimensional point obtained by normalizing the third three-dimensional point using a parameter is obtained as the blur amount. An image processing method comprising:

8. In the third acquisition step, 8. The image processing method according to claim 7, wherein a movement component of camerawork is determined based on a difference between the first three-dimensional point and the second three-dimensional point.

9. An image processing method performed by an image processing device, a first acquisition step in which a first acquisition means of the image processing device acquires a final start point and a final end point based on a first parallax based on a start point of a motion vector in one image of stereo images and a point corresponding to the start point in the other image of the stereo images, and a second parallax based on an end point of the motion vector in the one image and a point corresponding to the end point in the other image; a second obtaining step in which a second obtaining means of the image processing device obtains a blur amount in the stereo image based on the final start point and the final end point; a third acquisition step in which a third acquisition means of the image processing device calculates a calibration value for correcting a vertical deviation between the one image and the other image based on a plurality of feature points in the one image and a plurality of feature points in the other image; Equipped with In the first obtaining step, a first three-dimensional point is obtained using the calibration value; In the first obtaining step, a parallax is obtained based on a start point of a motion vector in the one image and a point in the other image corresponding to the start point, and the first three-dimensional point is obtained based on the start point and the parallax; In the first obtaining step, a parallax is obtained based on an end point of the motion vector in the one image and a point in the other image corresponding to the end point, and a second three-dimensional point is obtained based on the end point and the parallax; In the second obtaining step, the second three-dimensional point is obtained by using the calibration value, In the second obtaining step, a parameter that minimizes a value based on a difference between a three-dimensional point corresponding to the start point obtained in the first obtaining step and a three-dimensional point obtained by converting the three-dimensional point corresponding to the end point obtained in the first obtaining step using the parameter is obtained as the blur amount. An image processing method comprising:

10. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 5.

Images