Image acquisition device and control method of this

Abstract

Image acquisition device (100) which acquires an image signal using an imaging unit (109), comprising: a first acquisition unit configured to acquire initial information indicating a wobble of the image acquisition device (100) detected by a wobble detection unit; a second detection unit configured to detect secondary information indicating movement of an object detected in the image signal by the imaging unit (109); and an estimation unit (205) configured to estimate a position or location of the image acquisition device (100), wherein the estimation unit (205) comprises: a calculation unit (207) configured to calculate an estimate of the position or location of the image acquisition device (100) according to the band-limited first information or the first and second information, and a correction unit (208) configured to calculate a correction value for the estimate using the second information.

Description

BACKGROUND OF THE INVENTION Area of ​​the invention

[0001] The present invention relates to an optical device such as a video camera, a digital camera or an interchangeable lens thereof, and in particular to an image recording device with an estimation function of its own position and orientation, as well as a control method thereof. Description of the related technique

[0002] There is an image-capturing device that has a function for correcting image blur of an object or subject caused by hand shake, tremor, or the like. It is necessary to detect vibration and changes in position applied to the main body or housing of the image-capturing device using an inertial sensor in order to perform image blur correction. Furthermore, a change in position occurs due to movement intended by a photographer (hereinafter referred to as camera movement) during photography, where the photographer captures an object or subject (whether moving or stationary) within a photographic frame or viewpoint while moving with the object or subject. The image-capturing device must detect the change in position due to camera movement separately from the change in its orientation.

[0003] Methods for acquiring information about the position and orientation of a moving object include methods using inertial navigation data from an inertial sensor and positioning data from the Global Positioning System (GPS). US Patent 8,494,225 B2 discloses a method for correcting an estimation error in position and orientation information estimated by an inertial sensor using motion information from an image. Furthermore, a position and orientation estimation technique (optical and inertial sensor fusion) that uses a structure-from-motion (SFM) and an inertial sensor as a self-position estimation method for detecting the orientation and position of an image-capturing device is known. A method for estimating the three-dimensional position of an object in real space, as well as the position and orientation of an image-capturing device, using this technique is also known.

[0004] According to the current state of the art, correction using image information cannot be performed correctly if a bias error of an inertial sensor varies greatly due to a change in the photographic environment, or if the output of an inertial sensor varies significantly due to a sudden change in camera movement, and so on. Alternatively, problems arise such that a correct position and orientation estimation is not performed, and so on, until the correction is complete if it takes a long time to correct a bias error correctly.

[0005] US 2017 / 0026581A1 discloses an image stabilization device that performs image stabilization by using both an image sensor and a motion sensor. Image distortion and motion are stably corrected by using both the position of a feature point, extracted by the image sensor and image processing, and the motion position of the feature point, predicted by the motion sensor.

[0006] US 2011 / 0157380A1 discloses an image sensor that captures a moving image of an object whose image is produced by an imaging lens, a first detection unit that detects the vibration of an image-taking device, and a second detection unit that detects the movement of the object based on an image signal for each image-taking period that is output by the image sensor, a distance detection unit that detects an object distance which is a distance from the image-taking device to the object, a modification unit that modifies an output of the second detection unit based on the object distance detected by the distance detection unit, and a control unit that controls a vibration correction unit which optically performs a vibration correction based on an output of the first detection unit and an output of the second detection unit that is modified by the modification unit. SUMMARY OF THE INVENTION

[0007] The present invention provides an image acquisition device capable of capturing wobble information and performing position and orientation estimation with high accuracy, as well as a control method for this device.

[0008] A device according to one aspect of the present invention is an image acquisition device that acquires an image signal using an imaging unit, comprising: a first acquisition unit configured to acquire first information designating a wobble of the image acquisition device detected by a wobble detection unit; a second acquisition unit configured to acquire second information designating a movement of an object detected in an image signal by the imaging unit;and an estimation unit configured to estimate a position or location of the image acquisition device, wherein the estimation unit comprises: a calculation unit configured to calculate an estimated value of the position or location of the image acquisition device according to the band-limited first information or the first and second information, and a correction unit configured to calculate a correction value for the estimated value using the second information.

[0009] Further features of the present invention will become clear from the following description of exemplary embodiments (with reference to the accompanying drawings). BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a representation showing a configuration example of an image acquisition device according to embodiments of the present invention. Fig. Figure 2 is a representation showing a configuration example of a feature coordinate characteristic map and a position and location estimation unit according to the embodiments of the present invention. Fig. Figure 3 is a representation that describes an internal configuration of the position and orientation estimation unit according to a first embodiment of the present invention. Fig. Figure 4 is a representation that shows a relationship between a position and a location of an image recording device in a moving image frame, three-dimensional feature point coordinates, and feature point coordinates in a photographed image. Fig. Figure 5 is a relationship representation between a coordinate position of an object in world coordinates and a coordinate position in camera coordinates. Fig. Figure 6 is a representation that depicts a model of a perspective projection, where a virtual imaging surface is set up in front of a lens. Fig. Figure 7 is a diagram that describes a detailed internal configuration of the position and attitude correction unit. Fig. 8A and Fig. Figure 8B shows representations that depict relationships between a measure of vibration sensor noise and control parameters. Fig. Figure 9 is a flowchart of a position and location estimation process according to the first embodiment. Fig. Figure 10 is a representation describing an internal configuration of a position and orientation estimation unit according to a second embodiment of the present invention. Fig. Figure 11 is a flowchart of a position and location estimation process according to the second embodiment. DESCRIPTION OF THE EXAMPLES OF EXECUTION

[0010] Exemplary embodiments of the present invention are described with reference to the accompanying drawings. Each exemplary embodiment presents an image acquisition device with a position and orientation estimation function. [First embodiment]

[0011] Fig. Figure 1 is a block diagram showing a configuration example of an image acquisition device according to the present embodiment. An image acquisition device 100 is, for example, a digital camera and has a video photography function.

[0012] The image acquisition device 100 comprises a zoom unit 101. The zoom unit 101 forms an optical image generation system and includes a zoom lens for changing the photographic magnification. A zoom control unit 102 controls or drives the zoom unit 101 according to a control signal from a control unit 119. An image shake correction lens 103 (hereinafter referred to as a correction lens) is a movable optical element that corrects image shake. The correction lens 103 is movable in a direction orthogonal to an optical axis of an optical imaging system. An image shake correction lens control unit 104 controls the actuation or driving of the correction lens 103 according to a control signal from the control unit 119. An aperture / shutter unit 105 comprises a mechanical shutter with an aperture function. An aperture and shutter control unit 106 controls the actuation or driving of the aperture / shutter unit 105.The aperture / shutter unit 105 is driven by a control signal from the control unit 119. A focus lens 107 is a movable lens used for focal point adjustment, and its position along the optical axis of the optical imaging system can be changed. A focus control unit 108 controls or drives the focus lens 107 by a control signal from the control unit 119.

[0013] The imaging unit 109 acquires an image signal by converting an optical image formed by the optical imaging system into an electrical signal in a pixel unit by means of an imaging element such as a CCD image sensor or a CMOS image sensor. CCD stands for "Charge Coupled Device." CMOS stands for "Complementary Metal-Oxide Semiconductor." An imaging signal processing unit 110 performs analog-to-digital (A) / digital-to-digital (D) conversion, correlated double sampling, gamma correction, white balance correction, and color interpolation processing on an image signal output by the imaging unit 109 to convert it into a video signal.

[0014] A video signal processing unit 111 processes an image signal acquired by the imaging signal processing unit 110 according to an application. Specifically, the video signal processing unit 111 generates an image signal for display and performs processing to convert it into code or to convert it into a data file for recording. A display unit 112 optionally displays an image according to an image signal for display output by the video signal processing unit 111. A power supply unit 115 provides power to each unit of the image acquisition device according to an application. An external input / output terminal unit 116 is used to receive or output a communication signal or a video signal to or from an external device.An operating unit 117 has a control element, such as a button or switch, for a user to give an instruction to the image-capturing device. For example, the operating unit 117 has a release switch configured such that a first switch (designated SW1) and a second switch (designated SW2) are sequentially activated according to a certain amount of pressure applied to a release button. The operating unit 117 also has various types of mode-setting switches. A storage unit 118 stores various types of data, including image (or video) information or the like.

[0015] The control unit 119, for example, has a CPU, a ROM, and a RAM. CPU stands for "Central Processing Unit." ROM stands for "Read Only Memory." RAM stands for "Random Access Memory." The CPU executes a control program stored in the ROM and in the RAM and controls each unit of the image-capturing device 100 to perform various operations, which are described below. When SW1 is switched on by half-pressing the shutter release button enclosed in the control unit 117, the control unit 119 calculates an autofocus (AF) rating based on a video signal for display, which is output by the video signal processing unit 111 to the display unit 112. The control unit 119 performs automatic focus point detection and focus point adjustment control by controlling the focus control unit 108 based on the AF rating.Furthermore, the control unit 119 performs auto exposure (AE) processing to determine an aperture value and shutter speed to obtain a suitable exposure amount, based on luminance and brightness information from a video signal and a predetermined program sequence. Additionally, when SW2 is activated by fully pressing the shutter release button, the control unit 119 performs photographic processing using the determined aperture value and shutter speed, and controls each processing unit so that image data obtained by the imaging unit 109 is stored in the storage unit 118.

[0016] The control unit 117 has a control switch that is used to select an image shake correction (vibration protection / stability) mode.

[0017] When an image correction mode is selected by operating this control switch, the control unit 119 instructs the image correction lens control unit 104 to perform image correction operation. The image correction lens control unit 104 performs image correction operation according to a control command from the control unit 119 until an instruction to turn off image correction is given. The control unit 117 also has a photography mode selector switch capable of selecting between a still image photography mode and a moving image photography mode. A photography mode selection is performed by user operation of the photography mode selector switch, and the control unit 119 changes an operating condition of the image correction lens control unit 104.The image shake correction lens control unit 104 forms an image shake correction device of the present embodiment. The control unit 117 also has a playback mode selector switch for selecting a playback mode. When a user selects a playback mode by operating the playback mode selector switch, the control unit 119 performs a control operation to stop the image shake correction operation. The control unit 117 has a magnification change switch for issuing a command to change the zoom magnification. When a zoom magnification change is instructed by a user operation of the magnification change switch, the zoom control unit 102, which receives the instruction from the control unit 119, controls the zoom unit 101 to move a zoom lens to an instructed position.

[0018] Fig. Figure 2 is a block diagram showing a configuration for implementing position and orientation estimation of the image acquisition device according to the present embodiment. The image acquisition device 100 comprises a first vibration sensor 201 and a second vibration sensor 203. The first vibration sensor 201 is an angular velocity sensor that detects the angular velocity of the image acquisition device 100, and the second vibration sensor 203 is an accelerometer that detects the translational acceleration of the image acquisition device 100. An analog-to-digital converter 202 converts an analog signal detected by the first vibration sensor 201 into digital data. An analog-to-digital converter 204 converts an analog signal detected by the second vibration sensor 203 into digital data.

[0019] A motion vector detection unit 210 detects a motion vector based on a signal obtained by the imaging signal processing unit 110 processing image data acquired by the imaging unit 109. A feature point tracking unit 209 acquires the motion vector from the motion vector detection unit 210 and detects and tracks a motion position of coordinates in each frame or full image at the time of photographing or recording a moving image for a predetermined feature point in a photographed image.

[0020] A feature coordinate map and position and attitude estimation unit 205 (hereinafter referred to simply as the estimation unit) acquires shake or vibration information from each of the vibration sensors 201 and 203, from the A / D converters 202 and 204, and shake or vibration information based on image information from the feature point tracking unit 209. The estimation unit 205 estimates a feature coordinate map based on this information. The feature coordinate map represents information that describes a positional relationship comprising the position and attitude of the image acquisition device 100 and the depth of a photographed object or subject with respect to the image acquisition device. The estimation unit 205 is composed of a feature coordinate map estimation unit 206, a position and attitude estimation unit 207, a position and attitude correction unit 208, and a vibration sensor noise determination unit 211.

[0021] Next, with reference to Fig. 3 describes a configuration and processing of the position and location estimation unit 207. Fig. Figure 3 is a diagram showing a configuration example of the position and orientation estimation unit 207. The position and orientation estimation unit 207 consists of a position estimation unit 401 and a position estimation unit 420. The position estimation unit 401 estimates the orientation, position, or attitude of the image acquisition device 100, and the position estimation unit 420 estimates the location of the image acquisition device 100.

[0022] The position estimation unit 401 comprises adders 402, 405, and 406. In the following description, subtraction by an adder is performed as the addition of a negative value. The position estimation unit 401 includes a motion equation operation unit 403 for performing an operation with respect to a change in position of the image acquisition device 100, a high-pass filter 404, a delay element 408, and a rotation matrix conversion unit 407. The high-pass filter 404, which performs band limiting of a frequency band, is a filter capable of changing a cutoff frequency. The delay element 408 has a function to delay a signal by one sample or sample value of an operation cycle when a position estimation operation is performed with a predetermined cycle. A rotation matrix conversion unit 407 converts position information from a quaternion expression into a rotation matrix expression.

[0023] Furthermore, the position estimation unit 420 includes adders 409, 411, 413, 414 and 415, an acceleration operation unit 410 in world coordinates, high-pass filters 412 and 417, as well as delay elements 416 and 418. Both of the high-pass filters 412 and 417 are filters capable of changing a cutoff frequency.

[0024] First, a position estimation is described by the position estimation unit 401. The angular velocity of the image acquisition device 100 and an angular velocity bias estimate, which is calculated by the position and orientation correction unit 208 described below, are defined as follows.

[0025] ω m : An angular velocity of the image acquisition device 100, which is detected by the first vibration sensor 201 (angular velocity sensor).

[0026] b̂ g: An angular velocity bias estimate calculated by the position and attitude correction unit 208.

[0027] The adder 402 performs a bias correction using equation 1 and outputs a result relating to a change in position to the equation of motion operation unit 403. ω=ωm−b^g

[0028] A position estimate is defined as shown in Math 3 below. The delay element 408 outputs a position estimate one sample or sample value prior, which is delayed. One sample or sample value prior means that a time has elapsed by the length of one cycle (one sampling period) corresponding to a predetermined sampling frequency. q¯^

[0029] The equation of motion operation unit 403 is given a position using equation 2 based on the position estimate from a previous sample or sample value and ω from equation 1. Wcq¯^=12Ω(ω)Wcq¯^

[0030] The meanings of the symbols are as follows. q: Quaternion WCQ^: Quaternion, which represents a rotation of world coordinates (W) into camera coordinates (C) fixed to the image recording device 100.

[0031] World coordinates are fixed coordinates that specify the coordinates of an object regardless of the position of the imaging device, as described in Fig. Figure 5 is shown. To simplify the description of the present embodiment, it is assumed that a coordinate system used by the first vibration sensor 201 and the second vibration sensor 203 coincides with, or is identical to, the camera coordinates. Furthermore, Ω(ω) in Equation 2 can be calculated using Equation 3. Ω(ω)=[−⌊ω×⌋ω−ωT0]ω [ω×] is a symmetric distortion matrix

[0032] The equation of motion operation unit 403 outputs a position angle change magnitude, representing an operation result, to the high-pass filter 404. The high-pass filter 404 performs band limiting on the position angle change magnitude to eliminate a low-frequency component. The adder 405 adds an output from the high-pass filter 404 and an output from the delay element 408 (the position estimate from a previous sample) and operates on, or calculates, a position estimate at the next sample. By performing integration in this manner after eliminating a low-frequency component, it is possible to suppress drift fluctuations in a position estimate due to an angular velocity bias error, even in a state where an angular velocity bias estimation error b̂ g is estimated correctly, or the estimation error is large.

[0033] An output from adder 405 is sent to adder 406 and delay element 408, and delay element 408 outputs a delay signal to equation of motion operation unit 403 and rotation matrix conversion unit 407. A position estimate, which is an output from adder 405, is corrected in adder 406 according to the position estimate correction value described below and output as the final position estimate.

[0034] Next, a position estimation by the position estimation unit 420 is described. An acceleration of the image acquisition device 100 and an acceleration bias estimate, calculated by the position and attitude correction unit 208 described below, are defined as follows.

[0035] a m : An acceleration of the image acquisition device 100, which is detected by the second vibration sensor 203 (accelerometer).

[0036] b̂ a : An acceleration bias estimate calculated by the position and attitude correction unit 208.

[0037] The adder 409 performs a bias correction using equation 4 and outputs a result in world coordinates to the acceleration operation unit 410. a=am−b^a

[0038] A velocity estimate is given as V̂ c designated.

[0039] The delay element 416 outputs the velocity estimate from a previous sample, which is delayed. The acceleration operation unit 410 acquires the velocity estimate from a previous sample and an output from the adder 409. The acceleration operation unit 410 and the adder 411 obtain a velocity of the image acquisition device 100 using equation 5 based on a gravitational acceleration g in world coordinates and output this to the high-pass filter 417. wV^c=CWcq¯^T(a)−g CWcq¯^T: A rotation matrix, which is created by converting a position estimate value estimated by the position estimation unit 401. WCQ^ will be received.

[0040] The second vibration sensor 203, which is an accelerometer, detects acceleration in the camera coordinate system. It is necessary to convert this coordinate system to define the acceleration information in the world coordinate system for which it was originally intended to be estimated. Therefore, processing is required to convert acceleration information in the camera coordinate system into acceleration information in the world coordinate system according to a rotation matrix. CWcq¯^T performed, which represents a positional estimate of the image acquisition device.

[0041] Since the second vibration sensor 203 detects a gravitational acceleration in addition to an acceleration caused by a movement of the image acquisition device 100, the adder 411 eliminates an influence of the gravitational acceleration by subtracting the gravitational acceleration g from an output of the acceleration operation unit 410 according to equation 5.

[0042] Furthermore, in the present embodiment, measures are taken to counteract the influence of an estimation error, specifically an acceleration bias estimation error b̂. a and a position estimation error, which is estimated by the position estimation unit 401. A problem exists in that, due to these errors, a drift in a velocity estimate can occur. w V̂ CThis error can occur, which is calculated by integrating an acceleration. Therefore, the high-pass filter 412 eliminates a low-frequency component error of acceleration information in world coordinates from an output of the adder 411. An output of the high-pass filter 412 is captured by the adder 413 and added to a delay signal from a sample or sample value prior to the delay element 416. The delay element 416 captures an output of the adder 413 and outputs the delay signal to the acceleration operation unit 410.

[0043] A velocity estimate (Math 16) output by the adder 413 also has a low-frequency component, which is eliminated by the high-pass filter 417. The adder 414 adds an output from the high-pass filter 417 and a position estimate from a previous sample, which is output by the delay element 418. An output after the addition w p̂ C The data is sent to adder 415 and delay element 418. Adder 415 performs a correction by adding a position estimate correction value, described below, to an output from adder 414 and outputs a final position estimate.

[0044] As described above, a drift error can occur in a position estimation result because a position is estimated by second-order integration processing of an acceleration versus a position estimation operation if a bias error exists in the acceleration and velocity estimate. Furthermore, since a result of an estimate is used by the position estimation unit 401, if the influence of gravitational acceleration is removed from the acceleration information, a large drift due to a position estimation error can occur. Therefore, in the present embodiment, high-pass filter processing is performed in a position estimation operation before the acceleration bias estimation error b̂ is applied. aThe position is estimated, or when a large error occurs in a position estimate provided by the position estimation unit. It is possible to suppress drift in the position estimate by twice removing a low-frequency component using high-pass filters 412 and 417.

[0045] Next, a feature coordinate map estimation method is described in the feature coordinate map estimation unit 206. The feature coordinate map estimation unit 206 uses Harris vertices, SIFT features, or the like as a feature point extraction method. Furthermore, with regard to a feature point tracking method by the feature point tracking unit 209, when a square window centered on a feature point is provided and a new frame or full frame of a moving image is given, a method is used to obtain a point at which a remainder within a window between frames or full frames is smallest. A position and orientation of the image acquisition device and a position of a feature point in a field imaged by the image acquisition device are required.The recorded real space is estimated simultaneously according to a Structure From Motion (SFM) using feature point tracking information in each frame or full image obtained in this way.

[0046] Fig. Figure 4 is a schematic representation illustrating the relationship between the position and orientation of the image capture device 100 in a moving image frame, three-dimensional feature point coordinates, and feature point coordinates in a photographed image. Feature coordinates of a feature point coordinate field in three-dimensional space, encompassing depth information to be estimated, are expressed as (X, Y, Z). For a camera position and orientation in a first frame (frame 1), a camera position is expressed as O (origin), and a camera orientation is expressed as I (identity matrix). Feature coordinates on a photographic image surface or screen at that time are expressed as (u1, v1).

[0047] Furthermore, for a camera position and orientation in a second frame (frame 2), a camera position is expressed as T, a camera orientation is expressed as R, and feature coordinates on the photographic image area or screen at that time are expressed as (u2, v2).

[0048] With reference to Fig. 5, Fig. 6 to Fig. Section 7 describes a configuration of the position and attitude correction unit 208. The position and attitude correction unit 208 according to Fig. The device 7 is composed of a feature coordinate conversion unit for a photographed image 301, an adder 302, a position and location feedback gain unit 303, and a bias estimation gain unit 304. The feature coordinate conversion unit for a photographed image is hereinafter referred to simply as the coordinate conversion unit, and the position and location feedback gain unit is hereinafter referred to simply as the FB gain unit. A gain value of the FB gain unit 303 corresponds to a correction magnification or amplification for a position and location correction. Similarly, a gain value of the bias estimation gain unit 304 corresponds to a correction magnification or amplification for a bias correction. Each gain value is a variable value.

[0049] The coordinate conversion unit 301 performs processing to convert feature coordinate map information in three-dimensional space, including depth information estimated by the feature coordinate map estimation unit 206, and a position and orientation estimate by the position and orientation estimation unit 207 into feature point coordinates in a photographed image. The coordinate conversion unit 301 first converts three-dimensional feature coordinates in a world coordinate system, estimated by the feature coordinate map estimation unit 206, into three-dimensional feature coordinates in a camera coordinate system.

[0050] Fig. Figure 5 is a representation that shows a relationship between an object's coordinate position in world coordinates and its coordinate position in camera coordinates. T is a vector from an origin OW of the world coordinates to an origin OC of the camera coordinates. (rx, ry, rz) denote a unit vector that specifies a direction from each axis (x, y, z) of the camera coordinates from the perspective of the world coordinates. A point (x, y, z) in the camera coordinate system is assumed to be expressed as a point (X, Y, Z) in the world coordinate system. A relationship between these coordinates is as follows. [xyz]=R([XYZ]−T)R=[rxTryTrzT]

[0051] In equation 6, R represents a rotation matrix and T represents a parallel translation vector. R and T are calculated using the position estimation unit 207.

[0052] Next, a conversion of three-dimensional feature coordinates of the camera coordinate system into coordinates in the photographed image, which are converted by equation 6, is performed, for example by perspective conversion. Fig. Figure 6 shows a model of a perspective projection when a virtual imaging surface is fixed at a position of a focal point distance f in front of a lens. A point O in Fig. 6 represents the center of a camera lens, and a Z-axis represents an optical axis of a camera. Furthermore, a coordinate system with point O as its origin is called the camera coordinate system. (X, Y, Z) denotes the coordinate position of an object within the camera coordinate system. Image coordinates projected from the camera coordinates (X, Y, Z) of an object by perspective transformation are expressed as (x, y). An equation for transforming (X, Y, Z) into (x, y) is expressed as the following equation. x=fXY,y=fYZ

[0053] It is possible to convert a three-dimensional feature coordinate map, estimated by the feature coordinate map estimation unit 206 and encompassing a depth of an object or motif, into two-dimensional feature coordinates on a photographed image using equation 7.

[0054] The Adder 302 from Fig. Step 7 subtracts an output of the coordinate conversion unit 301 from an output of the feature point tracking unit 209. An error (a position coordinate error) is calculated between the position coordinates of feature coordinates actually observed by the feature point tracking unit 209 and two-dimensional feature coordinates obtained by conversion from a feature coordinate map of a target object or motif. The FB gain unit 303 and the bias estimation gain unit 304 multiply the position coordinate error by their respective gain values ​​and calculate a position and location estimation correction value and a bias correction value.

[0055] The position coordinate error calculated by the adder 302 is an error between feature point coordinates on a photographic image surface or screen that are actually observed and feature point coordinates obtained by projecting a three-dimensional coordinate map onto the photographic image surface or screen. Therefore, if a three-dimensional coordinate map value and a feature coordinate tracking value obtained by observation are correct, the position coordinate error is caused by an error in position and location estimation when the three-dimensional coordinate map is projected onto the photographic image surface or screen.A position and location estimation correction value and a bias estimation value (a correction value) are fed back to a result of the position and location estimation by the adders 406, 415, 402 and 409, so that this coordinate error amount becomes zero, and thereby the position and location estimation approaches its true value.

[0056] In the present embodiment, each gain value from the FB gain unit 303 and the bias estimation gain unit 304 and a frequency of a position and location estimation are changed according to a result of a determination by a vibration sensor noise determination unit 211 (hereinafter referred to simply as the noise determination unit).

[0057] With reference to Fig. 8A and Fig. Section 8B describes a relationship between noise determination by the noise determination unit 211 and a correction of a position and orientation estimate. Position and orientation estimation by the position and orientation estimation unit 207 performs operations using each output from the first vibration sensor 201 and the second vibration sensor 203. Therefore, if the bias and drift values ​​of the vibration sensors are large, or in a situation with high vibration, an error in the estimate may occur. If there is a possibility or probability that an error in the estimate may occur, the noise determination unit 211 calculates a weighting value and modifies a control input based on this value. As a result, it is possible to improve the accuracy of a position and orientation estimate.

[0058] Fig. Figure 8A is a schematic representation showing the relationship between a determined content by the noise determination unit 211 and a weighting value. The horizontal axis represents a quantity to be determined, and the vertical axis represents a weighting value. Fig. Figure 8B shows a relationship between a rating value and a correction control parameter. The horizontal axis represents a rating value, and the vertical axis represents a correction control parameter.

[0059] The noise determination unit 211 determines a noise weighting value according to the following conditions. - A case in which the outputs of the first vibration sensor are 201 and the output of the second vibration sensor is 203.

[0060] For example, if the image or viewing angle is significantly changed due to a swiveling of the image acquisition device 100, a steep position and orientation variation occurs. As a consequence of position and orientation estimation caused by this steep variation, an error may occur.

[0061] Accordingly, the noise determination unit 211 increases a rating value when outputs from the vibration sensors increase. - A case where a drift error (a variation amount) occurs in the outputs of the vibration sensors after bias correction.

[0062] Since there is a possibility or probability that a drift error may occur in a position and location estimate, the noise determination unit 211 increases a rating value when a drift error increases. - A case in which a reprojection error occurs

[0063] A reprojection error is an error between the tracking coordinates of a feature point, calculated by the adder 302 of the position and orientation correction unit 208, and a coordinate value obtained by converting a feature coordinate map into feature coordinates on a photographic image surface or screen. The noise determination unit 211 increases a rating value when the reprojection error increases. - A case in which a control amount or control speed of a control unit is large

[0064] The image acquisition device comprises a mechanism section that controls or drives the zoom unit 101, the image shake correction lens 103, the aperture / shutter unit 105, the focus lens 107, and the like. Alternatively, the mechanism section is provided in a lens or objective assembly that can be attached to a main body or housing of the image acquisition device. If the amount or rate of a drive input from a drive unit is large relative to the mechanism section, a vibration sensor detects vibration caused by mechanical drive input, and an error may occur in a position and orientation estimate. Therefore, the noise determination unit 211 increases a weighting value when the amount or rate of a drive input increases. - A case where a bias correction error occurs in the output of a vibration sensor after bias correction.

[0065] If a bias estimate deviates from its true value due to a bias correction error, there is a possibility or probability that a drift error may occur in a position and location estimate, and therefore the noise determination unit 211 increases an evaluation value when the bias correction error increases.

[0066] Based on the determination results of the aforementioned conditions, a control element is changed, as described in Fig. 8B is shown. - Control to increase the cutoff frequency of a high-pass filter when a weighting value increases.

[0067] When a weighting value increases, the cutoff frequencies of the high-pass filters 404, 412 and 417 are changed to higher frequencies than when a weighting value is low. - Control to increase a profit value (correction enlargement or amplification) when a valuation value increases.

[0068] When a valuation value increases, each gain or enhancement value of the FB gain unit 303 and the bias estimate gain unit 304 becomes larger than when a valuation value is low. - Control to increase a correction frequency when a rating value increases.

[0069] When a rating value increases, the frequency of a position and attitude correction by the position and attitude correction unit 208 and the frequency of the correction of a position and attitude estimation error caused by bias correction are set higher than when a rating value is low.

[0070] As described above, a problem exists in that an error in a position and orientation estimation due to a low-frequency component, such as low-frequency drift, can occur if a vibration sensor's noise weighting is determined to be high. In the present embodiment, measures are taken to eliminate a low-frequency component using a high-pass filter. Furthermore, if a low-frequency component is eliminated by a high-pass filter, there is a possibility or probability that a low-frequency movement of a position and orientation estimate may not be correctly detected by the position and orientation estimation unit 207.In the present embodiment, an estimation error in a low-frequency range can be corrected by increasing a gain value from the FB gain unit 303. There are other measures besides increasing a gain value for a position and location correction and a gain value for a bias estimation. For example, there is a method for shortening one estimation cycle of a feature coordinate estimation by the feature coordinate map estimation unit 206 and one operation cycle of a correction value and a bias correction value of a position and location estimation by the position and location correction unit 208. In this case, the noise determination unit 211 instructs the feature coordinate map estimation unit 206 with respect to one estimation cycle and instructs the position and location correction unit 208 with respect to one correction cycle (one operation cycle of a correction amount).Furthermore, it is possible to achieve the same effect as the method for increasing a correction gain value by increasing the frequency with which a correction is performed on a position and location estimate and a bias by the adders 402, 406, 409, and 415 at a predetermined time, namely by increasing a correction frequency. In this case, the noise determination unit 211 instructs the position and location estimation unit 207 with respect to the frequency with which a correction is performed per predetermined period.

[0071] When a gain value is increased by the FB gain unit 303 and the bias estimation gain unit 304, the correction rate for position and location increases, and it is therefore possible to bring an estimate closer to its true value sooner. However, in a case where tracking a feature point detected from an image is not possible, or where a noise component is superimposed on the tracking coordinates of a feature point, there is a possibility or probability that an error will occur in a corrected estimate. For this reason, it is desirable that the gain value is not set too high.In the present embodiment, the correction gain and the cutoff frequency of a high-pass filter are not always set high, and a control input is changed based on a determination result of a weighting value by the noise determination unit 211.

[0072] If vibration sensor noise or interference, or similar factors, are high and a rating is high, the control content is modified accordingly. As a result, it is possible to increase the accuracy of position and orientation estimation.

[0073] Referring to a flowchart according to Fig. Section 9 describes a position and orientation estimation processing of the image acquisition device 100. This includes processing of S101 to S105 and processing of S108 to S111. Fig. 9 are executed as parallel processing.

[0074] First, the imaging unit 109 in S108 converts an optical signal, generated by an optical imaging system from an object or motif, into an electrical signal and acquires an image signal. Next, the imaging signal processing unit 110 converts an analog signal into a digital signal and performs predefined image processing. The motion vector detection unit 210 calculates a motion vector based on a multitude of image data with different frame times (S109). The motion vector detection unit 210 acquires an image from a previous frame, which is stored in memory (S111), compares this image with an image from the current frame, and calculates a motion vector based on the difference between the images. Methods for detecting a motion vector include correlation, block matching, and similar techniques.In the present embodiment, any method can be used as a method for calculating a motion vector.

[0075] The feature point tracking unit 209 detects and tracks the motion position of a predetermined motion point on a photographed image based on motion vector information from a detected image in each frame at the time of photographing or recording moving images (S110). Regarding a feature point tracking technique, there is a method for obtaining a point where the smallest remainder exists in a window between frames when a square window centered on a feature point is provided and a new frame of a target image (or video) is given. Processing by this method is repeatedly executed with a cycle corresponding to the frame rate or frame frequency of an image capture of a moving image photograph or recording. After processing S110, the process proceeds to S106.

[0076] Next, the position and location estimation processing will be described with reference to S101 to S105. Fig. 9 described.

[0077] The first vibration sensor 201 detects the angular velocity of the image acquisition device 100 (S101), and the second vibration sensor 203 detects the acceleration of the image acquisition device 100 (S102). Next, bias correction of the first vibration sensor 201 and the second vibration sensor 203 is performed using a bias correction value estimated by the position and orientation correction unit 208 according to processing in a previous sampling cycle (S103). Each of the high-pass filters 404, 412, and 417, whose cutoff frequencies are set by the noise detection unit 211, outputs a band-limited signal (S104). A position and orientation estimate is calculated by the operation described above (S105).

[0078] The tracking coordinates of a feature point detected in S110 and the position and location estimate calculated in S105 are stored as a set of information in storage unit 118 (S106). In the following S107, it is determined whether a current frame number corresponds to a moving image frame number that is undergoing position and location estimate correction by the position and location correction unit 208. Whether a current frame is a moving image frame on which position and location estimate correction is performed is determined by whether as many sets of feature coordinates and position and location estimates as the number of frames required to estimate a feature coordinate characteristic field are concatenated or matched. For example, in the case of estimation processing of a in Fig. In the feature coordinate characteristic field shown in Figure 4, an estimate can be performed starting from a state in which the number of frames is two or more. Therefore, the process for processing S112 proceeds at a time when information for two frames is being / has been acquired. If the number of frames is insufficient, the process terminates. A method for shortening an estimation cycle or an operation cycle, or for increasing a correction frequency using the noise determination unit 211, as a measure distinct from the method for increasing a gain value of a position and location correction and a bias estimation gain value when a valuation value increases, has been described. In these cases, the number of frames subjected to a position and location estimation correction, as determined in S107, is set to the smaller number of frames.This is because a control is performed to shorten an operation cycle of a position and orientation estimation correction value and a bias correction value, and to increase the frequency of a correction of a position and orientation estimation value and a bias value using the adders 402, 406, 409 and 415 at a predetermined time.

[0079] After the estimation processing of a feature coordinate characteristic field in S112 is executed, a bias value is updated by the position and location correction unit 208 (S113). Then, the correction processing of a position and location estimate is performed (S114), and the processing sequence is terminated. If a negative determination (NO) is made in S107, the correction of a bias value and a position and location estimate is not performed, and the processing ends.

[0080] According to the present embodiment, the processing of a position and location estimate and a feature coordinate characteristic map estimate of an image acquisition device and a correction of a position and location estimate are performed repeatedly in each frame or full image of moving images, and therefore it is possible to obtain a correct position and location estimate. [Second embodiment]

[0081] Next, a second embodiment of the present invention is described. The same components of the present embodiment as in the first embodiment are given reference numerals that were already used in the first embodiment, and descriptions of these are omitted.

[0082] Fig. Figure 10 is a representation showing a configuration example of the position and orientation estimation unit 207 of the present embodiment. The following are differences compared to the one in Fig. 3 configurations shown. - The high-pass filters 404, 412 and 417, which perform a band limiting of a frequency band, have been removed. - In contrast to the position and location estimation of the first embodiment, high-pass filters 1001 and 1002 for change information of a position and location estimation for a feature point coordinate estimation have been added.

[0083] The high-pass filter 1001 performs filter processing on an output of the adder 406 and outputs a location estimate for a feature point coordinate estimation to the feature coordinate map estimation unit 206. The high-pass filter 1002 performs filter processing on an output of the adder 415 and outputs a position estimate for a feature point coordinate estimation to the feature coordinate map estimation unit 206. Both high-pass filters 1001 and 1002 are filters capable of changing a cutoff frequency.

[0084] Fig. Figure 11 is a flowchart describing the position and location estimation processing according to the present embodiment. The following are differences compared to previous versions. Fig. 9. - Without high-pass filter processing of S104 according to Fig. 9. After bias correction processing of S103, the process proceeds to position and location estimation processing of S105. - In S1101 between S105 and S106, the high-pass filters 1001 and 1002 each perform processing on change information of a position and location (change in the position and location estimate), which is used for a feature coordinate characteristic map estimation.

[0085] In the present embodiment, band limiting processing by a high-pass filter is not performed on an estimate of a position and orientation change at a low frequency by a vibration sensor in the position and orientation estimation unit 207. As a result, it is possible to improve the estimation accuracy of position and orientation changes that are slow at low frequencies. On the other hand, it has been found that the influence of a large deviation (in particular, a drift of a position) in a position and orientation estimate significantly impairs the estimation accuracy of a three-dimensional coordinate estimation by the feature coordinate characteristic map estimation unit 206.Therefore, a low-frequency component is eliminated by the high-pass filters 1001 and 1002 only for a change in a position and location estimate used for a three-dimensional coordinate estimation, in order to reduce and improve an error in the three-dimensional coordinate estimation caused by a drift error of a position estimate.

[0086] According to the present embodiment, it is possible to improve the estimation accuracy of a three-dimensional coordinate estimate and to improve the accuracy of a position and orientation estimate by correcting a position and orientation estimate. Even if there is a large variation or fluctuation in the error of vibration sensor information, it is possible to provide an image acquisition device capable of obtaining a position and orientation estimate with high accuracy, as well as a control method for this device.

[0087] While the present invention has been described with reference to exemplary embodiments, it is understood that the invention is not limited to these disclosed exemplary embodiments. The scope of the following claims is to be interpreted in the broadest possible way, so that all such modifications and equivalent structures and functions are included.

[0088] This application claims the benefit of Japanese patent application no. 2017-151024, filed on August 3, 2017.

[0089] A vibration sensor in an image capture device detects movement and captures shake information. An imaging unit outputs an image signal of an object to an imaging signal processing unit. A motion vector detection unit calculates a motion vector based on the image signal after imaging. A feature point tracking unit performs feature point tracking by calculating a coordinate value of an object on a photographic image area using the motion vector. A feature coordinate map and position and orientation estimation unit estimates the position or orientation of the image capture device based on information obtained by a band-limiting filter applying band-limiting to shake information from a vibration sensor and an output from the feature point tracking unit.An estimation unit assesses an estimation error and modifies a band limited by the band limiting filter based on the calculated assessment value, or modifies a correction gain at the time of a correction of the estimate according to a correction value.

Images