Information processing device and method

Abstract

The present invention relates to an information processing device and method. The distance estimation from a pitched vehicle is realized with low load and high accuracy. An information processing device for estimating distance based on an image is provided. The device comprises: an image acquisition unit, which acquires time-series images from a camera device; and a processing unit, which performs image processing. The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by shifting a reference image acquired before an object image in the time-series images in the up-down direction by a plurality of different shift amounts, and the processing unit determines the correction amount of the pitch of the camera device based on the difference between each of the shifted images and the object image.

Description

Technical Field

[0001] The present invention relates to an information processing device and method, and more particularly to an information processing device and method for determining a correction amount for a deviation caused by pitch when capturing an image. Background Art

[0002] There is a technique for estimating the distance to an object target such as a pedestrian or another vehicle in order to perform automatic driving or driving assistance of a vehicle. In order to perform automatic driving or driving assistance, the vehicle calls the attention of the driver according to the estimated distance, or automatically applies the brakes without the driver's operation. As one of the techniques for estimating the distance, there is a technique for identifying an object corresponding to an object target contained in an image captured by a camera, and estimating the distance to the object target. In this technique, the accuracy of the estimated distance may be reduced due to the shaking of the vehicle around the pitch axis (sometimes referred to as pitch), and thus a technique for preventing the reduction in accuracy has been proposed. Patent document 1 describes that, in the case of an axis offset of the pitch angle, the object detection processing area within the full field angle area of ​​each camera on the infrared image is corrected with a position correction amount corresponding to the pitch angle. In addition, in patent document 2, as a technique for estimating the pitch amount, a technique for calculating and integrating the vertical movement amount of a predetermined area in an image captured by a camera unit provided on the vehicle with a time difference, and estimating the pitch angle of the vehicle based on the integrated value is described.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2006-153778

[0006] Patent Document 2: Japanese Patent Application Publication No. 2017-20942 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, Patent Document 1 does not disclose a specific technique for determining the pitch angle of a camera. Patent Document 2 discloses a technique for determining a highly correlated region in two images and estimating the pitch angle based on the amount of movement thereof, but this technique requires determining a highly correlated region between two images, and is considered to have a high processing load.

[0009] The present invention has been made in view of the above-mentioned conventional examples, and an object of the present invention is to estimate a highly accurate correction amount by a light-load process.

[0010] Solutions for solving problems

[0011] In order to achieve the above object, the present invention has the following structure.

[0012] That is, according to one aspect of the present invention, there is provided an information processing device, characterized in that the information processing device has:

[0013] an image acquisition unit that acquires time-series images by an imaging device; and

[0014] a processing unit that performs image processing,

[0015] The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by shifting a reference image acquired before an object image in the time series images in an up-and-down direction by a plurality of different shift amounts, and the processing unit determines a correction amount for the pitch of the imaging component based on a difference between each shifted image of the plurality of shifted images and the object image.

[0016] According to another aspect of the present invention, there is provided an information processing method for use in an information processing device, characterized in that:

[0017] The information processing device comprises an image acquisition unit and a processing unit for performing image processing, wherein:

[0018] The image acquisition unit acquires time-series images by a camera component.

[0019] The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by shifting a reference image acquired before the object image in the time series images in the up-down direction by a plurality of different shift amounts, and the processing unit determines a correction amount for the pitch of the imaging component based on a difference between each shifted image of the plurality of shifted images and the object image.

[0020] Effects of the Invention

[0021] According to the present invention, a highly accurate correction amount can be estimated by light-load processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an explanatory diagram showing the structure of a vehicle system.

[0023] Figure 2 is a block diagram showing a structure for estimating distance.

[0024] Figure 3 is a block diagram showing a structure for estimating a pitch amount.

[0025] Figure 4 1 is a diagram showing an example of information processing for estimating a pitch amount.

[0026] Figure 51 is a diagram showing an example of information processing for estimating a pitch amount.

[0027] Figure 6 is a block diagram showing a structure for turning determination.

[0028] Description of Reference Numerals

[0029] 20-29: ECU; 41: Camera. DETAILED DESCRIPTION

[0030] The following embodiments are described in detail with reference to the accompanying drawings. Moreover, the following embodiments are not intended to limit the invention to which the scope of protection of the claims relates. In addition, the present invention does not require the combination of all the features described in the embodiments. It is also possible to arbitrarily combine two or more of the multiple features described in the embodiments. In addition, the same figure mark is added to the same or similar structure, and repeated descriptions are omitted.

[0031] [First embodiment]

[0032] Structure of vehicle control device

[0033] Figure 1 FIG. 1 is a block diagram of a vehicle control device according to an embodiment of the present invention, and controls a vehicle 1. Figure 1 , a vehicle 1 is schematically shown in a top view and a side view. The vehicle 1 is, as an example, a sedan-type four-wheel passenger vehicle. In addition, in this embodiment, the driving assistance provided by the vehicle control device is limited functions such as emergency braking, adaptive following driving, and lane keeping assistance, but it can also be an advanced function such as automatic driving when a destination is set.

[0034] Figure 1 The control device includes a control unit 2. The control unit 2 includes a plurality of ECUs 20 to 29 that are communicatively connected via an in-vehicle network. Each ECU includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, etc. The storage device stores programs executed by the processor, data used by the processor in processing, etc. Each ECU may also have a plurality of processors, storage devices, interfaces, etc.

[0035] The following describes the functions and the like of the ECUs 20 to 29. The number of ECUs and the functions they are responsible for can be designed appropriately, and can be further subdivided or integrated than in the present embodiment.

[0036] The ECU 20 performs control related to the automatic driving of the vehicle 1. In the automatic driving, at least any one of turning, acceleration, and deceleration of the vehicle 1 is automatically controlled.

[0037] ECU 21 controls the electric power steering device 3. The electric power steering device 3 includes a mechanism for turning the front wheels according to the driver's driving operation (turning operation) of the steering wheel 31. In addition, the electric power steering device 3 includes: a motor that generates a driving force for assisting the turning operation or automatically turning the front wheels; and a sensor for detecting the turning angle, etc. When the driving state of the vehicle 1 is automatic driving, ECU 21 automatically controls the electric power steering device 3 in accordance with the instruction from ECU 20 to control the travel direction of the vehicle 1.

[0038] The ECU 22 controls the detection unit 41 that detects the surrounding conditions of the vehicle and performs information processing on the detection results. The detection unit 41 is a camera that captures the front of the vehicle 1 (hereinafter, sometimes described as the camera 41). In the case of this embodiment, a camera is fixedly provided on the front part of the roof of the vehicle 1. The ECU 22 analyzes the image captured by the camera 41, thereby extracting the outline of the object target and the dividing line (white line, etc.) of the lane on the road. In addition, the distance to the object target in the image can be estimated.

[0039] ECU 24 controls GPS sensor 24b and communication device 24c, and performs information processing on detection results or communication results. GPS sensor 24b detects the current position of vehicle 1. Communication device 24c can wirelessly communicate with a server that provides map information and traffic information to obtain such information. ECU 24 can access database 24a of map information constructed in a storage device, and ECU 24 performs a route search from the current position to the destination, etc.

[0040] The ECU 26 controls the powertrain 6. The powertrain 6 is a mechanism that outputs a driving force for rotating the driving wheels of the vehicle 1, and includes, for example, an engine and a transmission. The ECU 26 controls the output of the engine in accordance with the driver's driving operation (accelerator operation or acceleration operation) detected by the operation detection sensor 7a provided on the accelerator pedal 7A, or switches the gear stage of the transmission based on information such as the vehicle speed detected by the wheel speed sensor 7c.

[0041] ECU 28 controls the input-output device 9. The input-output device 9 outputs information to the driver and receives information input from the driver. The sound output device 91 notifies the driver of information by sound. The display device 92 notifies the driver of information by displaying an image. The display device 92 is, for example, arranged in front of the driver's seat to constitute a dashboard, etc. Moreover, here, sound and display are illustrated, but information can also be notified by vibration or light. The input device 93 is arranged at a position where the driver can operate, and can be a switch group or a touch panel for instructing the vehicle 1, or can also include a sound input device.

[0042] ECU 29 controls the brake device 10 and the parking brake (not shown). The brake device 10 is, for example, a disc brake device, which is provided on each wheel of the vehicle 1 and decelerates or stops the vehicle 1 by applying resistance to the rotation of the wheel. ECU 29 controls the operation of the brake device 10 in accordance with the driver's driving operation (brake operation) detected by the operation detection sensor 7b provided on the brake pedal 7B. For example, when the distance to the object target detected from the image is less than a predetermined value, or when the time until the estimated collision is less than a predetermined value, ECU 29 can automatically control the brake device 10 in accordance with the instruction from ECU 20 to control the deceleration and / or stop of the vehicle 1. The brake device 10 and the parking brake can also be operated to maintain the stopped state of the vehicle 1. In addition, when the transmission of the powertrain 6 is equipped with a parking lock mechanism, it can also be operated to maintain the stopped state of the vehicle 1.

[0043] Image-based distance estimation function

[0044] Figure 2 (A) is a block diagram of the distance estimation function implemented by ECU 20. This structure can be implemented by a hardware circuit included in ECU 20, or by a CPU included therein executing a program recorded in a memory. In addition, it can also be implemented by a combination of them. This is also the same for other ECUs. Regardless of the structure, ECU 20 or other ECUs function as an information processing device, an image processing device, or a signal processing device.

[0045] The camera 41 is an imaging device such as a monocular camera, and captures images at a predetermined frame rate. The captured images are composed of frames separated by a predetermined time interval, and each frame is referred to as an image. The captured images are sent to the object detection unit 201, the correction amount estimation unit 203, and the turn determination unit 205 at a predetermined time interval. The predetermined time interval can be the inverse of the frame rate or an integer multiple thereof.

[0046] The object detection unit 201 detects an object corresponding to an object target such as a person or an object as a subject contained in the image. Information related to the detected object is sent to the distance estimation unit 202 as detected object information. The correction amount estimation unit 203 estimates the correction amount of the offset caused by the pitch (also referred to as the pitch offset) based on the latest image and the previous image. In addition, in the present embodiment, the estimated correction amount is the correction amount d' of the image, which is input to the correction amount selection unit 204. The turn determination unit 205 determines whether it is turning based on the latest image and the previous image, and inputs the result as a turn flag to the correction amount selection unit 204.

[0047] When the turn sign indicates that the vehicle is not turning, the correction amount selection unit 204 outputs the offset correction amount d', and when the turn sign indicates that the vehicle is turning, the correction amount selection unit 204 outputs "0" as the offset correction amount. The distance estimation unit 202 estimates the distance to the detected object target included in the image based on the image and the offset correction amount.

[0048] The distance estimation unit 202 may also estimate the distance as follows. The ground height of the camera 41 fixed to the vehicle 1 in a fixed or adjustable manner is set as H0, and the distance L0 from the camera 41 toward the shooting direction (optical axis direction) is set as the reference distance. In addition, the position in the height direction of the image corresponding to the height 0 (i.e., the ground) and the reference distance L0 in the captured image captured by the camera 41 is set as the reference position. The position in the height direction can be determined, for example, by the distance (in units such as the number of pixels) from the upper or lower side of the captured rectangular image. Conversely, the reference position in the image can also be determined, and the distance to the position corresponding to the reference position at the height 0 (i.e., the ground) is used as the reference distance. As a specific example, the bottom side of the image can be used as the reference position, or the position offset by a predetermined number of pixels from the bottom side can be used as the reference position. In addition, in the image obtained by capturing the object target at the reference distance L0, the height of the object target in the image corresponding to each pixel is set to k. Equivalent to each pixel can refer to the pixel pitch. At this time, the distance L from the camera 41 to the object target corresponding to the object at a height h (number of pixels) from the reference position in the image is expressed as L=L0·(H0 / (H0-h·k)). When the length from the camera 41 to the front end of the vehicle 1 is set to Lf, the distance D from the front end of the vehicle to the object target at a height k reflected in the image can be estimated as D=L0·(H0 / (H0-h·k))-Lf. Moreover, in the above formula, it is assumed that the optical axis of the camera 41 is horizontal. Even if, for example, the camera 41 has a depression angle but it is small, it can be approximately calculated by the above formula. Of course, this is just an example, and other methods can also be used, such as a more accurate method that takes into account the depression angle of the camera, and other methods can also be used. Regardless of which method is used, the distance is estimated based on the height position of the object target in the image.

[0049] Here, in the example described above, the pitch angle variation is not corrected. When correction is performed, the offset correction amount d' is added to (or subtracted from) the height of the object in the image. That is, using the above formula, the corrected distance L' becomes L'=L0·(H0 / (H0-(h+d')·k)), and the corrected distance D' becomes D'=L0·(H0 / (H0-(h+d')·k))-Lf.

[0050] Figure 2 (B) is a diagram showing an example of estimating the distance based on the captured image. In the image 210, the reference position is the bottom of the image, that is, the position of height 0. The object detection unit 201 detects the object target obtained by machine learning from the image. Figure 2 In (B), pedestrians 211 and bicycles 212 are detected, and the area of ​​the detected object targets is determined. This is equivalent to the detected object information. In the distance estimation unit 202, the distance from here to each object target is estimated. According to the above method, the offset correction amount is d, and the distance Ls to the bicycle 212 at the height Hs is Ls=L0·(H0 / (H0-(Hs+d')·k))-Lf. Similarly, the distance Lp to the pedestrian 211 at the height Hp is obtained by Lp=L0·(H0 / (H0-(Hp+d')·k))-Lf.

[0051] Estimate the correction amount due to pitch deviation

[0052] exist Figure 3 The details of the correction amount estimation unit 203 are shown in FIG. The image input from the camera 41 is input to the grayscale processing unit 301 and converted into a grayscale image. The image to be processed input here is image k. The suffix k indicates the order of the images, and the previous image of image k is image k-1. The previous image is not limited to the previous frame of image k captured by the camera 41, but can also be a frame separated by a predetermined number of frames. The grayscale image k is sent to the image cutout processing unit 302 and the image buffer 311.

[0053] The image buffer 311 is a buffer for holding the image k-1 that is the previous image k. The image k-1 is output from the image buffer 311 and input to the own vehicle movement amount correction unit 312. The own vehicle movement amount correction unit 312 receives the vehicle speed detected by the wheel speed sensor 7c, the vehicle speed calculated based on the position information, etc. as another input, and corrects the image k-1 based on the vehicle speed and the time difference between the image k and the image k-1.

[0054] exist Figure 4 An example of correction by the vehicle movement amount correction unit 312 is shown in FIG. Image 400 is image k-1 before correction. Image 420 is image k to be processed. Image 400 is an image captured at a time interval earlier than image 420, and is a reduced image compared to image 420. In order to obtain the difference between image k-1 and image k in subsequent processing, the vehicle movement amount correction unit 312 corrects the field of view angle of image k-1 with reference to image k. Figure 4In the example, since the region 401 can be determined as the shooting range of the image k based on the speed v and the time interval ti, when the region 401 is cut out and enlarged to the original size, the image 410 is obtained. Thus, the field of view of the image k-1 can be virtually made consistent with that of the image k.

[0055] The image k-1 corrected by the vehicle movement correction unit 312 is subjected to N types of shift processing by the up-down shift processing unit 313. The N types of shift processing mean that, for example, s is a constant, and for the nth image k-1(n), the shift amount px(n) is determined to be (n-1)·sf (here n=1, ..., N). In addition, the image k-1(n) can also be shifted upward or downward according to the shift amount px(n) (in this example, it is set to upward for the sake of explanation). sf is a pixel that serves as a unit of shift. Specifically, for example, sf=1, N=30, etc. In this case, 30 types of shift amounts of 0, 1, ..., 28, and 29 are determined. As a result, based on the image k-1, N images k-1(1), k-1(2), ..., k-1(N) shifted according to the shift amount are generated.

[0056] N images k-1 are input to the image cutout processing unit 314. In the image cutout processing unit 314, for the N images k-1(n), pixel columns corresponding to the maximum shift amount (in this example, (N-1)·sf) are cut out from the edge of the image on the shift direction side (the upper edge in this example). Through the shift processing, n·sf rows of pixel columns are discarded from the edge on the shift direction side, and the edge on the opposite side is filled with pixel columns that were originally not there (for example, white pixel columns). Through the cutout processing, (N-1)·sf rows of pixel columns including the filled white pixel columns are cut out from the edge on the opposite side of the shift direction. In the image k-1(N) obtained by performing the maximum shift amount, i.e., the shift equivalent to (N-1)·sf pixels, pixel columns equivalent to (N-1)·sf rows are discarded from the edge on the shift direction side of the original image k-1. This is the maximum number of pixel columns to be discarded from the original image k-1, so the other images k-1(n) are cropped in the same manner as the image k-1(N). That is, after the shift process with the shift amount set to (n-1)·sf, the pixel columns of (N-1)·sf rows are cropped from the side on the opposite side of the shift direction. In this way, the size of the image k-1(n) can be unified.

[0057] In the image cutout processing unit 302, image k is cropped to the same size as image k-1(n) cut out by the image cutout processing unit 314. Since it is unclear whether the offset between image k and image k-1 is in the up or down direction, in order to be able to detect in either direction, the same shifting and cropping as image k-1 ((N-1) / 2) shifted by an intermediate shift amount is implemented. In addition, the decimal part of (N-1) / 2 is rounded off, for example, a carry may be performed. Image k-1 ((N-1) / 2) is an image obtained by discarding ((N-1) / 2)·sf rows of pixels from the top of the original image k-1 through a shifting process and by cutting out (N-(N-1) / 2)·sf rows of pixels from the bottom through a cutout process. Therefore, the image cutting processing unit 302 also cuts out the pixel columns of ((N-1) / 2)·sf rows from the upper side of the original image k, and cuts out the pixel columns of (N-(N-1) / 2)·sf rows from the lower side. In this way, the size of the image k as the target image can be matched with the image k-1 as the comparison target of the reference image. In addition, if the pitch amount is 0, the image k can be cut out in a manner that the image k is consistent with (or approximately consistent with) the image k-1 ((N-1) / 2).

[0058] Image k is input from the image cutout processing unit 302 to the image difference processing unit 303, and N images k-1(n) are input from the image cutout processing unit 314 to the image difference processing unit 303. The image difference processing unit 303 obtains the difference between image k and each image k-1(n). In the image difference processing unit 303, synchronization is achieved between the input image k and image k-1(n) as needed. When the time when image k-1 is input to the correction amount estimation unit 203 is used as a reference, the processing time of the vehicle movement amount correction unit 312, the vertical shift processing unit 313, and the image cutout processing unit 314 is required for image k-1(n) until the image difference processing unit 303. On the other hand, the time difference (for example, frame interval) between image k-1(n) and image k and the processing time of the image cutout processing unit 302 are required for image k. In the image difference processing unit 303, synchronization can be achieved by keeping the shorter one.

[0059] The difference is the difference in the values ​​of corresponding pixels. For identical pixels, the difference is 0, and the closer they are, the smaller the value. An image composed of such difference values ​​is also called a difference image. Differential images are obtained for each of the N images k-1(n), thereby generating N differential images (n=1, ..., N). In the image binarization processing unit 304, the N differential images are binarized. Binarization can be performed, for example, in the following manner: a predetermined threshold is compared with each pixel value, and if it is above the threshold, it is set to 1, and if it is less than the threshold, it is set to 0. As a threshold, for example, the middle value of the number of tones of image k and image k-1(n) can be used. Through binarization, if the difference between a pixel of image k and the corresponding pixel of image k-1(n) is large, the pixel value is 1, and if the difference between the corresponding pixels is small, the pixel value is 0.

[0060] Finally, the offset correction amount estimation processing unit 305 estimates (or determines) the offset correction amount. Therefore, first, for each of the N binarized difference images, the total pixel value S(n) is calculated. The total pixel value S(n) represents the number of pixels with a value of 1, that is, the total number of pixels with a large difference between image k and image k-1(n). Image k-1(n) with a smaller pixel total S(n) is an image more similar to image k. Therefore, in the first method, the shift amount px(n)=(n-1)·sf corresponding to image k-1(n) with the smallest pixel total S(n) is directly set as the offset correction amount d. In the second method, the pixel total S is regarded as a function of the shift amount px, and a quadratic function with px as a variable that approximates the distribution of S(n) is determined. Then, px that gives the minimum value of the quadratic function is set as the offset correction amount d. In the latter method, the offset correction amount can be estimated based on a shift amount that has not actually been tried, so the shift amount can be determined with a larger step size. Regarding the second method, in the following Figure 5 This is described in more detail in the description of .

[0061] Details of offset correction estimation

[0062] exist Figure 5Schematic diagram of the processing from the image cutting processing unit 314, 302 to the offset correction amount estimation processing unit 305 is shown in FIG. Here, images k-1(1) to k-1(5) (i.e., N=5) are shown, which are shifted from image k-1 by 5 shift amounts. Images 511 to 515, i.e., images k-1(n) are shown, which are shifted and cut out from image 521, which is image k-1. For example, the shift amount of image 511 is 0. Pixel columns of the number of rows corresponding to the maximum shift amount are cut out from the bottom. In addition, the shift amount of image 515 is the largest, and pixel columns of the number of rows corresponding to the maximum shift amount are cut out from the bottom. Here, the upper area of ​​image 515 is the area that is discarded by the shift processing, and the lower side of image 515 is filled with white pixel columns of the number of rows corresponding to the maximum shift amount by the shift processing, although not shown. In image 515, the filled area is discarded. The same applies to the other images 512 to 514 .

[0063] In addition, as described above, an image 501 is cut out from the image 520 corresponding to the image k in the same manner as the image k-1 ((N-1) / 2) with the intermediate shift amount relative to the image k-1. The image 501 is input to one input of all the differentiators 530 included in the image difference processing unit 303, and the images 511 to 515 are input to the other input of each differentiator 530. Therefore, the difference of the corresponding pixel values ​​is obtained. Moreover, the difference is referenced as a measure of the size of the difference, and the difference thus obtained can be an absolute value. Therefore, when the pixel value does not have a normal sign bit, for example, a positive sign bit is added to the pixel value to calculate the difference, and when the resultant sign bit is inverted, the 2(10b) complement is obtained to convert it to an absolute value.

[0064] The difference image formed by the difference between the corresponding pixels is input from the differentiator 530 to the binarizer 540 included in the image binarization processing unit 304. In the binarizer 540, each difference image is binarized with a predetermined threshold. The threshold value can be, for example, 1 / 2 of the number of tones, or a smaller value of 1 / 4 or 1 / 8. Of course, it can also be a smaller value.

[0065] The binarized differential image is input to the pixel summation unit 550 of the offset correction amount estimation processing unit 305, and the sum of the pixel values ​​of each differential image is calculated. The sum represents the number of pixels where there is a difference of more than a threshold value in the corresponding pixel values ​​between image k (also called the object image) and image k-1 (n). That is, ideally, if they are completely consistent, the pixel values ​​of the differential image are all 0 and the sum is 0. At this time, the displacement amount of image k-1 (n) can be determined as the displacement caused by pitch. In this embodiment, the displacement amount of image k-1, which has the smallest sum of the pixel values ​​of the differential image, is determined as the offset.

[0066] Therefore, the sum value output by the pixel summation unit 550 is input to the estimation unit 560 together with the shift amount px(n) (=(n-1)·sf) of the corresponding image k-1(n). The following describes a method (the second method) in which the estimation unit 560 determines an approximate quadratic function S_hat(n) based on these values ​​and estimates the offset amount using this function.

[0067] Determine the approximate quadratic function

[0068] S_hat(n) is given by the following formula.

[0069] S_hat(n)=a(n-1)Px 2 (n)+b(n-1)Px(n)+c(n-1) (1)

[0070] =Φ T (n-1)ζ(n)

[0071] Here, n corresponds to n of image k-1(n), and is referred to as an image number (n=1 to N).

[0072] Here, the model parameters a, b, and c of S_hat represented by equation (1) are calculated by the following equation. The following equation is implemented in the control loop by the number of shift images, that is, N times. First, determine

[0073] Φ T (n) = [a(n) b(n) c(n)] (2)

[0074] ζ T (n) = [Px 2 (n) Px(n) 1] (3)

[0075] These two matrices. That is, equation (1) is the matrix Φ of equation (2) T The product of (n) and the matrix ζ(n) expressed by the transposition of equation (3) is given. Φ(n) is the coefficient matrix to be found, and the matrix ζ(n) is a matrix of constants determined by the shift amount px(n). In addition, the error e(n) is determined by equation (4).

[0076] e(n)=S(n)-S_hat(n) (4)

[0077] =S(n)-Φ T (n-1)ζ(n)

[0078] Here, S(n) is the sum of the differential pixel values ​​corresponding to image k-1(n). That is, the error e(n) represents the error between the observed value and the value given by the approximation.

[0079] Here, the coefficient matrix Φ(n) is determined by the following formula.

[0080] Φ(n)=ΔΦ(n-1)+KP(n)e(n) (5)

[0081] KP(n)=Pζ(n) / (1+ζ T (n)Pζ(n)) (6)

[0082] In equation (5), Φ(n-1) is a coefficient matrix for image number (n-1). Δ is a weight to be multiplied therewith. e(n) is an error expressed by equation 4. KP(n) is determined by equation (6), where P is an adaptive gain (P>0).

[0083] Repeat the above steps from n = 1 to n = N to determine Φ T (N) = [a(N)b(N)c(N)] This is the coefficient matrix of the quadratic function obtained, so S_hat(N) = a(N)Px 2 The value d(N) of px at the minimum value (= local minimum value) of +b(N)Px+c(N) is

[0084] d(N)=-b(N) / (2a(N)) (7).

[0085] Here, the initial value of Φ(n) may be zero. Alternatively, it may be set to the calculated value of the previous control cycle. That is, Φ(N) obtained when estimating the previous offset may be used as the initial value when estimating the next offset. When the calculated value of the previous control cycle is used as the initial value, high-frequency changes between control cycles can be removed, and inappropriate high-frequency changes can be removed from the pitch correction value.

[0086] Alternatively, for example, the coefficients of the quadratic function S_hat2 may be determined by the least squares method. In this case, the coefficients from the quadratic term to the constant term are set to a, b, and c, and the coefficients are determined so that the sum of square deviations between the sum of pixel values ​​of the difference image actually obtained and the sum of pixel values ​​derived from the quadratic function S_hat2 is minimized. Thus, for example,

[0087] 2Σ{S(n)-(a·px(n) 2 +b·px(n)+c)}(-px(n) 2 )=0

[0088] 2Σ{S(n)-(a·px(n) 2 +b·px(n)+c)}(-px(n))=0

[0089] 2Σ{S(n)-(a·px(n) 2 +b·px(n)+c)}(-1)=0

[0090] a, b, c are determined as coefficients of the quadratic function S_hat2. Here, Σ represents the sum of n=1 to N. If the coefficients are determined, the shift amount d that gives the minimum value can be determined as d=-b / (2a). As described above, the correction amount estimation unit 203 can estimate the offset correction amount d based on the image k-1 and the image k.

[0091] Here, the offset correction amount determined according to the above-mentioned method is a correction amount obtained by using image k-1 as the reference image. In fact, image k-1 may also have an offset in the up-down direction due to pitch relative to image k-2. This is also the same for earlier images, and the offset may be accumulated. Therefore, the offset correction amount d should be determined by accumulating past values ​​in principle. However, when the offset is accumulated, the error is also accumulated, which may reduce the accuracy of the estimate. Therefore, in the offset correction amount estimation processing unit 305, the accumulation is performed on the basis of multiplying the accumulated past estimated values ​​by a coefficient λ such as 0<λ<1 (which is called the forgetting coefficient). For example, the estimated value of the latest offset correction amount to be obtained is set to d', the estimated value of the offset correction amount obtained only based on the difference between image k and image k-1 is set to d, and the estimated value of the previous offset correction amount obtained using image k-1 as the object image is set to d'. k-1 . Set the forgetting coefficient to λ, and

[0092] d'=d+λd' k-1

[0093] As the output of the correction amount estimation unit 203. By setting the forgetting coefficient, it is possible to prevent the accumulation of errors and the reduction in estimation accuracy caused by the accumulation of errors.

[0094] Turn determination processing

[0095] Next, the determination of a turn by the turn determination unit 205 will be described. Figure 6(A) shows an overview of the turn determination unit 205. When image k is input, it is grayscaled by the grayscale processing unit 601 and passed to the turn determination processing unit 602 and the image buffer 603. Image k and image k-1 are synchronously input to the turn determination processing unit 602 to perform turn determination processing. The turn determination processing calculates the difference between the corresponding pixels of the two images, generates a differential image, and compares the sum of the pixel values ​​with the threshold. If the result of the comparison is that the sum is greater than the threshold (or is above the threshold), it is determined that there may be a turn. Then, if it is continuously determined that there may be a turn for both image k and image k+1, a turn flag is set. If it is not determined that there is a turn, the turn flag is reset.

[0096] In this example, since neither the image k-1 is enlarged nor the difference image is binarized, the sum of the difference images is somewhat large even when there is no turn, so a large value can be selected as the threshold. Alternatively, similarly to the estimation of the offset correction amount, Figure 4 In this way, the image k-1 is enlarged and corrected. Alternatively, the difference image may be binarized and the sum of the binarized pixel values ​​may be calculated.

[0097] As already described, if the turn flag is set, the correction amount selection unit 204 uses 0 as the offset correction amount, and if not set, the correction amount d' input from the correction amount estimation unit 203 is input to the distance estimation unit 202. In the distance estimation unit 202, the distance L'=L0·(H0 / (H0-(h+d')·k)) from the camera 41 to the object target at the height h in the image is estimated.

[0098] According to the above structure, the distance to the object target can be estimated on the basis of correcting the offset of the image in the up-down direction caused by the pitch of the vehicle. By correcting the offset caused by the pitch, the estimation accuracy can be further improved. In addition, by using the forgetting coefficient to accumulate past estimation results in a limited manner, the accuracy can be further improved. In addition, by correcting the difference in the field of view angle between the reference image k-1 and the object image k based on the shooting interval and speed, the estimation accuracy can be further improved. Moreover, in the case of determining that the vehicle is turning, by setting the offset correction amount to 0, erroneous estimation accompanying the turn can be prevented.

[0099] [Second Embodiment]

[0100] In the second embodiment, the correction amount estimation unit 203 uses the offset correction amount d' for shifting the image k instead of estimating the distance. In this case, the distance estimation unit 202 shifts the image k in the up-down direction by the offset correction amount d' instead of calculating the distance L' using the offset correction amount d'. For example, if the sign of the offset correction amount d' is positive, the image is shifted in the upward direction, and if it is negative, the image is shifted in the downward direction. The distance L' is calculated as L'=L0·(H0 / (H0-h'·k)). Here, h' represents the height of the object target in the corrected image k from the reference position. As described above, the same effect as the first embodiment is achieved.

[0101] [Third Embodiment]

[0102] In the third embodiment, the correction amount estimation unit 203 outputs a pitch correction amount θ instead of the offset correction amount d′. The distance estimation unit 202 corrects the offset of the estimated distance due to the pitch using the pitch correction amount θ.

[0103] exist Figure 6 (B) shows a schematic diagram for estimating the pitch correction amount. If the offset correction amount d is determined based on only the image k and the image k-1 in the manner of the first embodiment, the pitch correction amount θ is determined based on this. Figure 6 As shown in (B), the pitch correction amount is θ, the vertical offset (offset correction amount) of image k relative to image k-1 is d, and the field of view angle is α. Other quantities are also given as shown in the figure. In this case, θ is

[0104] tan((α / 2)+θ)=((h / 2)+d) / l. ​​Here, further,

[0105] tan(α / 2)=(h / 2) / l,

[0106] l = (h / 2) / tan(α / 2). Therefore,

[0107] tan((α / 2)+θ)=((h / 2)+d) / ((h / 2) / tan(α / 2)). h may be, for example, the number of pixels. In this formula, since the other than θ are constants, θ can be determined using the addition theorem or the like.

[0108] The thus determined θ can be multiplied by the forgetting coefficient λ and then accumulated in the same manner as in the first embodiment to obtain the pitch correction amount θ′=θ+λθ′ k-1. Moreover, the pitch correction amount θ' is input to the distance estimation unit 202 via the correction amount selection unit 204 and is referred to in the distance estimation process. Therefore, the steps opposite to the above steps can be performed to obtain the offset correction amount d' for distance estimation calculation in the same way as the first embodiment or the second embodiment. As a result, the same effect as the first embodiment is produced.

[0109] In addition, the estimated pitch correction amount can be used not only to estimate the distance, but also for other functions. For example, the illumination direction of the headlight can be rotated in a manner opposite to the pitch correction amount, or in the case of a radar, the illumination direction can be shifted in a manner opposite to the pitch correction amount. This makes it possible to set appropriate lighting and an appropriate radar search range.

[0110] Summary of implementation methods

[0111] The above-mentioned embodiments disclose at least the following information processing devices.

[0112] 1. The information processing device of the above embodiment has:

[0113] an image acquisition unit that acquires time-series images by an imaging device; and

[0114] a processing unit that performs image processing,

[0115] The processing unit generates a plurality of shift images, wherein the plurality of shift images are obtained by shifting a reference image acquired before an object image in the time series images in an up-and-down direction by a plurality of different shift amounts, and the processing unit determines a correction amount for the pitch of the imaging component based on a difference between each shift image of the plurality of shift images and the object image.

[0116] According to this configuration, a highly accurate correction amount can be estimated with a lighter load than in the related art.

[0117] 2. In the information processing device of the above embodiment, the processing unit enlarges the reference image based on the time difference from when the reference image is captured to when the reference image is to be captured and the moving speed of the camera component, and determines the correction amount using the enlarged reference image.

[0118] According to this configuration, it is possible to eliminate the influence of the movement of the imaging component on the estimation result.

[0119] 3. In the information processing device of the above-mentioned embodiment, the processing unit calculates the number of corresponding pixels that are different from each other between images to differ between each of the plurality of shifted images and the object image, and based on the correlation between the shift amount of each of the plurality of shifted images and the number of pixels, determines a quadratic function that gives the number of pixels with the shift amount as a variable, and determines the correction amount based on the shift amount that gives the minimum value of the quadratic function.

[0120] According to this configuration, interpolation can be performed using a quadratic function, and a correction amount with higher accuracy can be estimated.

[0121] 4. In the information processing device of the above embodiment, the processing unit further determines the correction amount by accumulating correction amounts obtained based on a plurality of target images in time series, and when accumulating, multiplies the correction amount determined for past target images by a forgetting coefficient.

[0122] According to this configuration, it is possible to suppress accumulation of errors and estimate a correction amount with higher accuracy.

[0123] 5. In the information processing device of the above embodiment, the processing unit determines the lateral movement of the imaging device based on the difference between the target image and the reference image, and sets the correction amount to zero when the lateral movement is determined.

[0124] According to this configuration, it is possible to prevent movement in the lateral direction from being misinterpreted as displacement due to pitching.

[0125] 6. In the information processing device of the above embodiment, the processing unit determines the distance to an object target corresponding to the object included in the target image based on the height to the object in the target image and the correction amount.

[0126] According to this configuration, it is possible to estimate the distance in which the offset due to pitch is corrected.

[0127] 7. In the information processing device of the above embodiment, the imaging unit is fixed to a vehicle.

[0128] This makes it possible to estimate the distance from the pitching vehicle to the target object with high accuracy.

[0129] 8. The information processing method of the above embodiment is used in an information processing device, wherein the information processing device has an image acquisition unit and a processing unit for performing image processing, wherein:

[0130] The image acquisition unit acquires time-series images by a camera component.

[0131] The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by shifting a reference image acquired before the object image in the time series of images in the up-down direction by a plurality of different shift amounts, and the processing unit determines a correction amount based on the pitch of the camera component according to a difference between each shifted image of the plurality of shifted images and the object image.

[0132] According to this configuration, it is possible to provide an information processing method capable of estimating a highly accurate correction amount with a lighter load than in the related art.

[0133] The invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. An information processing device, characterized in that: have: An image acquisition unit that acquires time-series images using a camera component fixed to the vehicle; as well as a processing unit that performs image processing, The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by amplifying a reference image acquired before the object image in the time series of images based on a time difference from capturing the reference image to capturing the object image and a speed at which the imaging component moves so that a field of view angle of a capturing range of the object image included in the reference image is consistent with a field of view angle of the object image, and the amplified reference image is shifted in the up and down directions by a plurality of different shift amounts, and the processing unit determines a correction amount for the pitch of the imaging component based on a difference between each shifted image of the plurality of shifted images and the object image.

2. The information processing device according to claim 1, characterized in that The processing unit calculates the number of corresponding pixels that are different from each other between images as the difference between each shifted image of the multiple shifted images and the object image, and based on the correlation between the shift amount of each of the multiple shifted images and the number of pixels, determines a quadratic function that gives the number of pixels with the shift amount as a variable, and determines the correction amount based on the shift amount that gives the minimum value of the quadratic function.

3. The information processing device according to claim 1, characterized in that The processing unit further determines the correction amount by accumulating correction amounts obtained based on a plurality of target images in time series, and multiplies the correction amount determined for a past target image by a forgetting coefficient when accumulating.

4. The information processing device according to claim 1, characterized in that The processing unit determines the movement of the imaging unit in the lateral direction based on the difference between the target image and the reference image, and sets the correction amount to zero when the movement in the lateral direction is determined.

5. The information processing device according to claim 1, characterized in that The processing unit determines a distance to an object target corresponding to the object included in the target image based on a height to the object in the target image and the correction amount.

6. An information processing method, used in an information processing device, characterized in that: The information processing device comprises an image acquisition unit and a processing unit for performing image processing, wherein: The image acquisition unit acquires time-series images by a camera component fixed to the vehicle. The processing unit generates a plurality of shifted images, wherein the plurality of shifted images are obtained by amplifying a reference image acquired before the object image in the time series of images based on a time difference from capturing the reference image to capturing the object image and a speed at which the imaging component moves so that a field of view angle of a capturing range of the object image included in the reference image is consistent with a field of view angle of the object image, and the amplified reference image is shifted in the up and down directions by a plurality of different shift amounts, and the processing unit determines a correction amount for the pitch of the imaging component based on a difference between each shifted image of the plurality of shifted images and the object image.

Images