Vehicle condition estimation device and vehicle condition estimation method

Abstract

To provide a technology that enables more accurate estimation of the vehicle's state, i.e., its attitude and motion, than conventional methods. [Solution] The vehicle state estimation device (8) is configured to estimate the vehicle state, which is the attitude and / or motion state of the vehicle, based on images captured by a camera mounted on the vehicle. The vehicle state estimation device comprises an image acquisition unit (801) and an estimation unit (802). The image acquisition unit acquires stereo images of stationary objects above the road surface (R) around the vehicle in a first and second region, which are at least two regions from the front, side, and rear regions of the vehicle. The estimation unit estimates the vehicle state based on the stereo images acquired by the image acquisition unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a vehicle state estimation device and a vehicle state estimation method for estimating a host vehicle state that is a posture and / or a motion state of the subject vehicle, on the basis of an image captured by a camera mounted on the vehicle.

BACKGROUND ART

[0002] A technique for estimating the position and motion of an object from a stereo image is known. For example, an object motion estimation device described in Patent Document 1 includes a stereo camera and a controller.

[0003] The stereo camera is composed of two cameras, a left stereo camera and a right stereo camera. The left stereo camera is mounted on the front left side of the host vehicle, and the right stereo camera is mounted on the front right side of the host vehicle. These stereo cameras may be arranged such that their respective fields of view overlap along the direction orthogonal to the imaging direction.

[0004] The controller includes a host vehicle motion estimation unit. The host vehicle motion estimation unit estimates the host vehicle motion, that is, the three-axis translational motion and three-axis rotational motion of the host vehicle. The three-axis translational motion is motion in the longitudinal direction, the vehicle width direction, and the vertical direction of the host vehicle. The three-axis rotational motion is rotational motion around three axes including a roll axis, a pitch axis, and a yaw axis. The host vehicle motion estimation unit can estimate the host vehicle motion from the optical flow and three-dimensional coordinates of feature points.

PRIOR ART DOCUMENTS

PATENT DOCUMENTS

[0005]

PATENT DOCUMENT 1

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] Vehicle motion control systems such as ABS and VSC are triggered by the detection or estimation of the vehicle's current attitude and motion state. ABS stands for Anti-lock Braking System, and VSC stands for Vehicle Stability Control. Therefore, technological development is progressing to pursue how to accurately grasp the current state of the vehicle, i.e., its attitude and motion state.

[0007] In this regard, in the technology described in Patent Document 1, the left and right stereo cameras are mounted in front of the vehicle. Therefore, for example, if only the rear tires slip, the displacement in the stereo cameras at the front is small, making it difficult to accurately detect the state of the vehicle.

[0008] This disclosure is made in view of the circumstances exemplified above. Specifically, this disclosure provides, for example, a technology that enables more accurate estimation of the vehicle's state, i.e., its attitude and motion, than in the past. [Means for solving the problem]

[0009] In one aspect of this disclosure, a vehicle state estimation device (800) estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle (V), An image acquisition unit (801) acquires stereo images of stationary objects above the road surface (R) surrounding the vehicle in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. An estimation unit (802) estimates the state of the vehicle based on the stereo image acquired by the image acquisition unit, It is equipped with. In another aspect of this disclosure, a vehicle state estimation method estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle (V). Stereo images of stationary objects above the road surface (R) surrounding the vehicle are acquired in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. Based on the acquired stereo image, the vehicle's state is estimated.

[0010] In addition, each element in the application documents may be denoted by a reference numeral in parentheses. However, such reference numerals merely indicate one example of the correspondence between the element and the specific means described in the embodiments below. Therefore, this disclosure is not limited in any way by the notation of the above reference numerals. [Brief explanation of the drawing]

[0011] [Figure 1] This is a side view showing the vehicle in motion, which is the subject of this disclosure. [Figure 2] Figure 1 is a plan view showing the arrangement and imaging area of ​​each of the multiple cameras mounted on the vehicle shown. [Figure 3] This block diagram shows a schematic configuration of the in-vehicle system shown in Figure 1. [Figure 4] Figure 3 is a block diagram showing the schematic functional configuration of a vehicle state estimation device according to one embodiment of the present disclosure, which is implemented in the ECU shown. [Figure 5] Figure 4 is a conceptual diagram illustrating the general operation of the vehicle state estimation device. [Figure 6] Figure 4 is a conceptual diagram illustrating the general operation of the vehicle state estimation device. [Figure 7] Figure 4 is a conceptual diagram illustrating the general operation of the vehicle state estimation device. [Figure 8] Figure 4 is a conceptual diagram illustrating the general operation of the vehicle state estimation device. [Figure 9] Figure 4 is a conceptual diagram illustrating the general operation of the vehicle state estimation device. [Figure 10]This is a conceptual diagram for explaining the outline of the operation of the vehicle state estimation device shown in FIG. 4. [Mode for Carrying Out the Invention]

[0012] (Embodiment) Hereinafter, exemplary embodiments or specific examples of the present disclosure will be described with appropriate reference to the drawings. Note that the descriptions of the following embodiments, their modifications, and the respective drawings related thereto are schematized or simplified for the purpose of concisely explaining the content of the present disclosure, and the content of the present disclosure is not limited in any way thereby. Therefore, it goes without saying that the descriptions in the respective drawings do not necessarily match the specific device configurations actually manufactured and sold. That is, it goes without saying that the present disclosure should not be construed in a limited manner by the descriptions in the respective drawings and the corresponding descriptions of the device configurations, their functions, or operations described below, unless the applicant expressly limits the present disclosure through the prosecution history of the present application.

[0013] (In-vehicle System Configuration) First, referring to FIG. 1, the in-vehicle system 1 is mounted on the host vehicle V and configured to execute various operations on the host vehicle V. That is, the host vehicle V is a vehicle equipped with the in-vehicle system 1 according to the present embodiment. In the present embodiment, the host vehicle V is a so-called ordinary automobile capable of traveling on roads, and has a box-shaped vehicle body V1.

[0014] The in-vehicle system 1 is configured to execute various operations including motion control operations of the host vehicle V and notification operations related thereto, based at least on the detection result of the target B around the host vehicle V using the camera 2. The target B includes objects above the road surface R, road markings formed on the road surface R, and the like. The "object" includes stationary objects and moving objects. The stationary objects include fixed objects such as poles, road signs and buildings, portable objects such as cones, and temporary stationary objects resulting from the stop of moving objects such as parked or stopped vehicles. The moving objects include pedestrians, other vehicles, and the like.

[0015] In the present embodiment, the in-vehicle system 1 is configured as a so-called advanced safety system, an ADAS system, or a driving automation system. ADAS is an abbreviation for Advanced Driver-Assistance Systems. "Driving automation" is a concept that includes driving assistance and automated driving.

[0016] That is, the in-vehicle system 1 is configured to be capable of realizing a driving automation level corresponding to at least any one of levels 1 to 5 defined in the standard "SAE J3016" published by SAE International. SAE is an abbreviation for Society Of Automotive Engineers. Level X in "SAE J3016" is hereinafter simply referred to as "SAE level X". X is any one of 0 to 5. SAE level 0 is referred to as manual driving. SAE level 1 is referred to as driving assistance. SAE level 2 is referred to as advanced driving assistance. SAE level 3 is referred to as conditional automated driving. SAE level 4 is referred to as highly automated driving. SAE level 5 is referred to as fully automated driving.

[0017] In the present embodiment, the in-vehicle system 1 is configured to be capable of continuously executing at least either one of the longitudinal vehicle motion control subtask and the lateral vehicle motion control subtask included in the dynamic driving task defined in "SAE J3016". The longitudinal vehicle motion control subtasks are starting, acceleration / deceleration, and stopping. The lateral vehicle motion control subtask is steering. Specifically, the in-vehicle system 1 is configured to be capable of realizing vehicle motion control such as ABS and VSC, for example.

[0018] The camera 2 includes an imaging element such as a CCD or a CMOS, and is mounted at a predetermined position on the host vehicle V to capture images of the surroundings of the host vehicle V. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

[0019] In this embodiment, the vehicle V is equipped with multiple cameras 2. Specifically, as shown in Figure 2, the vehicle V is provided with a front camera 2a, a left front side camera 2b, a right front side camera 2c, a left rear side camera 2d, and a right rear side camera 2e. Furthermore, the front camera 2a, left front side camera 2b, right front side camera 2c, left rear side camera 2d, and right rear side camera 2e are configured as so-called wide-angle cameras.

[0020] In Figure 2, the vehicle centerline CL is a hypothetical straight line that passes through the center of the vehicle body V1 in the width direction and is parallel to the vehicle length direction. The vehicle length direction is parallel to the direction of travel when the vehicle V is traveling in a straight line. The vehicle width direction is perpendicular to the vehicle length direction. The direction perpendicular to both the vehicle width direction and the vehicle length direction is called the vehicle height direction. The vehicle height direction is parallel to the direction of gravity when the vehicle V is placed on a horizontal road surface R in a state where it can travel. Figure 2 shows the vehicle V in a plan view. "Plan view" means viewing an object in the same direction as the direction of gravity when it is placed on a horizontal surface.

[0021] Furthermore, in Figure 2, an x-axis is defined parallel to the vehicle centerline CL, with the positive direction of travel being the direction of travel when the vehicle V is moving straight, and a y-axis is defined parallel to the vehicle width direction. In this case, the positive x-axis direction is called "forward," the negative x-axis direction is called "rearward," the positive y-axis direction is called "right," and the negative y-axis direction is called "left." These definitions of the front, back, left, and right directions are set to coincide with the front, back, left, and right directions as seen from the perspective of an occupant facing the direction of travel of the vehicle V.

[0022] The front camera 2a is mounted at the front of the vehicle body V1, at the center in the vehicle width direction, so that it has an imaging area in front of the vehicle V. Specifically, the front camera 2a can be installed, for example, at the front of the vehicle body V1 or in the rearview mirror inside the vehicle. The forward imaging area AF, which is the imaging area by the front camera 2a, has a predetermined field of view centered on the vehicle centerline CL, which extends from the front camera 2a in the forward imaging direction DF, which is in the same direction as the positive x-axis.

[0023] The left front side camera 2b is mounted on the front of the vehicle body V1, at the leftmost end in the vehicle width direction, so that it has an imaging area on the left front side of the vehicle V. The left front imaging direction DFL, which is the orientation of the left front side camera 2b, is set to form a predetermined acute angle (for example, about 30 to 45 degrees) with respect to the front imaging direction DF. The left front imaging area AFL, which is the imaging area by the left front side camera 2b, has a predetermined field of view centered on a virtual straight line extending from the left front side camera 2b to the left front imaging direction DFL.

[0024] The right front side camera 2c is mounted on the front of the vehicle body V1, at the rightmost end in the vehicle width direction, so that it has an imaging area on the right front side of the vehicle V. The right front side camera 2c is positioned symmetrically with respect to the left front side camera 2b with respect to the vehicle centerline CL. The right front imaging direction DFR, which is the orientation of the right front side camera 2c, is set to form a predetermined acute angle (for example, about 30 to 45 degrees) with respect to the front imaging direction DF. The right front imaging area AFR, which is the imaging area by the right front side camera 2c, has a predetermined field of view centered on a virtual straight line extending from the right front side camera 2c to the right front imaging direction DFR. The right front imaging area AFR is formed symmetrically with respect to the left front imaging area AFL with respect to the vehicle centerline CL.

[0025] The left rear side camera 2d is mounted on the left end of the vehicle body V1 in the vehicle width direction so that it has an imaging area on the left rear side of the vehicle V. The left rear imaging direction DRL, which is the orientation of the left rear side camera 2d, is set to form a predetermined acute angle (for example, about 30 to 45 degrees) with respect to the vehicle centerline CL that extends to the rear. The left rear imaging area ARL, which is the imaging area by the left rear side camera 2d, has a predetermined field of view centered on a virtual straight line extending in the left rear imaging direction DRL.

[0026] The right rear-side camera 2e is mounted on the right end of the vehicle body V1 in the vehicle width direction so that it has an imaging area on the right rear side of the vehicle V. The right rear-side camera 2e is positioned symmetrically with respect to the left rear-side camera 2d with respect to the vehicle centerline CL. The direction of the right rear-side camera 2e, the right rear imaging direction DRR, is set to form a predetermined acute angle (for example, about 30 to 45 degrees) with respect to the vehicle centerline CL that extends to the rear. The imaging area by the right rear-side camera 2e, the right rear imaging area ARR, has a predetermined field of view centered on a virtual straight line extending in the right rear imaging direction DRR. The right rear imaging area ARR is formed symmetrically with respect to the left rear imaging area ARL with respect to the vehicle centerline CL.

[0027] Referring to Figure 3, the in-vehicle system 1 includes, in addition to the camera 2, a radar sensor 3, a laser radar sensor 4, an ultrasonic sensor 5, a vehicle speed sensor 6, a yaw rate sensor 7, an ECU 8, and a motion control device 9. ECU stands for Electronic Control Unit. The camera 2, radar sensor 3, laser radar sensor 4, ultrasonic sensor 5, vehicle speed sensor 6, yaw rate sensor 7, and motion control device 9 are connected to the ECU 8 via an in-vehicle network, enabling them to exchange information or signals.

[0028] The radar sensor 3 is configured to detect targets B around the vehicle V based on the transmission and reception results of radar waves, which are radio waves in the millimeter-wave or sub-millimeter-wave band. Specifically, in addition to detecting the presence of targets B, the radar sensor 3 is configured to detect the distance to the targets B, the azimuth angle, and the relative speed of the targets B with respect to the vehicle V. The configuration of the radar sensor 3 is already well known at the time of filing this application, so no further detailed description is provided herein.

[0029] The laser radar sensor 4 is configured to detect target B by emitting detection light, which is laser light in the infrared band, outside the vehicle V and receiving the reflected light from target B. The laser radar sensor 4 may also be called LIDAR. LIDAR is an abbreviation for Light Detection and Ranging or Laser Imaging Detection and Ranging. The configuration of the laser radar sensor 4 is already well known at the time of filing of this application, so further detailed explanation is omitted in this specification.

[0030] The ultrasonic sensor 5 is an ultrasonic distance measuring sensor configured to detect target B by irradiating a probe wave in the ultrasonic band toward the space outside the vehicle V and receiving the reflected wave from target B. Specifically, the ultrasonic sensor 5 calculates the distance to target B based on TOF and the speed of sound. TOF stands for Time of Flight. Multiple ultrasonic sensors 5 are mounted on the vehicle body V1 of the vehicle V so as to be able to detect targets B located in front of, to the front side, to the side, to the rear side, and behind the vehicle V. The configuration and arrangement of such ultrasonic sensors 5 are already well known at the time of filing this application, so further detailed explanation is omitted in this specification.

[0031] The vehicle speed sensor 6 is provided to detect the travel speed of the vehicle V. Specifically, as the vehicle speed sensor 6, for example, a so-called wheel speed sensor capable of detecting the rotational speed of the wheels of the vehicle V may be used. The wheel speed sensor is attached to the wheel portion of the wheel so as to output a wheel speed signal corresponding to the rotational speed of the wheel. The configuration and arrangement of the vehicle speed sensor 6 are already well known at the time of filing of this application, so further detailed explanation is omitted in this specification.

[0032] The yaw rate sensor 7 is provided to detect the angular velocity of the vehicle V, that is, the rotational angular velocity around a vertical axis parallel to the vehicle height direction. Specifically, the yaw rate sensor 7 has a well-known configuration such as a piezoelectric vibration type gyro sensor. The configuration and arrangement of the yaw rate sensor 7 are already well-known at the time of filing this application, so further detailed explanation is omitted in this specification.

[0033] ECU8 is an ADAS-ECU or driving automation ECU, and is configured to control the ADAS or driving automation operations of the in-vehicle system 1. Specifically, ECU8 determines the operation control content related to ADAS or driving automation based on the detection results of target B using camera 2, etc., and the detection results of the motion state of the vehicle V using vehicle speed sensor 6, etc., and outputs it to motion control device 9.

[0034] The motion control device 9 is configured to perform motion control of the vehicle V, that is, control of the operation of the drive force generation mechanism, drive force transmission mechanism, braking mechanism, steering mechanism, etc., based on control signals generated and output by the ECU 8. In other words, the motion control device 9 is provided to perform longitudinal and / or lateral motion control of the vehicle V. More specifically, the motion control device 9 is configured to perform at least a portion of the motion control of the vehicle V, such as starting, acceleration / deceleration, braking, stopping, and steering. In this embodiment, the motion control device 9 has the same configuration as the ECU 8 as an on-board microcomputer.

[0035] (ECU) In this embodiment, the ECU8 has a configuration as an in-vehicle microcomputer comprising at least an arithmetic unit 81 and a storage device 82. The arithmetic unit 81 comprises at least one arithmetic unit having a configuration as a CPU or MPU, and its peripheral circuits (e.g., a timer circuit). CPU stands for Central Processing Unit. MPU stands for Micro Processing Unit.

[0036] The storage device 82 includes at least RAM and ROM and / or non-volatile rewritable memory, among various non-transitional physical storage media such as ROM, RAM, and non-volatile rewritable memory. ROM stands for Read Only Memory. RAM stands for Random Access Memory. Non-volatile rewritable memory is a storage device that allows information to be rewritten while the power is on, but keeps information unrewritable when the power is off, such as flash memory. Data and computer programs used by the arithmetic unit 81 are pre-stored in the ROM or non-volatile rewritable memory.

[0037] In other words, the ECU 8 is configured to realize predetermined functions related to ADAS or driving automation of the vehicle V by having the arithmetic unit 81 read a computer program from the storage device 82 and execute it. The storage device 82 stores the above-mentioned computer program along with various data necessary for its execution, such as initial values, maps, and lookup tables.

[0038] (Vehicle condition estimation device) Figure 4 shows an example of the functional block configuration of the vehicle state estimation device 800, which is realized by the arithmetic unit 81 in the ECU 8 executing a computer program. Specifically, the vehicle state estimation device 800 shown in Figure 4 is realized by the arithmetic unit 81 shown in Figure 3 reading and executing the vehicle state estimation program according to this disclosure from the storage device 82.

[0039] Referring to Figures 1 to 4 below, the vehicle state estimation device 800 is configured to estimate the vehicle state, which is the attitude and / or motion state of the vehicle V, based on images captured by a camera 2 mounted on the vehicle V. Specifically, as shown in Figure 4, the vehicle state estimation device 800 comprises an image acquisition unit 801 and an estimation unit 802 as a functional configuration realized by executing a computer program.

[0040] The image acquisition unit 801 acquires stereo images of stationary objects (hereinafter abbreviated as "target stationary objects") above the road surface R surrounding the vehicle V in at least two of the front, side, and rear regions of the vehicle V. Specifically, the image acquisition unit 801 receives image data from the front camera 2a, left front side camera 2b, right front side camera 2c, left rear side camera 2d, and right rear side camera 2e, and stores them in a predetermined capacity in a time series.

[0041] The estimation unit 802 estimates the state of the vehicle based on the stereo image acquired by the image acquisition unit 801. Specifically, the estimation unit 802 estimates the attitude and egomotion of the vehicle V from the displacement vectors of a stationary object in the front, rear, left, and right directions of the vehicle V. More specifically, the estimation unit 802 can estimate the yaw and driving speed of the vehicle V as egomotion. In addition, the estimation unit 802 can estimate the occurrence of slip on the rear wheel side based on the forward and rearward displacement of the vehicle V. More specifically, the estimation unit 802 comprises a forward displacement calculation unit 821, a rearward displacement calculation unit 822, a leftward displacement calculation unit 823, a rightward displacement calculation unit 824, and a calculation result integration unit 825.

[0042] The forward displacement calculation unit 821 estimates the actual displacement of the front end of the vehicle V from the calculation result of the relative movement state between the vehicle V and a stationary object in the forward region, based on the stereo image of the forward region acquired by the image acquisition unit 801. The stereo image of the forward region can be formed, for example, by the front camera 2a in the forward imaging region AF and the left front imaging region AFL taken by the left front side camera 2b or the right front imaging region AFR taken by the right front side camera 2c. In other words, the stereo image of the forward region is acquired by multiple cameras 2 whose fields of view partially overlap and have different angles, namely the front camera 2a and the left front side camera 2b or the right front side camera 2c.

[0043] The rearward displacement calculation unit 822 estimates the actual displacement of the rear end of the vehicle V from the calculation result of the relative movement state between the vehicle V and a stationary object in the rearward region, based on the stereo image of the rearward region acquired by the image acquisition unit 801. The stereo image of the rearward region can be formed, for example, by the left rearward camera 2d in the left rearward imaging region ARL and the right rearward camera 2e in the right rearward imaging region ARR. In other words, the stereo image of the rearward region is acquired by multiple cameras 2, namely the left rearward camera 2d and the right rearward camera 2e, whose fields of view partially overlap and have different angles.

[0044] The leftward displacement calculation unit 823 estimates the actual displacement of the left side of the vehicle V from the calculation result of the relative movement state between the vehicle V and a stationary object located in the leftward region, based on the stereo image of the leftward region acquired by the image acquisition unit 801. The stereo image in the leftward region can be formed, for example, by the left front side camera 2b in the left front imaging region AFL and the left rear side camera 2d in the left rear imaging region ARL. In other words, the stereo image in the leftward region is acquired by multiple cameras 2, the left front side camera 2b and the left rear side camera 2d, which have partially overlapping fields of view and different angles.

[0045] The rightward displacement calculation unit 824 estimates the actual displacement of the right side of the vehicle V from the calculation result of the relative movement state between the vehicle V and a stationary object located in the rightward region, based on the stereo image of the rightward region acquired by the image acquisition unit 801. The stereo image in the rightward region can be formed, for example, by the image captured in the right front imaging region AFR by the right front side camera 2c and the image captured in the right rear imaging region ARR by the right rear side camera 2e. In other words, the stereo image in the rightward region is acquired by multiple cameras 2, namely the right front side camera 2c and the right rear side camera 2e, which have partially overlapping fields of view and different angles.

[0046] The calculation result integration unit 825 calculates an estimated value of the vehicle's state by integrating the calculation results from the forward displacement calculation unit 821, the rearward displacement calculation unit 822, the leftward displacement calculation unit 823, and the rightward displacement calculation unit 824. One of the forward region, leftward region, rightward region, and rearward region corresponds to the "first region," and the other one corresponds to the "second region." In addition, one of the forward displacement calculation unit 821, rearward displacement calculation unit 822, leftward displacement calculation unit 823, and rightward displacement calculation unit 824 corresponds to the "first calculation unit," and the other one corresponds to the "second calculation unit."

[0047] The forward displacement calculation unit 821, the backward displacement calculation unit 822, the leftward displacement calculation unit 823, and the rightward displacement calculation unit 824 each comprise a stationary object detection unit 831, a distance calculation unit 832, and a displacement calculation unit 833, respectively. The stationary object detection unit 831 is configured to detect stationary objects that are commonly imaged between multiple cameras 2. The distance calculation unit 832 calculates the distance to a stationary object based on parallax information. The displacement calculation unit 833 calculates the actual displacement and angle using optical flow. Since these details are already publicly known or well-known at the time of filing this application, further detailed explanation is omitted in this specification.

[0048] (Operation overview) The following describes the operation of the device configuration according to this embodiment (i.e., the vehicle state estimation device 800), along with the effects achieved by the device configuration and the methods and programs executed thereunder. In the following description, the device configuration according to this embodiment, and the methods and programs executed thereunder, may be collectively referred to as "this embodiment."

[0049] The image acquisition unit 801 acquires images captured by wide-angle cameras: the front camera 2a, the left front side camera 2b, the right front side camera 2c, the left rear side camera 2d, and the right rear side camera 2e. This allows the acquisition of stereo images of stationary objects around the vehicle V in at least two of the front, side, and rear regions of the vehicle V. The estimation unit 802 estimates the state of the vehicle based on the stereo images acquired by the image acquisition unit 801.

[0050] Specifically, for example, in the area in front of the vehicle V, a stereo image of a stationary object can be acquired by the front camera 2a and the left front side camera 2b or the right front side camera 2c. In the forward displacement calculation unit 821, the stationary object detection unit 831 detects predetermined stationary objects that are commonly captured between the front camera 2a and the left front side camera 2b or the right front side camera 2c. Such predetermined objects may be, for example, road signs, guardrails, traffic lights, buildings, etc., and can be registered in advance or extracted by image recognition.

[0051] The distance calculation unit 832 calculates the distance to a stationary object detected in the area in front of the vehicle V based on parallax information. In addition, when calculating the distance, the distance measurement results from the radar sensor 3, laser radar sensor 4, and ultrasonic sensor 5 may also be used as supplementary or alternative methods. The displacement calculation unit 833 then calculates the actual displacement and angle of the stationary object using optical flow, and estimates the actual displacement of the front end, rear end, and side of the vehicle V from these calculation results.

[0052] The calculation result integration unit 825 calculates an estimated value of the vehicle's state by integrating the calculation results from the forward displacement calculation unit 821, the rearward displacement calculation unit 822, the leftward displacement calculation unit 823, and the rightward displacement calculation unit 824. Specifically, the calculation result integration unit 825 compares the calculation result of the forward displacement from the forward displacement calculation unit 821, the calculation result of the rearward displacement from the rearward displacement calculation unit 822, the calculation result of the leftward displacement from the leftward displacement calculation unit 823, and the calculation result of the rightward displacement from the rightward displacement calculation unit 824. Then, the calculation result integration unit 825 estimates the vehicle's state based on these comparison results.

[0053] Here, for example, distant objects (such as distant mountains or buildings) hardly displace even when the vehicle V moves. Therefore, estimating the vehicle's state using such objects tends to be inaccurate. In contrast, nearby objects (such as road signs) show a large apparent displacement even with slight movement of the vehicle V. Therefore, estimating the vehicle's state using such objects improves the accuracy of the vehicle state estimation. However, if the object is too close, lens distortion in camera 2 may occur. Therefore, the calculation result integration unit 825 can estimate the vehicle's state with good accuracy by using a stationary object at an appropriate distance from the vehicle V.

[0054] (Specific example 1: Yo) To estimate the yaw of the vehicle V, both the actual displacement and the distance are necessary. The following is an overview of yaw estimation.

[0055] Figure 5 shows the case where the relative position of a stationary object around the vehicle V moves from (x1, y1) to (x2, y2). In this case, θ can be calculated from Arcsinθ = Δ / d2, and an estimated yaw value can be calculated from θ / t. t is the elapsed time until the relative position moves from (x1, y1) to (x2, y2).

[0056] Figure 6 shows an example of estimating the state of the vehicle V from the detection results of stationary objects in the forward and rear regions of the vehicle. For stationary objects in the forward region, if the distance is R1 and the actual displacement is Δx1, then Arcsinθ = Δx1 / R1. Similarly, for stationary objects in the rear region, if the distance is R2 and the actual displacement is Δx2, then Arcsinθ = Δx2 / R2.

[0057] Figure 7 shows the situation when the vehicle V rotates clockwise around its center point Pc. In this case, the forward displacement Δxf at the front end of the vehicle V and the rearward displacement Δxr at the rear end are approximately the same in magnitude but opposite in direction. Similarly, the leftward displacement ΔyL at the left side of the vehicle V and the rightward displacement Δyr at the right side are approximately the same in magnitude but opposite in direction.

[0058] Figure 8 shows the situation when the rear wheels of the vehicle V slip. In this case, the vehicle V rotates around a rotation center Pr that is forward of the vehicle's center point Pc. As a result, the rearward displacement Δxr is greater than the forward displacement Δxf. On the other hand, the relationship between the leftward displacement ΔyL and the rightward displacement Δyr is almost the same as in Figure 7.

[0059] In this way, by integrating the calculation results from the forward displacement calculation unit 821, the rearward displacement calculation unit 822, the leftward displacement calculation unit 823, and the rightward displacement calculation unit 824, it becomes possible to estimate the vehicle's state more accurately than before. Based on this accurately estimated vehicle state, it becomes possible to perform appropriate vehicle motion control (for example, brake control to prevent slippage).

[0060] In this respect, conventional yaw detection using only the yaw rate sensor 7 did not provide detailed information about the vehicle's attitude V. On the other hand, as in this embodiment, by using the camera 2, it is possible to accurately estimate the vehicle's attitude and driving conditions, such as whether the vehicle V is spinning, whether only the rear wheels are slipping, or whether it is turning, based on the displacement vectors at each position of the vehicle V.

[0061] (Specific example 2: Vehicle posture and driving speed) The following is an overview of the estimation of vehicle posture and driving speed.

[0062] Figure 9 shows the situation when the vehicle V is turning to the right. When turning to the right, as shown in Figure 9, the leftward displacement ΔyL and the rightward displacement Δyr are in the same direction, but the leftward displacement ΔyL is larger in magnitude. This allows us to estimate that the vehicle V is turning to the right, and it is also possible to estimate the travel speed from the rightward displacement Δyr on the inner wheel side. On the other hand, if the leftward displacement ΔyL and the rightward displacement Δyr are the same in both direction and magnitude, it is possible to accurately estimate that the vehicle is traveling straight and determine its travel speed.

[0063] Figure 10 shows an example of determining the actual displacement from the displacement Δx on camera 2. As shown in Figure 10, assume that the stationary object moves horizontally by Δx pixels on the image sensor of camera 2. In this case, the formula ΔX = d·tanθ, which shows the relationship between the actual displacement ΔX, distance d, and angle θ, is given by θ [degrees] = Δx [pixels] / W [pixels] · field of view [degrees]. From this relationship, it is possible to calculate the actual displacement ΔX.

[0064] (modified version) This disclosure is not limited to the embodiments and specific examples described above. Therefore, the embodiments, etc. can be modified as appropriate. Representative modifications are described below. In the description of the modifications below, the differences from the embodiments, etc. will be mainly described. In addition, parts that are the same or equivalent to each other in the embodiments, etc. and the modifications below are denoted by the same reference numerals. Therefore, in the description of the modifications below, with respect to components that have the same reference numerals as in the embodiments, etc., the descriptions in the embodiments, etc. can be appropriately referenced unless there is a technical inconsistency or special additional explanation.

[0065] This disclosure is not limited to the specific uses or device configurations shown in the embodiments described above. For example, the vehicle V may be a regular passenger car or a large vehicle. In other words, there are no particular limitations on the shape or size of the vehicle body V1.

[0066] There are no particular limitations on the number or configuration of Camera 2. That is, each of the multiple Camera 2s may be a monocular camera or a compound stereo camera. There are also no particular limitations on the presence, number, or configuration of Radar Sensor 3, Laser Radar Sensor 4, or Ultrasonic Sensor 5.

[0067] The ECU8 and motion control device 9, in whole or in part, may be configured with digital circuits, such as ASICs or FPGAs, that are capable of realizing the functions or operations described above. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field Programmable Gate Array. In other words, the on-board microcomputer portion and the digital circuit portion can coexist in the ECU8 and motion control device 9.

[0068] The computer program according to this disclosure, which enables the execution of various operations, procedures, or processes described in the above embodiments, can be downloaded or upgraded via V2X communication using a communication device. V2X is an abbreviation for Vehicle to X. Alternatively, such a computer program can be downloaded or upgraded via terminal equipment installed at the manufacturing plant, repair shop, dealership, etc., of the vehicle V. The storage location of such a computer program may be a memory card, optical disk, magnetic disk, etc.

[0069] Thus, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring an arithmetic unit 81 and a storage device 82 programmed to execute one or more functions embodied by a computer program. Alternatively, each of the above functional configurations and processes may be realized by a dedicated computer provided by configuring an arithmetic unit 81 with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and processes may be realized by one or more dedicated computers configured by a combination of one or more arithmetic units 81 and one or more storage devices 82 programmed to execute one or more functions, and one or more other arithmetic units 81 configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitional substantial storage medium as instructions to be executed by the computer. That is, each of the above functional configurations and processes can also be represented as a computer program including procedures for realizing them, or as a non-transitional substantial storage medium storing said computer program.

[0070] It goes without saying that the elements constituting the above embodiments are not necessarily essential unless explicitly stated to be particularly essential or considered to be fundamentally essential. Furthermore, when numerical values ​​such as the number, numerical values, quantities, or ranges of components are mentioned, this disclosure is not limited to those specific numbers unless explicitly stated to be particularly essential or considered to be fundamentally limited to those specific numbers. Similarly, when the shape, orientation, positional relationship, etc., of components are mentioned, this disclosure is not limited to those shapes, orientations, positional relationships, etc., unless explicitly stated to be particularly essential or considered to be fundamentally limited to those specific shapes, orientations, positional relationships, etc.

[0071] Similar expressions such as "acquisition," "calculation," "estimation," "detection," and "detection" can be appropriately substituted for each other within the limits of what is technically consistent. Similarly, "exceeding the threshold" and "above the threshold" can be appropriately substituted for each other within the limits of what is technically consistent. The same applies to "below the threshold" and "below the threshold."

[0072] Modifications are not limited to the examples given above. For example, all or part of one of the modifications may be combined with all or part of another, provided that it does not conflict with the technical specifications. Furthermore, all or part of the specific examples may be combined with all or part of the modifications, provided that it does not conflict with the technical specifications.

[0073] (Disclosure perspective) As is evident from the above description of embodiments and modifications, this specification discloses at least the following:

[0074] [Perspective 1-1] A vehicle state estimation device (800) that estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle, An image acquisition unit (801) acquires stereo images of stationary objects above the road surface (R) surrounding the vehicle in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. An estimation unit (802) estimates the state of the vehicle based on the stereo image acquired by the image acquisition unit, A vehicle condition estimation device equipped with the following features. [Perspective 1-2] The estimation unit, A first calculation unit (821) calculates the relative movement state between the vehicle and stationary objects in the first region based on the stereo image in the first region, A second calculation unit (822) calculates the relative movement state between the vehicle and stationary objects in the second region based on the stereo image in the second region, A calculation result integration unit (825) estimates the state of the vehicle by integrating the calculation results from the first calculation unit and the second calculation unit, Equipped with, The vehicle condition estimation device described in Perspective 1-1. [Perspectives 1-3] The estimation unit estimates the yaw of the vehicle. A vehicle condition estimation device as described in perspective 1-1 or perspective 1-2. [Perspectives 1-4] The estimation unit estimates the driving speed of the vehicle. A vehicle condition estimation device described in any one of the viewpoints 1-1 to 1-3. [Perspectives 1-5] The estimation unit estimates the occurrence of slip on the rear wheel side based on the forward and rearward displacement of the vehicle. A vehicle condition estimation device described in any one of the viewpoints 1-1 to 1-4. [Perspectives 1-6] The stereo image is acquired by multiple cameras with partially overlapping fields of view and different angles. A vehicle condition estimation device described in any one of the viewpoints 1-1 to 1-5.

[0075] [Perspective 2-1] A vehicle state estimation method that estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle, Stereo images of stationary objects above the road surface (R) surrounding the vehicle are acquired in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. Based on the acquired stereo image, the state of the vehicle is estimated. A method for estimating vehicle condition. [Perspective 2-2] In the estimation of the vehicle's state, Based on the stereo image in the first region, the relative movement state between the vehicle and stationary objects in the first region is calculated. Based on the stereo image in the second region, the relative movement state between the vehicle and stationary objects in the second region is calculated. By integrating these calculation results, the state of the vehicle is estimated. The vehicle condition estimation method described in Perspective 2-1. [Perspective 2-3] In the estimation of the vehicle's state, the yaw of the vehicle is estimated. The vehicle condition estimation method described in perspective 2-1 or perspective 2-2. [Perspective 2-4] In the estimation of the vehicle's state, the vehicle's speed is estimated. A vehicle condition estimation method described in one of the following perspectives 2-1 to 2-3. [Perspectives 2-5] In the estimation of the vehicle's state, the occurrence of a slip in the rear is estimated based on the forward and rearward displacement of the vehicle. A vehicle condition estimation method described in one of the viewpoints 2-1 to 2-4. [Perspective 2-6] The stereo image is acquired by multiple cameras with partially overlapping fields of view and different angles. A vehicle condition estimation method described in one of the viewpoints 2-1 to 2-5. [Explanation of Symbols]

[0076] 2 cameras 8. Vehicle condition estimation device 81 Image acquisition unit 82 Estimation part 821 Forward Displacement Calculation Unit 822 Rearward Displacement Calculation Unit 823 Leftward displacement calculation unit 824 Rightward Displacement Calculation Unit 825 Calculation Result Integration Department V My vehicle

Claims

1. A vehicle state estimation device (800) that estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle, An image acquisition unit (801) acquires stereo images of stationary objects above the road surface (R) surrounding the vehicle in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. An estimation unit (802) estimates the state of the vehicle based on the stereo image acquired by the image acquisition unit, A vehicle condition estimation device equipped with the following features.

2. The estimation unit, A first calculation unit (821) calculates the relative movement state between the vehicle and stationary objects in the first region based on the stereo image in the first region, A second calculation unit (822) calculates the relative movement state between the vehicle and stationary objects in the second region based on the stereo image in the second region, A calculation result integration unit (825) estimates the state of the vehicle by integrating the calculation results from the first calculation unit and the second calculation unit, Equipped with, The vehicle state estimation device according to claim 1.

3. The estimation unit estimates the yaw of the vehicle. The vehicle state estimation device according to claim 1.

4. The estimation unit estimates the driving speed of the vehicle. The vehicle state estimation device according to claim 1.

5. The estimation unit estimates the occurrence of slip on the rear wheel side based on the forward and rearward displacement of the vehicle. The vehicle state estimation device according to claim 1.

6. The stereo image is acquired by multiple cameras with partially overlapping fields of view and different angles. The vehicle state estimation device according to claim 1.

7. A vehicle state estimation method that estimates the vehicle state, which is the attitude and / or motion state of the vehicle (V), based on images captured by a camera (2) mounted on the vehicle, Stereo images of stationary objects above the road surface (R) surrounding the vehicle are acquired in a first region and a second region, which are at least two regions among the front region, lateral region and rear region of the vehicle. Based on the acquired stereo image, the state of the vehicle is estimated. A method for estimating vehicle condition.

8. In the estimation of the vehicle's state, Based on the stereo image in the first region, the relative movement state between the vehicle and stationary objects in the first region is calculated. Based on the stereo image in the second region, the relative movement state between the vehicle and stationary objects in the second region is calculated. By integrating these calculation results, the state of the vehicle is estimated. The vehicle condition estimation method according to claim 7.

9. In the estimation of the vehicle's state, the yaw of the vehicle is estimated. The vehicle condition estimation method according to claim 7.

10. In the estimation of the vehicle's state, the vehicle's speed is estimated. The vehicle condition estimation method according to claim 7.

11. In the estimation of the vehicle's state, the occurrence of a slip in the rear is estimated based on the forward and rearward displacement of the vehicle. The vehicle condition estimation method according to claim 7.

12. The stereo image is acquired by multiple cameras with partially overlapping fields of view and different angles. The vehicle condition estimation method according to claim 7.

Images