On-vehicle object recognition device

Abstract

To provide an on-vehicle object recognition device capable of correctly calculating a ground speed of an object to be photographed by a photographing part of a camera which is arranged at a position different from a rotation center of a vehicle.SOLUTION: An on-vehicle object recognition device is to be mounted on a vehicle having a photographing part and includes an arithmetic unit. In the on-vehicle object recognition device, the arithmetic unit performs: detecting a stationary object based on an image photographed by the photographing part; estimating a moment arm length being a distance from a rotation center of the vehicle to the imaging part; and obtaining a ground speed of the object based on the estimated moment arm length.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an in-vehicle object recognition device. [Background technology]

[0002] Conventionally, there is known a technology for controlling a vehicle based on image information obtained by capturing images using a camera. Patent Document 1 describes a vehicle motion control device that uses stereo cameras arranged on either side of the rearview mirror in the vehicle cabin to capture images of objects in front of the vehicle and controls the vehicle's braking and turning behavior based on image information obtained by the stereo cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-074366 A Summary of the Invention [Problem to be solved by the invention]

[0004] The motion control device described in Patent Document 1 does not take into account the moment arm length, which is the distance from the turning center of the vehicle to the camera. Therefore, if the camera is installed in a position other than the turning center of the vehicle, it cannot accurately calculate the ground speed of an object captured by the camera.

[0005] An object of the present invention is to provide an in-vehicle object recognition device capable of accurately calculating the ground speed of an object imaged by an imaging unit installed at a position other than the center of turning of the vehicle. [Means for solving the problem]

[0006] An in-vehicle object recognition device according to one embodiment of the present invention is an in-vehicle object recognition device equipped with a computing device that is mounted on a vehicle having an imaging unit, wherein the computing device detects a stationary object based on an image captured by the imaging unit, estimates a moment arm length, which is the distance from the center of turning of the vehicle to the imaging unit, and calculates the ground speed of the object based on the estimated moment arm length. Effect of the Invention

[0007] According to the present invention, it is possible to accurately calculate the ground speed of an object imaged by an imaging unit installed at a position different from the turning center of the vehicle. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of a vehicle equipped with an on-vehicle object recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of the vehicle-mounted object recognition device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating a functional configuration of the vehicle-mounted object recognition device according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram showing an example of extracting feature points from a white line. [Diagram 5] FIG. 5 is an explanatory diagram showing an example of extracting feature points from a mailbox. [Figure 6] FIG. 6 is an explanatory diagram showing an example of feature point association. [Figure 7] FIG. 7 is a diagram showing an example of an estimated moment arm length. [Figure 8] FIG. 8 is an explanatory diagram for explaining the definitions of the distances Lp, Lc, and Lm. [Figure 9] FIG. 9 is an explanatory diagram for explaining the rotation angle and the relative movement amount of an object. [Figure 10] FIG. 10 is an explanatory diagram for explaining a method for calculating the relative movement amount. [Figure 11]FIG. 11 is an explanatory diagram for explaining the relationship between the vehicle turning center movement vector, the camera position movement vector, and the moment arm length. [Figure 12] FIG. 12 is a diagram showing the relationship between the assumed error in the moment arm length and the speed error. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of processing executed by the arithmetic device according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the flow of processing executed by the arithmetic device according to the second embodiment. [Figure 15] FIG. 15 is an explanatory diagram illustrating a change in the estimated value of the moment arm length over time. [Figure 16] FIG. 16 is a diagram showing an example of an estimated moment arm length. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment An on-vehicle object recognition device according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 13. FIG.

[0010] FIG. 1 is a schematic diagram of a vehicle equipped with an on-board object recognition device according to the first embodiment. The vehicle 1 is a cargo vehicle having a loading platform 100. A camera 2, which is an imaging device, is provided in the cabin of the vehicle 1. The camera 2 is a so-called stereo camera. The camera 2 captures an image of the front of the vehicle 1 and outputs a pair of captured images in the form of an imaging signal to the on-board object recognition device. When the vehicle 1 is stationary and nothing is loaded on the loading platform 100 of the vehicle 1, the center of gravity position (turning center position) of the vehicle 1 is at a position 102 shown in FIG. 1. In contrast, when a baggage 101 is loaded on the loading platform 100 of the vehicle 1 as shown in FIG. 1, the center of gravity position of the vehicle 1 when the vehicle 1 is stationary is at a position 103 shown in FIG. 1. In this way, the center of gravity position of the vehicle 1 changes depending on the state of the load (for example, the weight and loading position of the load). The center of gravity position of the vehicle 1 is also affected by the moving speed (vehicle speed) of the vehicle 1.

[0011] FIG. 2 is a block diagram showing a hardware configuration of the vehicle-mounted object recognition device according to the first embodiment. The vehicle-mounted object recognition device 3 is composed of a computer including an arithmetic device 11 such as a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), etc., a non-volatile memory 12 such as a read only memory (ROM), a flash memory, a hard disk drive, etc., a volatile memory 13 called a random access memory (RAM), an input / output interface 14, and other peripheral circuits. These hardware components work together to operate software to realize a plurality of functions. The vehicle-mounted object recognition device 3 may be composed of one computer or a plurality of computers. The arithmetic device 11 may include a dedicated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0012] The non-volatile memory 12 stores programs capable of executing various calculations. In other words, the non-volatile memory 12 is a storage medium (storage device) capable of reading programs that realize the functions of this embodiment. The volatile memory 13 is a storage medium (storage device) that temporarily stores the results of calculations performed by the calculation device 11 and signals input from the input / output interface 14. The calculation device 11 is a processing device that expands the programs stored in the non-volatile memory 12 into the volatile memory and executes calculations, and performs predetermined calculations on data taken from the input / output interface 14, the non-volatile memory 12, and the volatile memory 13 in accordance with the programs.

[0013] The input / output interface 14 is connected to the camera 2, the vehicle speed sensor 4, the vehicle steering angle sensor 5, and the yaw rate sensor 6. The input section of the input / output interface 14 converts signals input from various devices (such as the camera 2, the vehicle speed sensor 4, the vehicle steering angle sensor 5, and the yaw rate sensor 6) into data that can be calculated by the arithmetic device 11. In addition, the output section of the input / output interface 14 generates an output signal according to the calculation result in the arithmetic device 11, and outputs the signal to the various devices.

[0014] The camera 2 includes a left imaging section 20a and a right imaging section 20b. In the following description, the left imaging section 20a and the right imaging section 20b are collectively referred to as the imaging section 20. The imaging section 20 includes an image sensor, such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and an imaging optical system. The left imaging section 20a images the area ahead of the vehicle 1 and outputs an imaging signal to the vehicle-mounted object recognition device 3. The right imaging section 20b images the area ahead of the vehicle 1 and outputs an imaging signal to the vehicle-mounted object recognition device 3. The left imaging section 20a and the right imaging section 20b are installed so that the positions of the optical axis centers in the vertical direction (perpendicular direction) are approximately equal. The left imaging section 20a and the right imaging section 20b are installed so that the positions of the optical axis centers in the horizontal direction are separated by a predetermined distance. In the following description, the distance between the optical axis centers in the horizontal direction of the left imaging section 20a and the right imaging section 20b is referred to as the base length. Since the left imaging section 20a and the right imaging section 20b are disposed apart in the horizontal direction by the base line length, a parallax occurs in the captured images resulting from each imaging operation according to the distance from the imaging section 20 to the subject and the base line length.

[0015] The vehicle speed sensor 4 is a sensor capable of acquiring the translational movement amount of the vehicle 1 per unit time. The vehicle speed sensor 4 is realized, for example, by acquiring the rotational speed of each wheel or the outer circumference of the wheel. The vehicle speed sensor 4 may be configured using a GNSS (Global Navigation Satellite System) or the like. The vehicle steering angle sensor 5 is a sensor capable of acquiring the traveling direction (azimuth) of the vehicle 1. The vehicle steering angle sensor 5 may be realized, for example, by measuring the steering angle itself, or may be realized by estimating the steering angle using the operation angle of the steering wheel and the steering gear ratio. The yaw rate sensor 6 is a sensor capable of acquiring the yaw rate (turning angular velocity) of the vehicle 1. The yaw rate sensor 6 is realized, for example, by an inertial sensor attached near the center of gravity of the vehicle 1.

[0016] The in-vehicle object recognition device 3 is a device that recognizes an object from the imaging signal output by the camera 2, and calculates and outputs the position of the object and the moment arm length of the vehicle 1. The information output by the in-vehicle object recognition device 3 is used for vehicle control processing, such as determining the possibility of a collision between the recognized object and the vehicle 1, issuing a warning to the driver of the vehicle 1, or automatically braking the vehicle 1 to avoid a collision with the recognized object.

[0017] For example, when the vehicle 1 is turning, the in-vehicle object recognition device 3 calculates the ground speed of the object captured by the camera 2 and outputs it to a control device (not shown). The control device (not shown) judges whether or not there is a possibility of a collision between the vehicle 1 and the object in the future (performs collision judgment). The control device compares a future predicted position calculated from the position of the recognized object in the camera coordinate system and the ground speed with a future predicted position calculated from the traveling path of the vehicle 1, and if the recognized object and the vehicle 1 are likely to be within a certain distance in the future, the control device (not shown) judges that there is a possibility of a collision and issues a warning to the driver or automatically applies the brakes to avoid the collision. Note that, here, an example in which a control device (not shown) performs vehicle control processing such as collision judgment, warning, and braking of the vehicle 1 has been described, but these vehicle control processing may be performed by the in-vehicle object recognition device 3.

[0018] 3 is a block diagram showing a schematic functional configuration of the vehicle-mounted object recognition device according to the first embodiment. The vehicle-mounted object recognition device 3 includes an image correction unit 31, a stereo matching unit 32, a road surface estimation unit 33, an object detection unit 34, a feature point extraction unit 35, a tracking unit 36, a relative speed calculation unit 37, a moment arm length estimation unit 38, a ground speed calculation unit 39, and a camera movement amount calculation unit 40.

[0019] The image correction unit 31 performs brightness normalization and parallelization on a pair of captured images obtained from the camera 2. Depending on the angle of view of the imaging unit 20, objects of the same brightness may appear in the images with different brightness values. The brightness normalization process is a process for correcting such a difference in brightness value. In other words, the brightness normalization process is a process for normalizing brightness so that objects of the same brightness appear in the captured images with the same brightness. The parallelization process is a so-called stereo parallelization process. In other words, it is a process for converting images into images in which the same point in the world coordinate system is aligned at the same height in both captured images. The parallelization process is realized by performing a projective transformation from a state in which geometric distortion occurs due to the imaging optical system to a preset projection plane, for example, a perspective projection plane.

[0020] The stereo matching unit 32 executes a parallax calculation process to calculate the parallax between the captured images. For each pixel of one captured image, the stereo matching unit 32 searches for a pixel at the same point in the other captured image, calculates the difference in horizontal coordinates (i.e., parallax) of the same point in the other captured image based on the one captured image, and generates a parallax image. The stereo matching unit 32 uses a method such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) to search for the same point. Note that instead of searching for the same point and calculating the parallax on a pixel-by-pixel basis, the same point may be searched for and the parallax may be calculated for only one pixel out of four consecutive pixels, and the result may be applied to all four pixels. In this way, by searching for the same point and calculating the parallax only for some pixels, the computation cost of the parallax calculation process can be reduced.

[0021] The road surface estimation unit 33 estimates the area occupied by the road surface (road surface area) from the captured image or the parallax image using the result of the parallax calculation process by the stereo matching unit 32. The road surface is considered to be a relatively flat area with a normal vector that is nearly vertically upward. Therefore, the road surface estimation unit 33 converts each point included in the parallax image into a point cloud in a three-dimensional space, and specifies the road surface area by plane fitting the point cloud.

[0022] The object detection unit 34 detects stationary and non-stationary objects from the captured images and parallax images. Stationary objects detected by the object detection unit 34 include, for example, markings and white lines painted on the road surface, installations such as manholes installed on the road surface, patterns on the road surface, etc. The object detection unit 34 detects, as stationary objects, marks with high brightness and high contrast that have almost no height relative to the road surface, installations with characteristic shapes such as manholes, patterns on the road surface with characteristic brightness distribution, etc., using well-known technology.

[0023] The types and detection methods of stationary objects detected by the object detection unit 34 are not limited to those described above. For example, objects and object types may be identified by a known technique such as semantic segmentation processing, and among the identified objects, objects with characteristic shapes that are tall but located near the road surface, such as buildings, residential buildings, store buildings, utility poles, traffic lights, and postboxes, may be detected as stationary objects.

[0024] The object detection unit 34 detects non-stationary objects from the captured image and the parallax image by well-known techniques such as inference using a neural network model (e.g., YOLO, etc.) and template matching. The method by which the object detection unit 34 detects stationary and non-stationary objects is not limited to the above-described method. For example, the object detection unit 34 may detect objects without distinguishing between stationary and non-stationary objects, and then classify the objects into stationary and non-stationary objects. In addition, both stationary and non-stationary objects may be detected at once by inference using a neural network model.

[0025] The feature point extraction unit 35 extracts characteristic points (so-called feature points) in stationary objects and non-stationary objects detected by the object detection unit 34. For example, SIFT (Scale-Invariant Feature Transform) or FAST (Features from Accelerated Segment Test), which is specialized for corner detection, can be used as the feature points.

[0026] The feature point extraction unit 35 calculates the feature amount including the position information of the feature point corresponding to the feature point together with the position of the feature point. When using a method that can describe the feature amount together with the feature point, such as SIFT, the feature amount can be acquired at the same time as the feature point. When such a method is not used, the feature amount is calculated separately after the feature point is extracted, for example, by a method such as BRIEF (Binary Robust Independent Elementary Features) or ORB (Oriented FAST And Rotated BRIEF).

[0027] FIG. 4 is an explanatory diagram showing an example of extracting feature points from a white line. A captured image 110 shows an intersection, and a white cross line 112 is drawn at the center of the intersection. An enlarged image 111 is an enlarged region of the entire captured image 110 in which the white cross line 112 appears. A grid in the enlarged image 111 separates the pixels. A rectangle 113 superimposed on a corner of the white cross line 112 is a corner of an edge on the image that is extracted as a feature point. The corner portion highlighted by the rectangle 113 can be extracted by, for example, FAST.

[0028] 5 is an explanatory diagram showing an example of extracting feature points from a mailbox. A photographed image 120 shows a mailbox 121, which is a stationary object that is tall relative to the road surface. When extracting feature points from the mailbox 121, which is a three-dimensional object, for example, a point indicated by a rectangle 122 superimposed on the corner of the mailbox 121, a center point 123 of the side (intersection of diagonals), etc. are extracted. The feature points of such an object having a three-dimensional shape can be extracted by recognizing the three-dimensional shape of the object from the coordinates of each pixel of the three-dimensional object obtained from parallax.

[0029] The tracking unit 36 ​​shown in FIG. 3 associates feature points between images captured at different times. That is, a feature point in an image captured at a certain time is associated with a feature point at the same point in an image captured next (after that time). In other words, feature points are associated between consecutive frames. Whether feature points in two captured images indicate the same point can be determined by comparing the feature amounts of those feature points. If the difference in the feature amount is very small (for example, equal to or less than a predetermined threshold), it can be considered that those feature points indicate the same point. The tracking unit 36 ​​calculates a predicted position of each feature point in a subsequent frame from the position of each feature point in a previous frame using the movement amount (position change) and movement direction (orientation change) of the camera 2. The tracking unit 36 ​​associates the feature points with each other by searching for a feature point having a feature amount close to the feature amount of the feature point in the previous frame from among feature points located near the predicted position.

[0030] When performing this matching, not only the feature values ​​but also the degree of similarity of the luminance values ​​of the pixels surrounding the feature points may be taken into consideration. By adding additional conditions in this way, the accuracy of the matching can be improved. Alternatively, more flexible matching may be performed by applying all feature values ​​including the positions of the feature points to a multidimensional space multiplied by an arbitrary coefficient for each dimension, and matching between feature points that are closest to each other in the multidimensional space. By setting an arbitrary coefficient in advance in consideration of the accuracy of the matching, it is possible to perform matching with higher accuracy.

[0031] FIG. 6 is an explanatory diagram showing an example of matching feature points. FIG. 6 shows an example of matching feature points between a photographed image 130 of a previous frame and a photographed image 131 of a subsequent frame. The tracking unit 36 ​​matches feature points 132, 133, 134, and 135 extracted from a white line on a road surface in a previous frame with feature points 137, 138, 139, and 140 extracted from the white line on the same road surface in a subsequent frame. Similarly, the tracking unit 36 ​​matches feature point 136 extracted from the diagonal center of a side surface of a three-dimensional object in a previous frame with feature point 141 extracted from the diagonal center of a side surface of the same three-dimensional object in a subsequent frame. As described above, in order to match feature points of the same location of the same object, feature points having similar feature amounts (differences less than a certain amount) are selected. When there are multiple methods for extracting feature points, by not matching feature points of different extraction methods with each other, erroneous matching is suppressed and accuracy is improved.

[0032] The relative velocity calculation unit 37 shown in FIG. 3 performs time differentiation of the coordinates (camera coordinate system) in the previous and next frames for the feature points pairs associated with each other in the previous and next frames, and calculates the relative velocity of the point corresponding to the feature points pair with respect to the camera 2.

[0033] The moment arm length estimation unit 38 estimates the moment arm length of the vehicle 1 from the relative speed of the feature points of the stationary object, vehicle behavior such as the speed and yaw rate of the vehicle 1 acquired by various sensors, and the angle of view of the camera 2. In the following description, the distance from the turning center of the vehicle 1 to the mounting position of the camera 2 (the center of the line segment connecting the pair of imaging units 20) is referred to as the moment arm length. The moment arm length estimation unit 38 estimates the moment arm length with high accuracy, making it possible to estimate the speed of a non-stationary object relative to the ground while the vehicle 1 is turning with high accuracy.

[0034] The relationship shown in the following equation (1) holds between the coordinates of feature points of a stationary object between frames and the moment arm length. R(θ)(P1+M)-M+D=P2 (1) Here, M is the moment arm length, P1 is the coordinate of the feature point of the stationary object in the previous frame (camera coordinate system), P2 is the coordinate of the feature point at the same point in the subsequent frame (camera coordinate system), D is the translational movement of vehicle 1 between frames, and R(θ) is the yaw rate of vehicle 1.

[0035] The yaw rate of the vehicle 1 is acquired by the yaw rate sensor 6. The translational movement amount of the vehicle 1 is acquired by the vehicle speed sensor 4. The coordinates of the characteristic points in the front and rear frames are known. Therefore, the moment arm length estimation unit 38 can estimate the moment arm length M from the above formula (1).

[0036] FIG. 7 is a diagram showing an example of an estimated moment arm length. The moment arm length estimation unit 38 stores the estimated moment arm length for each speed of the vehicle 1 at the time of estimation. Since the turning center of the vehicle 1 changes according to the speed of the vehicle 1, the moment arm length also changes according to the speed of the vehicle 1. Therefore, the moment arm length estimation unit 38 estimates a model quantitative value by fitting the actually estimated moment arm length and the speed of the vehicle 1 to a function model in which the relationship between the moment arm length and the speed of the vehicle 1 is modeled in advance. In this way, it becomes possible to appropriately estimate the moment arm length that changes according to the speed while suppressing the estimation error of the moment arm length. The moment arm length estimation unit 38 stores the relationship of the moment arm length according to the speed and calculates the moment arm length according to the speed of the vehicle 1, thereby enabling accurate calculation of the ground speed.

[0037] FIG. 8 is an explanatory diagram for explaining the definitions of distances Lp, Lc, and Lm. In FIG. 8, a turning center position 701 and a camera position 702 are illustrated superimposed on a vehicle 1. In the following explanation, a method for calculating the ground speed of an object 703 shown in FIG. 8 will be explained. The object 703 is located at a position distance Lp away from the turning center position 701. The object 703 is also located at a position distance Lc away from the camera position 702. The moment arm length Lm is the distance from the turning center position 701 to the camera position 702.

[0038] FIG. 9 is an explanatory diagram for explaining the turning angle and the relative movement amount of the object. In FIG. 9, the vehicle 1 at time t1 is indicated by reference numeral 1a, the object 703 at time t1 is indicated by reference numeral 703a, the vehicle 1 at time t2 after time t1 is indicated by reference numeral 1b, and the object 703 at time t2 is indicated by reference numeral 703b. Here, the position of the object 703 at time t2 indicated by reference numeral 703b is a virtual position on the assumption that the relative position between the object 703 and the camera 2 does not change between time t1 and time t2. In addition, the turning center position 801 and the camera position 802 at time t1, and the turning center position 804 and the camera position 805 at time t2 are illustrated superimposed on the vehicle 1. Reference numeral 806 indicates the turning angle of the vehicle 1 between time t1 and time t2.

[0039] If the object 703 is stationary, the object 703, which appears in the photographed image at the position indicated by the reference symbol 703a (a position distant from the camera position 802 by a distance Lc) at time t1, should be present at the position indicated by the reference symbol 703a at time t2. In this case, the camera 2 moves to the camera position 805 at time t2 and rotates by a rotation angle indicated by the reference symbol 806, so that the object 703 appears to have moved by the relative movement amount 807 in the photographed image. That is, at time t2, the relative speed calculation unit 37 calculates the relative speed of the object 703 corresponding to the relative movement amount 807 (time differential value of the relative movement amount 807). Therefore, the ground speed of the object can be calculated by subtracting the amount equivalent to the relative movement amount 807 from the relative speed calculated by the relative speed calculation unit 37.

[0040] 10 is an explanatory diagram for explaining a method for calculating the relative movement amount 807. The movement vector of the turning center of the vehicle 1 from time t1 to time t2 is expressed as a turning center movement vector 900. The movement vector of the position of the camera 2 from time t1 to time t2 is expressed as a camera position movement vector 902. Reference numeral 903 indicates the camera turning movement amount of the object 703. The camera turning movement amount 903 of the object 703 corresponds to the movement amount rotated by a turning angle 806 from the virtual position 703b. The turning angle 806 is calculated by multiplying the yaw rate (angle change per unit time) detected by the yaw rate sensor 6 by the elapsed time from time t1 to time t2.

[0041] The relative movement amount 807 of the object 703 is calculated by combining the camera rotation movement vector corresponding to the camera rotation movement amount 903 of the object 703 and the camera position movement vector 902. As described above, if the relative movement amount 807 can be calculated, the ground speed of the object 703 can be calculated from the relative speed of the object 703. The turning center movement vector 900 of the vehicle 1 is calculated from the traveling path of the vehicle 1 determined by the output values ​​of sensors such as the vehicle speed sensor 4, the vehicle steering angle sensor 5, and the yaw rate sensor 6. The camera position movement vector 902 is calculated by the camera movement amount calculation unit 40, which will be described later. As shown in FIG. 10, the camera position movement vector 902 has a subtle difference from the turning center movement vector 900 of the vehicle 1. If this difference cannot be calculated accurately, the ground speed cannot be calculated accurately.

[0042] 11 is an explanatory diagram illustrating the relationship between the turning center movement vector, the camera position movement vector, and the moment arm length of the vehicle 1. The difference between the turning center movement vector 900 and the camera position movement vector 902 of the vehicle 1 corresponds to an arc 909 obtained by cutting out the circumference of a circle with a radius of the moment arm length 908 by the turning angle 806. Therefore, if the moment arm length 908 can be estimated, the ground speed can be calculated from the relative speed of the object even when the vehicle 1 is turning.

[0043] FIG. 12 is a diagram showing the relationship between the assumed error in the moment arm length and the speed error. If an error occurs in the estimated value of the moment arm length, as shown in FIG. 12, an error occurs in the calculated ground speed according to the magnitude of the error. When the camera 2 is installed on the front wheels of the vehicle 1, the moment arm length is about half the length of the wheelbase, depending on the speed. The wheelbase refers to the length from the center of the front wheels to the center of the rear wheels. If the moment arm length is not taken into consideration, a ground speed error of 1 km / h or more may occur.

[0044] In addition, since the moment arm length also changes depending on the load of luggage, a similar error may occur even if the moment arm length is taken into consideration at the time of design. Therefore, if the moment arm length during actual driving is not taken into consideration, the movement of a pedestrian (non-stationary object) moving at a low speed cannot be accurately captured, and vehicle control such as warning and automatic braking cannot be performed appropriately. In this embodiment, this problem is solved by appropriately estimating the moment arm length.

[0045] 3 calculates the movement amount and movement vector of the camera 2 between frames. The camera movement amount calculation unit 40 calculates the camera movement amount and camera position movement vector 902 between frames from the moment arm length estimated by the moment arm length estimation unit 38 and the movement amount of the turning center of the vehicle 1 calculated from the outputs of sensors such as the vehicle speed sensor 4, the vehicle steering angle sensor 5, and the yaw rate sensor 6.

[0046] The ground speed calculation unit 39 calculates the ground speed of the stationary object and the non-stationary object detected by the object detection unit 34. The ground speed calculation unit 39 selects a moment arm length corresponding to the current speed of the vehicle 1 from the moment arm lengths for each speed estimated and stored by the moment arm length estimation unit 38. The ground speed calculation unit 39 calculates the ground speed of the non-stationary object, for example, by subtracting a speed corresponding to a relative movement amount 807 obtained by taking into account the selected moment arm length from the relative speed of the non-stationary object calculated by the relative speed calculation unit 37. As described above, the ground speed calculation unit 39 calculates the relative movement amount 807 by combining the camera position movement vector 902 and the camera rotation movement vector corresponding to the camera rotation movement amount 903 of the object 703.

[0047] 13 is a flowchart showing an example of the flow of processing executed by the arithmetic device 11 according to the first embodiment. This processing is repeatedly executed by the arithmetic device 11.

[0048] In step S100, the arithmetic device 11 determines whether or not it is necessary to reset the stored moment arm length. When there is a possibility that loading or unloading of luggage has occurred, the arithmetic device 11 determines that it is necessary to reset the stored moment arm length to an initial value. For example, when the engine ignition is turned on, when the vehicle is stopped for a certain period of time or longer, or when sensors such as a luggage compartment door opening / closing sensor, a weight sensor, or a suspension extension sensor detect that there is a high possibility that loading or unloading of luggage has occurred, the arithmetic device 11 determines that there is a possibility that loading or unloading of luggage has occurred.

[0049] If it is determined in step S100 that the stored moment arm length needs to be reset, the process proceeds to step S110. In step S110, computing device 11 resets the stored moment arm length. Then, the process proceeds to step S120. On the other hand, if it is determined in step S100 that the stored moment arm length does not need to be reset, the process proceeds from step S100 to step S120.

[0050] In step S120, the camera 2 captures an image, and a pair of image signals are input from the camera 2 to the in-vehicle object recognition device 3. In step S130, the image correction unit 31 performs luminance normalization processing and parallelization processing on the pair of image signals input in step S120. In step S140, the stereo matching unit 32 performs disparity calculation processing on the pair of image signals to generate a disparity image. In step S150, the road surface estimation unit 33 estimates a road surface area using the disparity image generated in step S140. In step S160, the object detection unit 34 detects objects (stationary objects and non-stationary objects) within a detection range set in front of the vehicle 1. In step S170, the feature point extraction unit 35 extracts feature points from the object detected in step S160. In step S180, the tracking unit 36 ​​associates feature points between frames. In step S190, the relative speed calculation unit 37 calculates the relative speed of the object detected in step S160. In step S200, moment arm length estimation unit 38 estimates the moment arm length, and stores it in association with the current speed.

[0051] The process shown in Fig. 13 is executed only while the vehicle 1 is turning. This is because in the first embodiment, the moment arm length can be estimated only during a turning operation. When the turning speed is slow, the estimation error is large, so it is desirable to execute the process when the turning speed reaches a predetermined constant or higher. Also, when the turning speed is changing, such as when the steering angle change amount is large, the estimation error is large, so it is desirable to stop the estimation operation.

[0052] In addition, in a situation where the vehicle 1 is driven under automatic driving control by LKAS (Lane Keep Assist System) or the like, when an object detectable as a stationary object such as a broken line or a zebra strip on a road is present near the vehicle 1, vehicle control involving a turning operation such as lane change may be intentionally performed in order to perform the estimation process of the moment arm length. When a stationary object is detected during automatic driving of the vehicle 1, the arithmetic device 11 outputs an execution command for the turning operation to a control device (not shown). The control device (not shown) executes vehicle control involving the turning operation of the vehicle 1 based on the execution command for the turning operation. The arithmetic device 11 increases the frequency of outputting the execution command for the turning operation of the vehicle 1 compared to when a stationary object is not detected. In other words, when a stationary object is detected during automatic driving of the vehicle 1, the arithmetic device 11 of the in-vehicle object recognition device 3 increases the frequency of vehicle control involving the turning operation of the vehicle 1 compared to when a stationary object is not detected. In this way, by increasing the frequency of vehicle control involving a turning operation, the frequency of estimating the moment arm length is increased, and the vehicle can be more appropriately controlled using the moment arm length during subsequent manual driving. Note that the vehicle control process may be executed by the in-vehicle object recognition device 3.

[0053] According to the above-described first embodiment, the following advantageous effects are achieved.

[0054] (1) The vehicle-mounted object recognition device 3 according to this embodiment is mounted on a vehicle 1. The vehicle 1 is equipped with a camera 2 having an imaging unit 20. The arithmetic unit 11 of the vehicle-mounted object recognition device 3 detects a stationary object based on an image captured by the imaging unit 20 of the camera 2, estimates a moment arm length, which is the distance from the turning center of the vehicle 1 to the imaging unit 20 of the camera 2, and calculates the ground speed of the object based on the estimated moment arm length. As a result, even if the imaging unit 20 of the camera 2 is installed at a position different from the turning center of the vehicle 1, the ground speed of the object captured by the imaging unit 20 of the camera 2 can be accurately calculated. Furthermore, even if the turning center of the vehicle 1 changes depending on the state of the load (for example, the weight or loading position of the load), the moment arm length can be estimated in real time and the ground speed of the object can be accurately calculated.

[0055] (2) The calculation device 11 estimates the moment arm length for each speed of the vehicle 1. Since this is done, it becomes possible to accurately estimate the moment arm length, which changes depending on the speed, in accordance with the speed of the vehicle 1.

[0056] (3) The arithmetic unit 11 calculates the ground speed of an object (e.g., a non-stationary object) based on the moment arm length corresponding to the current speed of the vehicle 1. In this manner, the ground speed of the object can be accurately calculated using the moment arm length that changes depending on the speed.

[0057] (4) The calculation device 11 estimates the moment arm length for each speed of the vehicle 1 by fitting the estimated moment arm length and the actual speed of the vehicle 1 to a function model that models the relationship between the moment arm length and the speed of the vehicle 1. Since this is done in such a way, when the moment arm length according to the speed can be modeled in advance, the variation in the estimated value can be suppressed by the model constraint force.

[0058] (5) The computing device 11 estimates the road surface and detects an object whose height difference with the estimated road surface is less than a certain level as a stationary object. This makes it possible to easily and reliably recognize white lines drawn on the road surface using only height information.

[0059] (6) The arithmetic unit 11 identifies the type of object and detects a predetermined type of object from among the identified objects as a stationary object. Since this is done, the stationary object can be easily and reliably recognized from the identification result.

[0060] (7) The computing device 11 matches the same object appearing in a pair of images captured by the imaging unit 20 at different times using features including position information, and calculates the relative speed between the vehicle 1 and the object from the time derivative of the object's position. This makes it possible to perform reliable matching using the camera 2, which is a stereo camera.

[0061] <Second embodiment> An on-vehicle object recognition device according to a second embodiment of the present invention will be described with reference to Fig. 14 and Fig. 15. Note that the same reference symbols are used for configurations that are the same as or equivalent to those described in the first embodiment, and differences will be mainly described.

[0062] Fig. 14 is a flowchart similar to Fig. 13, showing an example of the flow of processing executed by arithmetic device 11 according to the second embodiment. In the flowchart of Fig. 14, processing of steps S300 to S330 is executed instead of the processing of step S200 in the flowchart of Fig. 13. Furthermore, by resetting the moment arm length in step S110, the stored estimate value of the moment arm length is set to an initial value (default value).

[0063] In step S300, camera movement amount calculation unit 40 calculates the amount of movement of camera 2 between frames and camera position movement vector 902. In step S310, ground speed calculation unit 39 calculates the ground speed of the stationary object detected in step S160. In step S320, moment arm length estimator 38 executes an adjustment process for the estimated value of the moment arm length, and adjusts the estimated value of the moment arm length to a value closer to the true value.

[0064] If the current estimated value of the moment arm length is appropriate, the apparent ground speed of the stationary object calculated by the ground speed calculation unit 39 should be 0. This is because a stationary object is stationary, and therefore its ground speed is always 0. In other words, if the apparent ground speed of a stationary object is calculated and is not 0, this means that the estimated value of the moment arm length used to calculate the ground speed is inaccurate. If the estimated value of the moment arm length is shorter than the true value, the relative speed cannot be completely offset by the movement amount of camera 2 (relative movement amount 807) and some relative speed remains, and conversely, if the estimated value of the moment arm length is longer than the true value, the relative speed is excessively reduced by the movement amount of camera 2 (relative movement amount 807).

[0065] Therefore, in step S320, the moment arm length estimator 38 adjusts the current estimated value of the moment arm length to a longer value if the polarity of the calculated ground speed of the stationary object is the same as that of the relative speed, i.e., a value that is positively correlated. Conversely, if the polarity of the calculated ground speed of the stationary object is the opposite to that of the relative speed, i.e., a value that is negatively correlated, the moment arm length estimate is adjusted to a shorter value. Here, an initial value of the adjustment amount to be added to or subtracted from the estimated value of the moment arm length is set in step S110. The initial value of the adjustment amount may be set, for example, assuming a maximum unevenly loaded state of the vehicle 1 compared to a standard state in which the vehicle 1 is empty or the luggage compartment floor is evenly loaded with luggage, and in the latter state, the estimated value of the moment arm length is adjusted to a desired value by turning several times.

[0066] For example, assume that when the load is maximized as described above, the moment arm length deviates from the standard state by 1 m, and the process in Fig. 14 is repeated 50 times per second. In this case, if one rotation takes 3 seconds, the process in step S320 will be executed 150 times. If the adjustment amount is set to about 1 mm to 5 mm, it is expected that the estimated value of the moment arm length will converge to the true value after several rotations.

[0067] In step S330, moment arm length estimator 38 reduces the adjustment amount of the moment arm length described above in accordance with the total number of times the process in Fig. 14 has been executed. However, the adjustment amount does not become a negative value. By gradually reducing the adjustment amount, the true value is quickly converged upon at the start of processing, and erroneous estimation results are less likely to become apparent when the vehicle is traveling for a long period of time.

[0068] Fig. 15 is an explanatory diagram illustrating an example of the change in the estimated value of the moment arm length over time. At the start of the processing in Fig. 14, the estimated value of the moment arm length set by the moment arm length estimating unit 38 approaches asymptotically to true value 922. At the start of the processing, the estimated value approaches true value 922 within a short time, but the adjustment amount gradually becomes smaller, so the fluctuation in the estimated value of the moment arm length becomes smaller over time. When there is a possibility of unloading luggage, such as when the ignition is turned off, the estimated value of the moment arm length and the adjustment amount return to their initial values. Then, in the same way as before, the estimated value approaches asymptotically to true value 923, which is different from the previous value.

[0069] In this way, the estimated value of the moment arm length gradually approaches the true value, so that even if each estimate is incorrect, the adverse effect of using a significantly off-point estimate can be suppressed, and the time it takes to approach the true value within a certain range is kept within an acceptable range. Even if the true value of the moment arm length changes due to loading or unloading of cargo, the estimated value of the moment arm length is adjusted to a corresponding value in a short period of time. This enables accurate estimation of the ground speed.

[0070] According to the above-described second embodiment, the following advantageous effects are achieved.

[0071] (1) The calculation device 11 estimates a moment arm length at which the apparent ground speed of a stationary object is equal to or less than a predetermined value. For example, the calculation device 11 estimates a moment arm length at which the ground speed of a stationary object is 0 (zero). In this manner, it is possible to reliably reduce the estimation error of the moment arm length.

[0072] The following modified examples are also within the scope of the present invention, and it is also possible to combine the configurations shown in the modified examples with the configurations described in the above-mentioned embodiments, or to combine the configurations described in the different embodiments described above.

[0073] <Variation 1> The estimated moment arm length may be stored for each fixed speed zone, rather than for each speed of the vehicle 1. For example, the calculation device 11 calculates the ground speed of the object by taking the average value of the moment arm lengths estimated for each predetermined speed zone as the moment arm length for that speed zone.

[0074] Fig. 16 is a diagram showing an example of an estimated moment arm length. As shown in Fig. 16, the speed of vehicle 1 is divided into certain intervals. Then, the average value of the moment arm lengths estimated for each speed interval, shown by the dotted lines in Fig. 16, is treated as the moment arm length for that speed interval. In this way, it is possible to accurately calculate the ground speed of an object captured by camera 2 without investigating the behavior of each vehicle in advance in a situation where a model of the moment arm length for the speed cannot be defined in advance, such as for an unknown vehicle whose vehicle behavior cannot be determined.

[0075] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0076] 1...vehicle, 2...camera, 3...vehicle-mounted object recognition device, 11...arithmetic unit, 12...non-volatile memory, 13...volatile memory, 14...input / output interface, 31...image correction unit, 32...stereo matching unit, 33...road surface estimation unit, 34...object detection unit, 35...feature point extraction unit, 36...tracking unit, 37...relative speed calculation unit, 38...moment arm length estimation unit, 39...ground speed calculation unit, 40...camera movement amount calculation unit

Claims

1. An in-vehicle object recognition device including a computing device, the device being mounted on a vehicle having an imaging unit, The computing device includes: Detecting a stationary object based on an image captured by the imaging unit; A moment arm length, which is a distance from a turning center of the vehicle to the imaging unit, is estimated; An in-vehicle object recognition device that determines the ground speed of the object based on the estimated moment arm length.

2. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that estimates a moment arm length at which the apparent ground speed of the stationary object is equal to or less than a predetermined value.

3. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that estimates the moment arm length for each speed of the vehicle.

4. The vehicle-mounted object recognition device according to claim 3, The computing device is an in-vehicle object recognition device that calculates the ground speed of an object based on the moment arm length corresponding to the current speed of the vehicle.

5. The vehicle-mounted object recognition device according to claim 3, The computing device is an in-vehicle object recognition device that calculates the ground speed of an object by using an average value of the moment arm lengths estimated for each specified speed zone as the moment arm length for that speed zone.

6. The vehicle-mounted object recognition device according to claim 3, The computing device is an in-vehicle object recognition device that estimates the moment arm length for each speed of the vehicle by fitting the estimated moment arm length and the actual speed of the vehicle to a function model that models the relationship between the moment arm length and the speed of the vehicle.

7. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that estimates a road surface and detects an object whose height difference from the estimated road surface is less than a certain level as the stationary object.

8. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that identifies a type of object and detects a predetermined type of object from among the identified objects as the stationary object.

9. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that matches the same object that appears in a pair of images captured by the imaging unit at different times using features including position information, and calculates the relative speed between the vehicle and the object from the time derivative of the object's position.

10. The vehicle-mounted object recognition device according to claim 1, The computing device is an in-vehicle object recognition device that increases the frequency of vehicle control involving turning operations of the vehicle when the stationary object is detected during autonomous driving of the vehicle compared to when the stationary object is not detected.

Images