External recognition device

Abstract

To provide an environment recognition device which can track a moving object with high accuracy on the basis of images captured by a camera.SOLUTION: An environment recognition device 10 comprises: an object position detection unit which detects a position of a moving object as an observation value on the basis of an image captured by a camera provided on a self vehicle; a region determination unit which determines a region to which the moving object belongs in the image on the basis of the detected observation value; an observation error setting unit which calculates an error for the observation value; a state prediction unit which predicts a state of the moving object at a current time on the basis of observation values detected by a preceding time and the error calculated by the observation error setting unit; an associating unit which associates the predicted state of the moving object at the current time and the observation value detected by the object position detection unit with each other; and a state updating unit which updates the state of the moving object on the basis of an association result of the associating unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an external environment recognition device. [Background technology]

[0002] Currently, various driving assistance systems have been developed and commercialized. One of these driving assistance systems is the collision damage mitigation braking system. The collision damage mitigation braking system needs to accurately detect objects (moving bodies) moving around the vehicle using sensors mounted on the vehicle and track the moving bodies.

[0003] A method using a Kalman filter is known as a method for tracking a moving object, which can predict the position of a moving object at a time after a predetermined time based on the change in the position of the moving object up to a predetermined time.

[0004] However, with this method, if the observation error for the obtained observation (the position of the detected moving object) cannot be set accurately, the accuracy of the prediction decreases. When using information on a moving object detected based on an image, the magnitude of the observation error changes depending on the position where the moving object is detected in the image. Also, depending on the angle of view and mounting angle of the camera that captures the image, the moving object to be tracked may fall outside the image and cannot be detected. In these cases, tracking of the moving object may fail, and the vehicle may collide with the moving object.

[0005] The following method has been proposed as a method for tracking moving objects, particularly people, using a camera attached to a moving object (see, for example, Patent Document 1).

[0006] That is, a camera device attached to a moving object captures images of the surroundings of the moving object within the field of view of the camera device at regular time intervals, and an image processing system identifies pixels corresponding to the moving object to be tracked based on multiple images taken in chronological order.

[0007] For each moving object, a pixel is identified that represents the projection into image coordinates of the moving object corresponding to the spatial point where the moving object touches the road surface. The motion of the spatial point where the moving object touches the road surface is tracked by a state estimator using a state vector of at least four dimensions whose components are the position of the spatial point where the moving object touches the road surface and the relative velocity of the road surface.

[0008] The state estimator tracks the motion by generating a prediction for each state vector, transforming this prediction into image coordinates via a suitable projection equation, calculating the expected error in this prediction in image coordinates via a covariance matrix, and updating this prediction by comparing it with pixels extracted in more recent images. This motion tracking tracks the moving object. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2009-089365 A Summary of the Invention [Problem to be solved by the invention]

[0010] Systems such as collision damage mitigation braking systems that calculate the distance from the position of surrounding moving objects to the vehicle's own vehicle to estimate the time until collision and automatically apply the brakes based on the estimated time when there is a risk of collision need to accurately measure and track the distance to moving objects. Therefore, in order to realize a collision damage mitigation braking system, it is important not only to track moving objects continuously, but also to avoid tracking erroneous detection results.

[0011] When the technology proposed in Patent Document 1 is applied to an automatic parking system, the expected error in the observed value becomes large in the vicinity of the vehicle, and the vehicle's immediate position is associated with a misaligned detection result. As a result, the distance to the moving object being tracked cannot be measured accurately, and the vehicle may collide with the moving object.

[0012] The present invention has been made in consideration of the above circumstances, and has an object to provide an external environment recognition device that can track moving objects with high accuracy based on images acquired by a camera. [Means for solving the problem]

[0013] The present invention is an external environment recognition device that detects and tracks moving objects around a moving object equipped with a camera, and includes an object position detection unit that detects the position of the moving object as an observation value based on an image captured by the camera, an area determination unit that determines an area in the image to which the moving object belongs based on the observation value detected by the object position detection unit, an observation error setting unit that calculates an error for the observation value based on a result of the determination by the area determination unit, a state prediction unit that predicts a state of the moving object at the current time based on the observation value up to a previous time that is a time before the current time and the error calculated by the observation error setting unit, a correspondence unit that corresponds the state of the moving object at the current time predicted by the state prediction unit and the observation value detected by the object position detection unit, and a state update unit that updates the state of the moving object based on a result of the correspondence by the correspondence unit. Effect of the Invention

[0014] According to the external environment recognition device of the present invention, it is possible to track a moving object with high accuracy by using an image acquired by a camera. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an external environment recognition device according to an embodiment of the present invention. [Diagram 2] 1 is a schematic diagram showing an example of a person and a vehicle as detection results of a moving object. [Diagram 3] FIG. 13 is a diagram illustrating the definition of the position of the detection result in the image. [Figure 4] 1 is a flowchart showing the flow of processing in the first embodiment. [Figure 5A]FIG. 1 is a diagram illustrating the definition of the world coordinate system. [Figure 5B] FIG. 2 is a diagram illustrating a definition of a moving body coordinate system. [Figure 6] FIG. 2 is a diagram showing an example of divided regions of an image. [Figure 7A] FIG. 13 is a schematic diagram showing a state in which an image is scanned in the vertical direction as an example of a criterion for dividing an area. [Figure 7B] This graph is a curved approximation of the relationship calculated to determine how much the distance to a person changes with each change of 1 pixel in the y coordinate in the image by scanning Figure 7A. [Figure 8A] 13 is an overhead image showing an example in which a detection target (for example, a person) is relatively far away from the host vehicle. [Figure 8B] 13 is an overhead image showing an example in which a detection target (for example, a person) is relatively close to the host vehicle. [Figure 8C] 8C is a diagram of a moving body coordinate system onto which the detection target shown in FIG. 8A and the detection target shown in FIG. 8B are projected. [Figure 9] 13 is a diagram showing a method of calculating a distance from a position in an image of a detection result to the vehicle itself; FIG. [Figure 10] FIG. 13 is a diagram showing an example of association. [Figure 11] FIG. 13 is a diagram showing an example of a cut-off result. [Figure 12] 13 is a flowchart of an update process of a Kalman filter. [Figure 13] FIG. 13 is a diagram illustrating an example of a state management table of the Kalman filter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, specific embodiments of the external environment recognition device according to the present invention will be described with reference to the drawings.

[0017] [Embodiment 1] FIG. 1 is a block diagram showing an external environment recognition device 10. The external environment recognition device 10 shown in the figure is an embodiment of the external environment recognition device according to the present invention. The external environment recognition device 10 is mounted on a moving body such as an automobile. Hereinafter, the moving body on which the external environment recognition device 10 is mounted will be referred to as the host vehicle in the following description.

[0018] The external environment recognition device 10 is used in combination with an external system 40. The external environment recognition device 10 operates at a predetermined cycle, for example, every 100 ms. The external environment recognition device 10 has a function of acquiring vehicle information such as speed, steering angle, shift and brake states from the vehicle, and a function of acquiring images captured by a camera.

[0019] As shown in Fig. 1, the external environment recognition device 10 includes an object position detection unit 11, an area determination unit 12, an observation error setting unit 13, a matching unit 14, a state prediction unit 15, and a state update unit 16. The external environment recognition device 10 includes, for example, a CPU, a GPU, a RAM, a ROM, and the like, and realizes the functions of the object position detection unit 11 and the like by expanding a predetermined program stored in the ROM into the RAM and executing it with the CPU. Note that the external environment recognition device 10 may realize some or all of its functions using hardware such as an FPGA or an ASIC.

[0020] The external environment recognition device 10 receives as input images 20 captured by a camera attached to the vehicle and movement information 30 acquired from the vehicle via a CAN or the like. The images 20 are acquired sequentially at a predetermined frame rate by a camera mounted on the vehicle. The movement information 30 is information related to the movement of the vehicle and information equivalent to information related to the movement acquired from the vehicle, such as vehicle speed, steering angle, vehicle turning radius, accelerator and brake states, and shift lever state. The movement information 30 is also acquired sequentially at a predetermined cycle, similar to the images 20.

[0021] The object position detection unit 11 detects a desired moving object (moving object) such as a person or a vehicle by using a known method such as pattern matching based on the image 20. Fig. 2 is a schematic diagram showing a person and a vehicle as examples of the detection results of a moving object.

[0022] 2, the detection result of the moving object is information for specifying the position of the moving object (a person or a vehicle) in an image. Specifically, the information is regulated by a height H which is a vertical (length) dimension of the rectangular frame F, a width W which is a horizontal (width) dimension of the rectangular frame F, and a position of an origin C of the rectangular frame F which is a specific corner of the rectangular frame F (the upper left corner in this embodiment), which defines a rectangular frame F surrounding the moving object. The object position detection unit 11 outputs the detection result to the area determination unit 12 and the association unit 14.

[0023] 3 is a schematic diagram for explaining the position of a detected moving object in an image 20 (the upper left corner is the image origin O(0,0), the horizontal right direction is the positive direction of the x-axis, and the vertical downward direction is the positive direction of the y-axis). The region determination unit 12 determines, based on the detection result input from the object position detection unit 11, to which region in the image 20 the detected moving object (detected moving object) belongs when the image 20 is divided into a plurality of regions.

[0024] 3, the position of the detected moving object is represented by the horizontal center point (a position at the height H of the rectangular frame F and half the width W of the rectangular frame F from the origin of the rectangular frame F) of the lower edge of the rectangular frame F identified by the above-mentioned object position detection unit 11. Then, the area determination unit 12 outputs the area in the image 20 to which this representative point belongs to the observation error setting unit 13 as the position of the moving object.

[0025] The observation error setting unit 13 determines an error assumed to be included in the observation value (the position of the moving object detected by the object position detection unit 11 based on the image 20) based on the result of the area determination unit 12. Furthermore, when the observation error setting unit 13 determines based on the result of the area determination unit 12 that the area cannot be detected by pattern recognition using pattern matching due to a cut-off part or part of the leg that is not captured in the image, such as a part of the foot (foot or bottom of the tire) of the detected moving object, the observation error setting unit 13 sets a value indicating that the detection is invalid. Then, the observation error setting unit 13 outputs the set error or the value indicating invalidity to the association unit 14.

[0026] The matching unit 14 matches the position of the moving object at the current time (predicted moving object position) predicted by the state prediction unit 15 of the state estimator based on the detection result of the object position detection unit 11 and the observation error set by the observation error setting unit 13. At this time, if there is no moving object predicted position to be matched, it is regarded as a newly detected moving object (newly detected moving object). The matching unit 14 outputs the result of the matching and information on the newly detected moving object to the state update unit 16.

[0027] The state prediction unit 15 predicts the state of the moving object at the current time based on the state of the detected moving object obtained by updating in the state update unit 16, using the result of detection by the object position detection unit 11 up to a time before the current time (the previous time). Here, the state has at least a position and a speed. The prediction result by the state prediction unit 15 is used in the association unit 14.

[0028] The state update unit 16 updates the state using the result of associating the results of the state prediction unit 15 and the object position detection unit 11 in the association unit 14, and starts tracking the moving object that has been determined as a newly detected moving object. The updated state is output to the external system 40, and the external system 40 uses this result in a system that measures the time required until a collision based on, for example, the distance between the vehicle and a person (an example of a moving object) and automatically applies the brakes so that the vehicle can be stopped within the time required until the collision.

[0029] Next, the operation of the external environment recognition device 10 will be described with reference to a flowchart. Fig. 4 is a flowchart showing the overall flow of processing by the external environment recognition device 10. In addition, this embodiment will be described assuming that a camera is mounted on the rear of the vehicle.

[0030] Prior to describing the flowchart, the coordinate systems (world coordinate system, moving body coordinate system) used in the processing of the external environment recognition device 10 will be described.

[0031] In FIG. 5A, the position of the vehicle when the external system 40 is started is set as the origin, and the rear of the vehicle when the external system 40 is started is set as +x w , +x w +y side w , +x w From +y w Rotate to +θ w FIG. 5B is a diagram showing a world coordinate system in which the values ​​are on a two-dimensional xy plane, and FIG. 5B is a diagram showing a world coordinate system in which the rear of the host vehicle is +x v , +x v +y side v , +x v From +y v Rotate to +θ v This is a diagram showing a moving body coordinate system in which the values ​​are on a two-dimensional xy plane. The moving body coordinate system is the same as the world coordinate system in the positive direction of the x axis, the positive direction of the y axis, and the positive rotation direction of θ, but the origin moves with the movement of the host vehicle.

[0032] Next, a description will be given of the flowchart in Fig. 4. The overall process of the flowchart starts when the external system 40 is started up.

[0033] The example shown in the flowchart of FIG. 4 will be described using a monocular camera (hereinafter, simply referred to as the camera) attached to the host vehicle facing backward. The period of data acquired from this camera and the host vehicle is 33 [ms]. The size of the image acquired by the camera is 640 [pixels] horizontal (width) × 400 [pixels] vertical (height). The vehicle information is information required to calculate how the host vehicle has moved. Furthermore, a model that assumes uniform motion is used as a model for predicting the position of the detected moving object.

[0034] In step 101 (S101), the object position detection unit 11 detects a moving object that can be a tracking target based on the image 20 read in S100. Here, the moving object to be tracked is assumed to be, for example, a person or a vehicle.

[0035] In this embodiment, a moving object that can be a tracking target is detected using known methods, Histograms of Oriented Gradients (HoG) and Support Vector Machine (SVM).

[0036] Here, HoG is a histogram created by calculating the gradient strength for each gradient direction of pixel values ​​in a local region, and this histogram is called an HoG feature. Since HoG features use gradients as features, they have the advantage of being robust against scale changes in the image 20. On the other hand, HoG features have the disadvantage of being vulnerable to rotation changes.

[0037] SVM is a classifier that classifies inputs as correct or incorrect. Based on a large amount of data and label pairs, SVM learns the relationship between data and labels, and learns boundaries for classifying correct and incorrect answers. When unknown data is input, it uses the learned boundaries to determine whether the input data is correct or incorrect.

[0038] In this embodiment, a classifier is created by learning HoG features and the labels of people and vehicles. As shown in Fig. 2, the detection result assumes the origin of the detection result, width W and height H for drawing a rectangular frame F surrounding the detection result, and labels such as people and vehicles.

[0039] Each of these techniques is well known, and detailed description thereof will be omitted. In addition, other well-known techniques such as pattern recognition may be used as a technique for detecting people and vehicles, and the detection technique is not limited to the above-mentioned technique. In addition, moving objects other than people and vehicles may be detected and tracked.

[0040] In S101 of the flowchart, the position of a moving object in image 20 is detected based on the moving object detection result as shown in Fig. 3. As shown in Fig. 3, the coordinates of image 20 are +y in the downward direction of image 20 and +x in the rightward direction of image 20. The position (x, y) in the image is calculated by adding width W / 2 (x-axis direction) and height H (y-axis direction) to the image origin O (0, 0).

[0041] FIG. 6 is a diagram showing an example of divided areas A1, A2, A3, and A4 of image 20. FIG. 7A is a schematic diagram showing how image 20 is scanned vertically as an example of a criterion for dividing areas. FIG. 7B is a graph showing a curved approximation of the correspondence between the change in distance to a person for each change of 1 pixel in the y coordinate in image 20 by scanning FIG. 7A.

[0042] The determination of the region in S102 is a process of determining which region the position of the detection result (moving object) belongs to when the image 20 is divided into, for example, four vertical regions A1, A2, A3, and A4 as shown in Fig. 6. The division into regions A1 to A4 is based on, for example, the distortion of the camera lens and the mounting attitude of the camera on the vehicle, and the distance error corresponding to a deviation of 1 [pixel] is determined for each position in the y-axis direction in the image 20 based on the curve in Fig. 7B.

[0043] In the example of Fig. 6, the area is divided into four areas A1, A2, A3, and A4. Area A1 is an area where a 1-pixel shift in the position of a detected moving object will result in a distance error of several meters or where the sky is reflected, area A2 is an area where a 1-pixel shift in the position of a detected moving object will result in a relatively large error, smaller than area A1, area A3 is an area where a 1-pixel shift in the position of a detected moving object will result in an error smaller than area A2, and area A4 is an area where the distance between the vehicle and the detection target is physically close, and it is considered that the detection target cannot be detected correctly.

[0044] In addition, the division criteria (boundaries) into the four regions A1, A2, A3, and A4 were set as follows: region A1 is above the vanishing point in image 20 (see Figure 6); region A2 is from the point where the detected position on the moving body coordinate system changes at a distance of approximately 0.8 [m] with an error of 1 [pixel] to region A1; region A3 is from the point where the detected position on the moving body coordinate system changes at a distance of approximately 0.2 [m] with an error of 1 [pixel] from region A2 to region A2; and region A4 is below region A3.

[0045] The area determination unit 12 determines which of areas A1 to A4 the position of the detection result in the image 20 is in (S102), and outputs the result to the observation error setting unit 13. Furthermore, a detected moving object that is in a position with a very large detection error, such as area A1, or a detected moving object that is not considered to be located on the ground, is not considered as a tracking target and is deleted.

[0046] Here, the method of dividing the areas A1 to A4 is not limited to using physical parameters such as the distortion of the camera lens or the mounting position of the camera. The divided areas A1 to A4 may be mixed with those divided horizontally as well as vertically, and the number of divisions is not limited to four.

[0047] In S103, the position of the moving object being tracked in the world coordinate system at the current time is predicted by a state estimator. The state estimator may be a Kalman filter, an extended Kalman filter, an unscented Kalman filter, or the like, but in this embodiment, an example using a Kalman filter will be described.

[0048] First, since the Kalman filter in this embodiment uses a uniform motion model, the state to be estimated is a four-dimensional vector x = (x, y, vx, vy). Furthermore, x and y indicate a position in the world coordinate system, and vx and vy indicate a velocity in the world coordinate system. The position in the world coordinate system at the current time and the value of the error covariance matrix for this position are predicted by the following equations (1) and (2).

number

number

[0049] Here, the subscript t indicates time, - indicates a predicted value, and + indicates a value updated based on observation. In addition, in formulas (1) and (2), F indicates a state transition matrix, and Q indicates system noise, which are expressed by formulas (3) and (4) in this embodiment.

number

number

[0050] Δt in equation (3) is the processing period, and σ in equation (4) 2 sysytemnoise is an error expected when the model of this embodiment is applied. The process of S103 is performed for all Kalman filters that are not in an invalid state.

[0051] In S104, movement information 30 is acquired from the vehicle by any method such as CAN (Control Area Network). Here, the movement information 30 is information required to calculate the movement distance in S105. For example, the movement information 30 is the speed of the vehicle, the number of rotations of the tires, the steering amount of the steering wheel, etc.

[0052] In S105, the position of the host vehicle is estimated using the movement information 30 acquired in S104. In this embodiment, the vehicle position is estimated according to the formulas (5) to (8) using the movement distance calculated from the tire rotation speed and tire diameter, and the turning radius estimated from the steering amount of the steering wheel. Here, it is assumed that the host vehicle moves on a two-dimensional plane, and the state is pose. car =(x carpose ,y carpose ,θ yaw )

number

number

number

number

[0053] Here, t represents time, R represents the turning radius, ρ represents the curvature, and ΔS represents the travel distance. As a method for estimating the position of the vehicle, a method using another model or Visual Odometry that estimates the position and attitude of the vehicle using the image 20 may be used.

[0054] In S106, an error in the observation result is set based on the result of the area determination. In this embodiment, for areas A1 and A4, an invalid value is set as an error in the observation result as an invalid detection result. For area A2, an error is dynamically set as shown in FIG. 8. The procedure for setting the error is to first add the errors x and y of the detection position assumed on the image to the coordinates in the image 20 of the detection target. After that, the coordinates in the image 20 to which the error has been added are projected onto the moving body coordinate system using the camera attitude and the height of the camera from the ground according to a pinhole camera model.

[0055] Figure 8A is an overhead image showing an example of a case where the detection target (e.g., a person) is relatively far away from the host vehicle, Figure 8B is an overhead image showing an example of a case where the detection target (e.g., a person) is relatively close to the host vehicle, and Figure 8C is a diagram of a moving body coordinate system onto which the detection target shown in Figure 8A and the detection target shown in Figure 8B are projected.

[0056] When the coordinates in image 20 with the error added are projected onto the moving body coordinate system, if the detection target (e.g., a person) is relatively far away from the host vehicle in the overhead image of Fig. 8A, the detection target is projected at a position far from the host vehicle as shown in Fig. 8C. On the other hand, if the detection target is relatively close to the host vehicle in the overhead image of Fig. 8B, the detection target is projected at a position close to the host vehicle as shown in Fig. 8C.

[0057] The image center position projected onto this moving body coordinate system (Figure 8C) with expected error added is taken as the value of the observation error, and the average of the differences in the x direction projected in the forward / backward direction relative to the vehicle is taken as the value of the observation error in the x direction, and the average of the differences in the y direction projected in the left / right direction is taken as the value of the observation error in the y direction.

[0058] In order to model the movement of the detection target by uniform motion, the measurement error is expressed as follows: x is the observation error in the forward and backward directions from the viewpoint of the vehicle, σ y is the observation error in the left and right direction as seen from the vehicle.

number

[0059] In addition, if a change in the x coordinate in the image causes a change in the y position on the moving body coordinate system, or conversely, if a change in the y coordinate causes a change in the x position on the moving body coordinate system, the covariance value may be set as follows:

number

[0060] In area A3, if the observation error is set in the same way as area A2, the value may become too small. For this reason, after calculating the observation error in the same way as area A2, it is compared with a preset minimum value, and if it is smaller than the minimum value, the minimum value is used.

[0061] Dynamically estimating and setting the observation error as described above has the effect of improving the ability to track distant targets and reducing the ability to track targets that are erroneously detected near the vehicle.

[0062] The setting of the observation error in S106 is performed for all detection results in S101 that are processed in S102.

[0063] In S107, the position of the moving object being tracked at the current time predicted by the state estimator and the position of the detection target in the image 20 detected from the image 20 are associated with each other based on the result of projecting them onto the world coordinate system.

[0064] Fig. 9 is a diagram showing a method for calculating the distance from the position in the image of the detection result to the vehicle. As shown in Fig. 9, the calculation of the world coordinate system uses the height from the ground to the origin of the camera mounted on the vehicle, the camera mounting position and attitude relative to the vehicle, and the camera's internal parameters, and is based on a pinhole camera model and the assumption that the detected moving object exists on the same plane as the vehicle, and calculates the intersection of the line connecting the camera origin and the position in the image 20 with the ground as the foot position (x car ,y car ) is calculated as

[0065] Next, the foot position on the moving body coordinate system obtained by the calculation is transformed into the world coordinate system. Since the world coordinate system is a coordinate system whose origin is the position at the time when the external system 40 is started, the transformation can be performed based on the position and attitude of the host vehicle at the current time. For this reason, the (x carpose ,y carpose ) is added to the foot position in the moving body coordinate system obtained by the method shown in FIG. 9, thereby converting from the moving body coordinate system to the world coordinate system.

[0066] In addition, the observation error value is also rotated according to the direction of the vehicle. When the present invention is implemented using a fixed camera, the world coordinate system and the moving body coordinate system defined in Fig. 5 always coincide with each other, so there is no need to implement the formulas (5) to (8) and the formulas (11) and (12).

number

number

[0067] An example of matching is shown in Fig. 10. After coordinate transformation is performed on all results detected from the image, matching is performed based on the position of the moving object being tracked at the current time predicted by the state estimator and the position of the target detected in the image 20 on the world coordinate system, as shown in Fig. 10.

[0068] In this embodiment, the Mahalanobis distance, which is a type of distance used in statistics, is used as an index of distance for matching. The Mahalanobis distance is an index suitable for clarifying the similarity of new data based on the variance-covariance value of a certain data sample. The calculation of the Mahalanobis distance is shown below.

number

number

[0069] Here, dist is the Mahalanobis distance, z pred is the predicted position (x pred ,y pred ), z detect is the position of the moving object detected in image 20 (x world ,y world This process is repeated the number of times equal to the product of (the number of predictions of the Kalman filter) and (the number of moving objects detected from the image), and the Kalman filter prediction value that minimizes the Mahalanobis distance is associated with the moving object detected from the image 20.

[0070] At this time, a plurality of Kalman filters may be associated with one moving object detected from the image 20. By associating a plurality of Kalman filters with one moving object detected from the image 20, unnecessary deletion of Kalman filters is prevented.

[0071] Other than the above, known methods such as the Hungarian algorithm may be used as a matching method. Euclidean distance may be used as a matching index.

[0072] Fig. 11 is a diagram showing an example of the result of cut-off. In S107, the Kalman filter associated with the area (A4 in this embodiment) where cut-off has occurred and the entire object is not captured as a result of the area determination in S102, as shown in Fig. 11, attempts detection using alternative means such as optical flow and median flow using the predicted value of the Kalman filter.

[0073] If detection is possible using the alternative means, the position on the world coordinate system and the Mahalanobis distance are calculated in the process of S107 based on the result of detection by the alternative means, and are used as the observation value.

[0074] Detection by optical flow is done by using the internal state x of the predicted value of the Kalman filter. - Of =(x, y, vx, vy), x and y are projected onto the image 20 to draw a rectangular frame F, and the velocity vector within the rectangular frame F is calculated. The width W and height H required to draw the rectangular frame F can be obtained by using the height H and width W of the observation value last associated with the Kalman filter.

[0075] Here, if the alternative method fails to detect the object, it is determined that there is no observed value. Methods such as optical flow are well known, and therefore a description thereof will be omitted.

[0076] In S108, the state of the moving object being tracked is updated using the detection result associated with the predicted value of the Kalman filter. In this embodiment, the state of the Kalman filter is managed in five states: tracking start, tracking candidate, tracking, interpolating, and invalid.

[0077] The tracking start initializes the state of the Kalman filter to start tracking a newly detected moving object in the state when the moving object is detected. The tracking candidate indicates a state in which not much time has passed since the start of detection and the internal state of the Kalman filter is likely to be unstable.

[0078] "Tracking" refers to a state in which time has passed since detection began and the internal state of the Kalman filter is stable. "Interpolating" refers to a state in which the tracked target has gone undetected for some reason after tracking of the detected moving object began. "Invalid" refers to a state in which the target Kalman filter is not tracking anything. The internal state of the Kalman filter is assumed to be the internal state x, the error covariance matrix P, and the number of interpolations Intp.

[0079] Fig. 12 is a flowchart of the Kalman filter update process. As shown in Fig. 12, first, in S201, it is confirmed whether or not there is a detection result (observation value) corresponding to the Kalman filter. If there is an observation value, the process proceeds to S202, and if there is no observation value, the process proceeds to S207.

[0080] In S202, a rejection test is performed using the Mahalanobis distance calculated by equation (14) based on the predicted value and observed value of the Kalman filter. The rejection test was performed using the x-squared test. The x-squared test is a test in which the test statistic is x under the null hypothesis. 2 This is a statistical test that assumes that the data follows a distribution.

[0081] In this embodiment, the Mahalanobis distance is calculated from (x, y), resulting in an x-squared distribution with a degree of freedom of 2. In this embodiment, the significance level is set to 5% so the x-squared value is 5.99146. If the Mahalanobis distance is equal to or greater than 5.99146 (or exceeds 5.99146), it is determined that the association is unlikely based on the predicted value and the process proceeds to S203, whereas if the Mahalanobis distance is less than 5.99146 (or equal to or less than 5.99146), the process proceeds to S210.

[0082] Furthermore, any correspondence with an x-squared value of 5.99146 or more is considered to be an invalid correspondence, and the corresponding observation value is treated as a newly detected moving object.

[0083] Fig. 13 is a diagram showing an example of a state management table of the Kalman filter. S203 is a process in the case of a newly detected moving object, and as shown in Fig. 13, an invalid state is searched for from a table (t-1) held within the program.

[0084] If there is an invalid value (=empty space in the table) (Yes in S204), the process proceeds to S205, and if there is no invalid value (No in S204), the process proceeds to S206. In S205, a new Kalman filter is registered in the invalid value location in the table for tracking a newly detected moving object, and the internal state x and error covariance matrix P are initialized, after which the state of the table is set to start tracking.

[0085] To initialize the internal state, the world coordinate system of the observed value is substituted as follows. The velocity component is set to 0. A preset value, for example, 0.1, is substituted for the error covariance matrix P. For example, 0 is substituted for the number of interpolations Intp.

number

[0086] S206 is a process to be performed when there is no free space in the table. When there is no free space in the table, the moving object being tracked that is considered to be the most unnecessary according to a predetermined criterion is deleted, and instead, a newly detected moving object is registered (registration of a newly detected moving object).

[0087] In this embodiment, first, the Euclidean distance between the newly detected moving object and the vehicle is compared with the Euclidean distance between the moving object being tracked and registered in the table and the vehicle itself. If the moving object being tracked registered in the table has the largest Euclidean distance, the moving object being tracked that is farthest from the vehicle itself is determined to have the lowest possibility of collision, and is deleted as the moving object being tracked that is considered to be the least unnecessary. Then, the newly detected moving object is registered by the same process as in S205.

[0088] If the farthest object is a newly detected moving object, the process ends without registering the moving object. Here, the condition for deleting the moving object from the table is not limited to the magnitude of the Euclidean distance, and other conditions such as the state of the moving object being tracked (interpolation in progress, etc.) may be applied.

[0089] In S207, it is determined whether or not to perform an interpolation process on a Kalman filter that is not associated with an observed value. The determination is made based on the number of times of interpolation. In this embodiment, if the number of times of interpolation is, for example, 3 or more, the process proceeds to S208, and if the number of times is less than 3, the process proceeds to S209.

[0090] In S208, since the number of interpolations exceeds a threshold value, it is determined that the target to be tracked has been lost for some reason, such as by moving out of the shooting range, and the state of the corresponding Kalman filter table is changed from being interpolated to an invalid value to exclude the target from tracking.

[0091] In S209, the predicted value of the object being tracked predicted in S103 is updated as the predicted value, and the number of interpolation times is incremented by 1.

number

number

number

[0092] In S210, the process is determined based on the state of the Kalman filter. If the state is tracking, the process proceeds to S211-a. If the state is tracking start, tracking candidate, or interpolation, the process proceeds to S211-b. In S211-a and S211-b, the internal state is updated based on the observation value associated with the Kalman filter. First, the observation error R′ and the prior error covariance matrix P - , the observation matrix H, and the world coordinates z of the observations are used to calculate the Kalman gain K, where I represents the unit matrix.

number

number

number

number

[0093] In S211-a, the process ends. In S211-b, the process proceeds to S212. When the process up to this point has been performed, the tracking start state is changed to a tracking candidate state, and if the initial speed has already been calculated in the interpolation state, the state is changed to a tracking state, and if the initial speed has not been calculated, the state is changed to a tracking candidate state, and the table value is updated.

[0094] In S212, it is determined whether or not to calculate a velocity in which 0 is substituted at the start of tracking by the Kalman filter. In this embodiment, the conditions for calculating the initial velocity are that the Kalman filter has been updated three or more times and that the object has moved a certain amount or more from the tracking start position.

[0095] Here, the threshold value for a certain amount of movement is determined based on the result of the area determination of the observation value performed in S102. In this embodiment, the distance is set to 1.5 [m] when detected in area A2, and 1.0 [m] when detected in area A3. The formula for estimating the initial speed is shown below.

[0096] In the formula, n is the number of times of tracking, and x world ,yworld is the world coordinate value of the observation value associated during tracking, and framerate is the frame rate calculated from the shooting cycle. In this embodiment, it is 30 [fps].

number

number

[0097] The value calculated here is x + The initial velocity is determined by substituting vx and vy in S108. After the initial velocity is calculated, the Kalman filter in the tracking candidate state is changed to the tracking state. The series of processes in S108 is performed for all Kalman filters and newly detected moving objects.

[0098] In S109, the position of the moving object being tracked is output to the external system 40. In this embodiment, up to three moving objects being tracked are output. The moving objects to be output are the moving objects being tracked, the moving object candidate being tracked, and the moving object being interpolated, and are output in the order of the Euclidean distance between the vehicle and the moving object being closest as they have a high possibility of collision. Moving objects in the tracking start state are not output because there is a possibility of momentary erroneous detection.

[0099] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment. For example, the above-mentioned embodiment describes the present invention in detail, and it is not necessary to have all the configurations described. It is also possible to add the configuration of other embodiments to the configuration. In addition, it is possible to add, delete, or replace part of the configuration. [Explanation of symbols]

[0100] 10 External world recognition device 11 Object position detection unit 12 Area determination section 13. Observation error setting section 14 Mapping section 15 State Prediction Unit 16 Status Update Section 20 images 30 Travel Information

Claims

1. An external environment recognition device that detects and tracks moving objects around a moving object equipped with a camera, an object position detection unit that detects the position of the moving object as an observation value based on an image captured by the camera; an area determination unit that determines an area to which the moving object belongs in the image based on the observation value detected by the object position detection unit; an observation error setting unit that calculates an error for an observation value based on a result of the determination by the area determination unit; a state prediction unit that predicts a state of the moving object at the current time based on the observed values ​​up to a previous time that is a time prior to the current time and an error calculated by the observation error setting unit; a correspondence unit that associates the state of the moving object at the current time predicted by the state prediction unit with the observation value detected by the object position detection unit; a state update unit that updates a state of the moving object based on a result of the association by the association unit, The area determination unit uses the image divided into a plurality of areas according to a magnitude of an error corresponding to the position of the moving object detected in the image, based on physical parameters such as the mounting position, mounting attitude, and lens distortion of the camera relative to the moving object, to determine which of the plurality of areas the observation value in the image detected by the object position detection unit belongs to.

2. The plurality of regions include a first region above a vanishing point of the image, a second region set below and adjacent to the first region, a third region set below and adjacent to the second region, and a fourth region set below and adjacent to the third region, The observation error setting unit sets an invalid value as the error for the first domain, dynamically sets the error for the second domain, dynamically sets the error for the third domain, and when the set error is smaller than a preset minimum value, sets the error to the minimum value. The external environment recognition device according to claim 1, wherein an invalid value is set as the error for the observation value for the fourth domain.

3. 3. The external environment recognition device according to claim 1, wherein when the observation value in the image detected by the object position detection unit is associated with a region among the plurality of regions that is likely to not include the entire moving object, the moving object is detected by an alternative means.

4. The observation value in the image detected by the object position detection unit is associated with a region among the multiple regions that is likely to not include the entire moving object, and when detecting the moving object by the alternative means, detection is performed within a region of the image calculated by the predicted value of the state prediction unit. The external environment recognition device according to claim 3.

5. The external environment recognition device according to claim 1 , wherein, when the observation value in the image detected by the object position detection unit is associated with a region above a vanishing point of the image among the plurality of regions, the position of the moving object is deleted.

6. The measurement error setting unit is adding an error expected in the observation value in the image detected by the object position detection unit to the observation value in the image in each of an upward direction, a downward direction, a rightward direction, and a leftward direction of the image; projecting the observed value in the previous image after adding the error onto a moving body coordinate system using a pinhole camera model; The external environment recognition device according to claim 1 , further comprising: setting, as an observation error, an average of absolute values ​​of differences in each direction between the observed value in the image and a position projected onto a moving body coordinate system.

7. The association unit is a correspondence is established between the state of the moving object at the current time predicted by the state prediction unit and the observation value detected by the object position detection unit using a Mahalanobis distance; The external environment recognition device according to claim 1 , wherein a result of the association by the association unit is verified by an x-squared test.

8. the state prediction unit predicts the observed value not on the coordinate system of the image but on a coordinate system having an origin at a point at the time of system startup by using movement information of the moving object; The external environment recognition device according to claim 1 , wherein the state update unit updates the observation value on the coordinate system predicted by the state prediction unit to track a target moving object.

9. In a state estimator having the state prediction unit and the state update unit, when the target to be tracked has an internal state including a speed, the state update unit calculates the moving speed of the target to be tracked based on the observation value associated with each state estimator. The external environment recognition device according to any one of claims 1 to 8.

10. When the observation value cannot be obtained by a state estimator having the state prediction unit and the state update unit, the internal state and the error covariance matrix are updated by the state update unit using the prediction result of the state prediction unit. The external environment recognition device according to any one of claims 1 to 9.

Images