EXTERNAL RECOGNITION DEVICE

Abstract

EXTERNAL RECOGNITION DEVICE The external recognition device (10) includes an object position detection unit (11) that detects the position of a moving object as an observation value, based on an image; a region determination unit (12) that determines the region in the image to which the moving object belongs; an observation error establishment unit (13) that calculates an error on an observation value, based on a determination result; a state prediction unit (15) that predicts the state of the moving object at a current time, based on the observation value up to a previous time and the error; an association unit (14) that associates the state of the moving object at the current time with the observation value; and a state update unit (16) that updates the state of the moving object. Figure for abstract: 1

Description

Title of the invention: RECOGNITION DEVICE EXTERNAL CROSS REFERENCE TO AN ASSOCIATED REQUEST

[0001] The present application is based on and claims priority benefit from Japanese patent application No. 2021-018600, filed on February 8, 2021. Scope of invention

[0002] The present invention relates to an external recognition device. BACKGROUND

[0003] Currently, various driver assistance systems have been developed and marketed. One such driver assistance system is a collision damage mitigation braking system. This system uses a sensor installed in the vehicle to accurately detect an object (moving object) moving near the vehicle and track that object.

[0004] A technique using a Kalman filter is known as a technique for tracking a moving object. The technique using a Kalman filter can predict the position of a moving object at a time later than a predetermined time, based on the change in position of the moving object up to the predetermined time.

[0005] However, this technique reduces the accuracy of a prediction when an observation error relative to an observed position (the position of a detected moving object) cannot be precisely determined. When using information about a detected moving object based on an image, the magnitude of an observation error varies depending on the position where the moving object is detected in the image. Furthermore, a moving object intended to be a tracking target may move out of the image and therefore not be detectable, depending on the viewing angle and mounting angle of the camera capturing the image. In such cases, there is a concern that tracking a moving object may fail and that a clean vehicle may collide with the moving object.

[0006] The following technique has been suggested as a method for tracking a moving object, in particular a person, using a camera mounted on a moving body (for example, see JP 2009-089365 A).

[0007] More specifically, images of the periphery of a moving body are captured at constant time intervals by a camera device mounted on a moving body within the camera device's field of view. Then, an image processing system specifies the pixels corresponding to the moving object, which is a tracking target, based on a plurality of preceding and following images in a time series. which are obtained through capture.

[0008] For each moving object, a pixel corresponding to a spatial point where the moving object comes into contact with the surface of a road and representing a projection in image coordinates is specified. The motion of the spatial point that comes into contact with the road surface is tracked by a state estimator using at least one four-dimensional state vector whose components are the position of the spatial point coming into contact with the road surface and a relative velocity of the road surface.

[0009] Motion tracking by the state estimator is performed by a step of generating a prediction for each state vector, a step of converting it to image coordinates via a projection equation adapted to the prediction, a step of calculating an expected error in the prediction, in the image coordinates using a covariance matrix, and a step of updating by comparing the prediction with a pixel extracted from a more recent image. A moving object is tracked by the motion tracking. SUMMARY

[0010] A system, such as a collision damage mitigation braking system, which estimates the time until a collision by calculating a distance from the position of a moving object around a vehicle, and automatically applies the brakes based on the estimated time when there is a risk of collision, must accurately measure the distance to the moving object and track it. Therefore, in order to implement the collision damage mitigation braking system, it is important not only to perform continuous tracking but also to avoid relying on erroneous detection results.

[0011] When the prior art suggested by JP 2009-089365 A is applied to an automatic parking system, an expected error for an observation value becomes significant in the vicinity of a clean vehicle, and the clean vehicle's most recent position is associated with a deviated detection result. Consequently, the distance to a tracked moving object cannot be measured correctly, and there is therefore a risk that a vehicle may collide with the moving object.

[0012] The present disclosure was made in view of the above circumstances, and aims to provide an external recognition device that can perform very accurate tracking of a moving object, based on an image acquired by a camera.

[0013] The present invention relates to an external recognition device that detects and tracks a moving object around a moving body equipped with a camera. The external recognition device comprises: an object position detection unit that detects the position of the moving object as an observation value, based on an image captured by the camera; and a region determination unit that determines a a region to which the moving object belongs in the image, based on the observation value detected by the object position detection unit; an observation error determination unit that calculates an error relative to an observation value, based on a result of a determination by the region determination unit; a state prediction unit that predicts the state of the moving object at a current time, based on the observation value up to a previous time that is a time prior to the current time, and an error calculated by the observation error establishment unit; an association unit that associates the state of the moving object at the current time predicted by the state prediction unit with the observation value detected by the object position detection unit; and a state update unit that updates the state of the moving object, based on a result of the association by the association unit.

[0014] An external recognition device according to this disclosure can perform very precise tracking of a moving object, using an image acquired by a camera. Brief description of the drawings

[0015] [Fig-1] Fig. 1 is a functional diagram illustrating a recognition device external according to an embodiment of the present invention.

[0016] [Fig.2] The [Fig.2] is a schematic diagram in which a person and a vehicle are an example of the result of detecting a moving object.

[0017] [Fig.3] The [Fig.3] is a diagram illustrating the definition of the position of a detection result in an image.

[0018] [Fig.4] The [Fig.4] is a flowchart illustrating a processing flow according to the first embodiment.

[0019] [Fig.5A] The [Fig.5A] is a diagram illustrating the definition of a global coordinate system.

[0020] [Fig.5B] The [Fig.5B] is a diagram illustrating the definition of a coordinate system of the moving body.

[0021] [Fig.6] The [Fig.6] is a diagram illustrating an example of divided regions of an image.

[0022] [Fig.7A] The [Fig.7A] is a schematic diagram illustrating how an image is scanned vertically, as an example of a criterion for dividing a region.

[0023] [Fig.7B] The [Fig.7B] is a graph in which an association relationship of the magnitude of the change in distance to a person is obtained by calculation each time a y coordinate in an image changes by one [pixel] by scanning on the [Fig.7A], and estimated by a curve.

[0024] [Fig.8A] Fig.8A is an aerial image illustrating an example of a case where a detection target (e.g., a person) is relatively far from a clean vehicle.

[0025] [Fig.8B] The [Fig.8B] is an aerial image illustrating an example of a case where a detection target (e.g., a person) is relatively close to a clean vehicle.

[0026] [Fig.8C] The [Fig.8C] is a diagram of a coordinate system of a moving body on which the detection target illustrated in the [Fig.8A] and the detection target illustrated in the [Fig.8B] are each projected.

[0027] [Fig.9] The [Fig.9] is a diagram illustrating a method of calculating a distance to a clean vehicle from the position of a detection result in an image.

[0028] [Fig. 10] The [Fig. 10] is a diagram illustrating an example of association.

[0029] [Fig. 11] The [Fig. 11] is a diagram illustrating an example of a loss result of vision.

[0030] [Fig. 12] The [Fig. 12] is a processing flowchart for updating a Kalman filter.

[0031] [Fig. 13] The [Fig. 13] is a diagram illustrating an example of a state management table of a Kalman filter. DETAILED DESCRIPTION

[0032] A specific embodiment of an external recognition device according to this disclosure is described below with reference to the drawings.

[0033] First embodiment

[0034] Figure 1 is a functional diagram illustrating an external recognition system 10. The external recognition device illustrated 10 is an embodiment of an external recognition device according to the present invention. The external recognition device 10 is installed in a moving body such as an automobile. Hereinafter, a moving body in which the external recognition device 10 is installed is referred to as a clean vehicle and described accordingly.

[0035] The external recognition device 10 is used in combination with an external system 40. Furthermore, the external recognition device 10 operates at a predetermined cycle, for example, in 100 ms cycles. The external recognition device 10 also has a function for acquiring vehicle-related information from a clean vehicle, such as speed, steering angle, and the states of a gear change and braking, and a function for acquiring an image captured by a camera.

[0036] As shown in [Fig. 1], the external recognition device 10 includes an object position detection unit 11, a region determination unit 12, a The external recognition device 10 comprises, for example, a central processing unit (CPU), a graphics processing unit (GPU), random access memory (RAM), read-only memory (ROM), and other components, and performs the functions of the object position detection unit 11 and other functions described above by deploying a predetermined program stored in ROM into RAM and executing the program with the CPU. Furthermore, the external recognition device 10 can perform all or part of its functions using hardware such as an FPGA or an ASIC.

[0037] An image 20 captured by a camera mounted on a clean vehicle, and motion information 30 acquired from the clean vehicle via a CAN bus or similar device, are entered into the external recognition device 10. The image 20 is acquired sequentially at a predetermined frame rate by a camera installed in the clean vehicle. The motion information 30 consists of information relating to the movement of a clean vehicle obtained from the clean vehicle and information equivalent to the motion information, and includes, for example, vehicle speed, steering angle, turning radius, accelerator and brake states, and gearshift state. The motion information 30 is also acquired sequentially at a predetermined rate, in the same way as for the image 20.

[0038] The object position detection unit 11 detects a desired moving object (moving object), such as a person or a vehicle, using a well-known technique such as pattern comparison, based on the image 20. [Fig.2] is a schematic diagram in which a person and a vehicle are an example of the result of detecting a moving object.

[0039] As shown in [Fig. 2], a moving object detection result is information specifying the position of a moving object (a person or a vehicle) in an image. More precisely, a moving object detection result is information that defines a rectangular frame F surrounding a moving object and is prescribed by a height H, where H is a dimension of the rectangular frame F in a perpendicular (vertical) direction, a width W is a dimension of the rectangular frame F in a horizontal (lateral) direction, and the position of an origin C of the rectangular frame F is a specific corner portion (in the present embodiment, an upper left corner portion) of the rectangular frame F. The object position detection unit 11 provides a detection result to the region determination unit 12 and the association unit 14.

[0040] Fig. 3 is a schematic diagram illustrating a position in image 20 (a portion of the upper left corner is an image origin O(0, 0), a direction ho A perpendicular direction to the right is the positive direction of an x-axis, and a perpendicular direction downwards is the positive direction of a y-axis) of a detected moving object. The region determination unit 12 determines, by the detection result entered from the object position detection unit 11, to which region of the image 20, when the image 20 is divided into a plurality of regions, a detected moving object (detecting moving object) belongs.

[0041] Here, as shown in [Fig. 3], it is assumed that the representative point of the position of the moving object being detected is a horizontal central point (a position that is at the height H of the rectangular frame F from the origin of the rectangular frame F, and that is half the width W of the rectangular frame F) in the lower edge of the rectangular frame F specified by the object position detection unit 11 described above. Then, the region determination unit 12 sends a region in the image 20 to which this representative point belongs, as the position of the moving object.

[0042] The observation error setting unit 13 determines, based on a result in the region determination unit 12, an error assumed to be included in an observation value (the position of the moving object detected by the object position detection unit 11 based on the image 20). Furthermore, when it is determined, based on a result from the region determination unit 12, that a region is found to be one that cannot be detected by pattern matching recognition due to a loss of foot or a loss of view in which a part of a foot (a foot, a lower part of a tire, or similar) of the detecting moving object is not captured in an image, the observation error setting unit 13 sets a value representing that the detection is invalid.Next, the observation error establishment unit 13 sends the established value representing an error or invalidity to the association unit 14.

[0043] The association unit 14 associates the position (predicted position of the moving object) of the moving object with a current time, predicted by the state prediction unit 15 of a state estimator, based on the detection result entered from the object position detection unit 11 and an observation error defined by the observation error establishment unit 13. In this case, when no predicted position of the moving object can be associated, the moving object is designated as a newly detected moving object (newly detected moving object). The association unit 14 provides, as output to the state update unit 16, an association result and information about the newly detected moving object.

[0044] The state prediction unit 15 predicts, based on the state of the detection moving object obtained by updating in the state update unit 16, the state of the moving object at a current time, using a result detected by the detection unit of The position of object 11 is determined up to a time (previous time) prior to the current time. Here, it is assumed that a state has at least one position and one velocity. A prediction result from the state prediction unit 15 is used in the association unit 14.

[0045] The state update unit 16 begins the state update using a result from combining the results of the state prediction unit 15 and the object position detection unit 11 in the association unit 14, and tracks the designated moving object as a new detectable moving object. An updated state is sent to the external system 40, and the external system 40 uses this result to measure the time required until a collision, based, for example, on the distance between a vehicle and a person (an example of a moving object), and uses this time in a system that automatically applies the brakes so that the vehicle can be stopped within the time required for a collision.

[0046] Next, the operation of the external recognition device 10 is described with reference to a flowchart. Figure 4 is a flowchart illustrating the sequence of all processing by the external recognition device 10. Furthermore, the present embodiment is described assuming that a camera is installed at the rear of a vehicle.

[0047] Before the description of the flowchart, a coordinate system (a global coordinate system, a moving body coordinate system) used in the processing of the external recognition device 10 is described.

[0048] Figure 5A is a diagram illustrating a global coordinate system in which the position of a vehicle at the start of the external system 40 is an origin, and in which the rear side of the vehicle at the start of the external system 40 is +xw, the side towards +xw is +yw, and the rotation from +xw to +yw is a value on a two-dimensional xy plane being +0W. Figure 5B is a diagram illustrating a moving-body coordinate system that the external recognition device 10 uses to represent the position and orientation of a vehicle, in which the rear side of the vehicle is +xv, the side towards +xv is +yv, and the rotation from +xv to +yv is a value on a two-dimensional xy plane being +0V. In the moving body coordinate system, the + direction of an x-axis, the + direction of a y-axis, and the + direction of rotation from 0 are the same as those of the global coordinates, but the origin moves according to the movement of a vehicle.

[0049] Next, the flowchart of [Fig. 4] is described. The entire flowchart processing begins with the startup of the external system 40.

[0050] An example illustrated in the flowchart of [Fig. 4] is described assuming the use of a monocular camera (hereinafter simply referred to as the "camera") mounted in the vehicle and directed towards the rear of the vehicle. Furthermore, it is assumed The data acquisition cycle for the camera and the vehicle is 33 ms. Furthermore, the image size of the camera is assumed to be 640 pixels (width) x 400 pixels (height). The vehicle information is necessary to calculate its displacement. Additionally, a model assuming uniform motion is used to predict the position of the target moving object.

[0051] At step 101 (S101), the object position detection unit 11 detects a moving object which can be a tracking target, based on the image 20 read in S100. It is assumed here that the target moving objects to be tracked are, for example, a person and a vehicle.

[0052] In the present embodiment, the detection of a moving object that can be a tracking target uses oriented gradient histograms (HoG) and a support vector machine (SVM), both of which are well-known techniques.

[0053] Here, HoG is a gradient strength that is calculated for each gradient direction of a pixel value in a local region and converted into a histogram. The conversion of this histogram into a feature quantity is called the HoG feature quantity. The HoG feature quantity uses the gradient as its feature quantity and has the advantage of being robust against image scaling. On the other hand, the HoG feature quantity has the disadvantage of being weak against rotation.

[0054] A SVM is a classifier that classifies an input as either a correct or an incorrect answer. The SVM learns a boundary for classification into a correct or incorrect answer by learning the relationship between the data and a label, based on a large set of data and a label, and when unknown data are entered, the SVM determines whether the entered data represent a correct or an incorrect answer using the learned boundary.

[0055] In the present embodiment, a classifier is generated by learning a quantity of HoG features and each of the person and vehicle labels. Furthermore, as illustrated in [Fig. 2], a detection result is assumed to have the width W and the height H to draw the rectangular frame F surrounding an origin of the detection result and the detection result, and a label such as a person and a vehicle.

[0056] Each technique is well known, and a detailed description is therefore omitted. Furthermore, any other well-known technique, such as pattern recognition, can be used as a technique for detecting a person or a vehicle, and the detection technique is not limited to the technique described above. Moreover, as a moving object, objects other than a person or a vehicle can be a detection target or a tracking target.

[0057] In S101 of the flowchart, the position of a moving object in the image 20 is detected by a moving object detection result as shown in [Fig. 3]. As illustrated in [Fig. 3], in the coordinates of the image 20, the downward direction of the image 20 is +y, and the rightward direction of the image 20 is +x. A position (x, y) in the image constitutes the sum of the width W / 2 (direction of the x-axis) and the height H (direction of the y-axis) at the origin of the image O (0, 0).

[0058] Figure 6 is a diagram illustrating an example of divided regions A1, A2, A3, and A4 of image 20. Figure 7A is a diagram illustrating how image 20 is scanned vertically, as an example of a criterion for dividing a region. Figure 7B is a graph in which an association relationship of the magnitude of the change in distance to a person is obtained by calculation each time a y-coordinate in image 20 changes by one pixel per scan on Figure 7A, and estimated by a curve.

[0059] Determining a region in S102 constitutes a process for determining which region the position of a detection result (moving object) belongs to, when the image 20 is separated, for example, into four vertical regions A1, A2, A3, and A4, as illustrated in [Fig. 6]. The division into regions A1 to A4 determines a distance error corresponding to a pixel deviation for each position in the y-axis direction in the image 20, based, for example, on the distortion of a camera lens and the camera mounting orientation on a vehicle, and based on the curve in [Fig. 7B].

[0060] The example in [Fig. 6] is divided into four regions: A1, A2, A3, and A4. Region A1 is a region in which an error of several meters occurs due to the deviation of the position of a moving detection object by one pixel, or in which the sky is included. Region A2 is a region in which an error smaller than that of region A1, but still relatively large, occurs due to the deviation of the position of a moving detection object by one pixel. Region A3 is a region in which an error smaller than that of region A2 occurs due to the deviation of the position of a moving detection object by one pixel. Region A4 is a region in which the distances between a vehicle and a detection target are physically close, and in which the detection target is considered not to be able to be correctly detected.

[0061] Furthermore, a separation criterion (limit) for dividing the separation into four regions A1, A2, A3, and A4 is fixed such that a portion above a vanishing point (see [Fig. 6]) in image 20 is region A1, a portion up to region A1 from a point where a detection position of a distance of approximately 0.8 m on the coordinate system of the moving body changes due to an error of one pixel is region A2, a portion up to region A2 from a point where a detection position of a distance of about 0.2 m on the coordinate system of the moving body changes from region A2 due to an error of one pixel is region A3, and a part lower than region A3 is region A4.

[0062] The region determination unit 12 determines in which of the regions Al to A4 the position of the detection result is located in the image 20 (S102), and sends the determination to the observation error establishment unit 13. In addition, a moving detection object located in a position such as the region Al where the detection error is extremely large or a moving detection object whose position is considered not to be on the ground is not targeted for tracking and is deleted.

[0063] Here, the method of dividing into regions A1 to A4 is not limited to using a physical parameter such as the distortion of a camera lens and the camera mounting position. Furthermore, the regions A1 to A4 to be divided can be a mixture of not only vertical but also lateral divisions, and the number of divisions is not limited to four.

[0064] In S103, the position of a moving object tracked by the state estimator at a current time on a global coordinate system is predicted. Although a Kalman filter, an extended Kalman filter, an unscented Kalman filter, and others are used as state estimators, a Kalman filter is used in the example described in this embodiment.

[0065] First, the Kalman filter in the present embodiment uses a uniform motion model, and a state to be estimated is therefore a four-dimensional vector x = (x, y, vx, vy). Furthermore, x, y denote the position on the global coordinate system, and vx, vy denotes a velocity on the global coordinate system. The position on the global coordinate system at a given time and a value of an error covariance matrix for this position are predicted by equations (1) and (2) below.

[0066] Equation (1)

[0067] %t = F * (1)

[0068] Equation (2)

[0069] Pt = F * P^ * Ft + Q (2)

[0070] Here, the index / superscript t represents a time, - represents a predicted value, and + represents a value updated based on an observation. Furthermore, in Equations (1) and (2), F and Q represent respectively a state transition matrix and system noise, and are represented by Equations (3) and (4) in the present

[0071]

[0072]

[0073]

[0074]

[0075] method of implementation. Equation (3) / 1 0 At 0 \ { 0 1 0 At | f QA F “ I 0 0 1 0 I Vo o oi / Equation (4) f^systemnoise ® 0 0 \ $ ^systemnoise 0 0 , Q - 2 W 0 0 ^systemnoise ® \ 0 0 0 ^systemnoised / At in Equation (3) is a processing cycle, and o2sytemnoiSe in Equation (4) is an assumed error when the model according to the present embodiment is applied. The processing in S103 is performed for all Kalman filters that are not

[0076]

[0077]

[0078]

[0079] not in an invalid state. In S104, the 30 motion data points are acquired from the vehicle by any method, such as a Control Area Network (CAN). Here, the 30 motion data points are necessary to calculate a movement distance in S105. For example, the 30 motion data points include information such as the vehicle's speed, the number of tire rotations, and the steering wheel angle. In S105, the vehicle position is estimated using the motion information acquired in S104. In the present embodiment, a vehicle position is estimated by Equations (5) to (8), using a movement distance calculated from the number of rotations of a tire and a tire diameter, and a turning radius estimated from the turning radius of a steering wheel. Here, the vehicle assumes motion on a two-dimensional plane and has three vehicle posture states: (xcarpose, ycarpose, 0yaw)- Equation (5)

[0080]

[0081] 1 Pt ~ Equation (6) (5) (6)

[0082] Equation (7)

[0083] (^Carposef^carposet_^\ l | VIIV ) ~^ l / \ I v) ycarposej ^carpose^JV'n(Wl) /

[0084] Equation (8)

[0085] ^yawt “ ^yawt„1 + ^yawc ($)

[0086] Here, t represents a time, R represents a turning radius, p represents a curvature, and AS represents a travel distance. As a method for estimating the position of a vehicle, one can use a method that employs other models, such as visual odometry, which estimates the position and orientation of a vehicle using image 20, or other methods.

[0087] In S106, an error of an observation result is defined based on the result of a region determination. In the present embodiment, with respect to each of the regions A1 and A4, an invalid value as an error of an observation result is defined as an invalid detection result. With respect to region A2, an error is dynamically set as in [Fig. 8A], 8B, and 8C. For the error establishment procedure, first, an x, y error of an assumed detection position on an image is added to the coordinates of the detection target in image 20. Then, the coordinates of image 20 to which the error is added are projected onto the coordinate system of the moving body using the orientation of a camera and the height of the camera relative to the ground, according to a pinhole model.

[0088] Figure 8A is an aerial image illustrating an example of a case where a detection target (e.g., a person) is relatively far from a clean vehicle. Figure 8B is an aerial image illustrating an example of a case where a detection target (e.g., a person) is relatively close to a clean vehicle. Figure 8C is a diagram of a moving body coordinate system onto which the detection target illustrated in Figure 8A and the detection target illustrated in Figure 8B are each projected.

[0089] The coordinates in image 20, to which the error is added, are projected onto the coordinate system of the moving body. Then, when the detection target (e.g., a person) is relatively far from the vehicle in the aerial image of [Fig. 8A], the detection target is projected to a position far from the vehicle, as illustrated in [Fig. 8C]. On the other hand, when the detection target is relatively close to the vehicle in the aerial image of [Fig. 8B], the detection target is projected towards a position close to the vehicle as illustrated in [Fig.8C].

[0090] An average of the differences in an x ​​direction in which a position of the center of the image projected onto the coordinate system of the moving body ([Fig.8C]) with the addition of an assumed error is projected in a front-back direction relative to the vehicle as a value of an observation error is a value of an observation error in the x direction, and an average of the differences in a y direction in which the position of the center of the image is projected in a left-right direction is a value of an observation error in the y direction.

[0091] An observation error is expressed as follows to convert the motion of a detection target into a uniform motion model. In this embodiment, ox is an observation error in the front-to-back direction when viewed from a vehicle according to the moving body's coordinate system, and oy is an observation error in the left-to-right direction when viewed from a vehicle.

[0092] Equation (9)

[0093] R =

[0094] Moreover, when a position y on the coordinate system of the moving body changes due to the change of x coordinates in an image, and conversely, when a position x on the coordinate system of the moving body changes due to the change of y coordinates, a covariance value can be defined as follows.

[0095] Equation (10)

[0097] In region A3, when an observation error is established using the same method as in region A2, it is conceivable that the value may become too small. Thus, it is assumed that after calculating an observation error using the same method as in region A2, the observation error is compared to a predefined minimum value, and the minimum value is used when the observation error is less than the minimum value.

[0098] The dynamic estimation and adjustment of an observation error as described above has the effect of improving the tracking capability of a distant tracking target and reducing the tracking capability of a detection target that has been mistakenly detected in the vicinity of a vehicle.

[0099] The adjustment of an observation error by S106 is performed for all detection results in S101 that have become treatment targets in S102.

[0100] In S107, a position at a current time, predicted by the state estimator, of the moving object being tracked is associated with a position, in image 20, of the detection target detected from image 20, based on a projection result on the global coordinate system.

[0101] Figure 9 is a diagram illustrating a method for calculating a distance to a vehicle from the position of a detection result in an image. As illustrated in Figure 9, the calculation of the global coordinate system is performed using the height between the ground and an origin of a camera installed in a vehicle, a camera mounting position on the vehicle, the camera orientation, and an internal camera parameter, such that the intersection between the ground and a straight line connecting the camera origin and a position in the image is a foot position (xcar, ycar) on the moving body's coordinate system, based on the pinhole camera model, and the assumption that a detected moving object is present on the same plane as the vehicle.

[0102] Next, the foot position on the moving body coordinate system obtained by calculation is converted to the global coordinate system. Since the global coordinate system is a coordinate system in which a starting position of the external system 40 is an origin, it suffices to perform the conversion based on the position and orientation of the vehicle at a given time. Thus, the conversion from the moving body coordinate system to the global coordinate system is carried out by adding (xcarpose, ycarpose) obtained by Equation (7) to the foot position on the moving body coordinate system obtained by the method illustrated in [Fig. 9].

[0103] Furthermore, the value of an observation error is rotated according to the direction of the vehicle. Here, when the present invention is implemented using a fixed camera, Equations (5) to (8) and Equations (11) and (12) do not need to be implemented because the global coordinate system defined in Figures 5A and 5B and the coordinate system of the moving body always correspond to each other.

[0104] Equation (11)

[0105] McarposeA (11) ^yworldS xycarJ \.ycarpose t /

[0106] Equation (12)

[0107]

[0108]

[0109]

[0110] [YES]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] i COS ( Gy aw ) S in ( 0y £IW r = æ * ; v ù ysin \$yaWçJ £®S\^yawtJ (12) Figure 10 is a diagram illustrating an example of association. After being subjected to a coordinate conversion, all results detected from the image are associated on the basis of a position at a given time, predicted by the state estimator, of the tracked moving object, and a position, detected from image 20, of the detection target on the global coordinate system, as illustrated in Figure 10. In this embodiment, a Mahalanobis distance, a type of distance used in statistics, is used as an index of association distance. The Mahalanobis distance is an index that helps clarify the similarity of new data based on the variance-covariance value of a specific sample of data. Calculations of the Mahalanobis distance are shown below. Equation (13) (13) Equation (14) dtst Ç^pred pred Here, dist is a Mahalanobis distance, zpred is a position predicted (xpred, ypred) by a Kalman filter, and zdetect is a position (xworid, yworid) of a moving object detected from image 20. This processing is repeated by the number of products of (the number of predictions of the Kalman filter) and (the number of moving objects detected from the image), and a predictive value of the Kalman filter that minimizes the Mahalanobis distance is associated with the moving object detected from image 20. In this case, a plurality of Kalman filters can be associated with a moving object detected from image 20. Unnecessary deletion of a Kalman filter is avoided by associating a plurality of Kalman filters with a moving object detected from image 20. Furthermore, as with other association methods, well-known techniques such as the Hungarian algorithm can be used. Additionally, a Euclidean distance or similar can be used as an association index. Figure 11 is a diagram illustrating an example of a vision loss outcome. In S107, a Kalman filter is associated with a region (region A4 in the embodiment) in which the entire target is considered not to be contained. due to the appearance of vision loss as in [Fig. 11] following determination of the region in S102, uses a predictive value of the Kalman filter to attempt detection using an alternative such as optical flow or median flow.

[0118] Here, when detection has been successfully carried out by the alternative, a position on the global coordinate system and a distance from Mahalanobis are calculated, based on the result detected by the alternative, by processing in S107 and used as observation values.

[0119] In optical flow detection, x, y are projected onto the image 20 from the internal state x- = (x, y, vx, vy) of a predictive value of the Kalman filter to draw the rectangular frame F, and a velocity vector in the rectangular frame F is calculated. For the width W and height H required to draw the rectangular frame F, it suffices to use the height H and width W included in a last observation value associated with the Kalman filter.

[0120] It is assumed here that there is no observation value when, in the alternative method, detection has failed. Techniques such as optical flow are well-known methods, and their description is therefore omitted.

[0121] In S108, the state of the tracked moving object is updated using a detection result associated with the predicted value of the Kalman filter. In this embodiment, the state of the Kalman filter is managed by five states: start tracking, candidate for tracking, tracking, interpolating, and invalid.

[0122] For the start of tracking, the state of the Kalman filter is initialized to begin tracking a moving object detected in a state where the moving object is newly detected. The candidate for tracking indicates a state where no time has elapsed since the start of detection, and there is a high probability that the internal state of the Kalman filter is not stable.

[0123] The tracking state indicates a state where time has elapsed since the start of detection and the internal state of the Kalman filter is stable. The interpolating state indicates that a tracking target is in an undetected state for some reason after the start of tracking a moving detection object. The invalid state indicates a state where a target Kalman filter is not tracking anything. Furthermore, it is assumed that a Kalman filter has, as internal states, an internal state x, an error covariance matrix P, and an interpolation count Intp.

[0124] Figure 12 is a flowchart for updating a Kalman filter. As illustrated in Figure 12, first, S201 is checked to see if there is a detection result (observation value) associated with a Kalman filter. The flow moves to S202 when there is an observation value, while the flow moves to S207 when there is no observation value.

[0125] In S202, a rejection test is performed using a Mahalanobis distance calculated by Equation (14), based on a predictive value and an observation value of a Kalman filter. A chi-square test was used for the rejection test. The chi-square test is a statistical test performed under the assumption that a test statistic follows a χ² distribution under the null hypothesis.

[0126] The present embodiment exhibits a chi-square distribution with two degrees of freedom because a Mahalanobis distance is derived from (x, y). Since the significance level is 5 [%] in the present embodiment, the chi-square value is 5.99146. When a Mahalanobis distance has a value of 5.99146 or more (or greater than 5.99146), the flow shifts to S203, determining that the association is difficult to conceive from a predictive value. When a Mahalanobis distance has a value less than 5.99146 (or 5.99146 or less), the flow shifts to S210.

[0127] Furthermore, assuming that an association in which a chi-square value is equal to or greater than 5.99146 is an invalid association, an associated observation value is designated as a new moving detection object.

[0128] Figure 13 is a diagram illustrating an example of a state management table for a Kalman filter. S203 is the processing in the case of a new moving object being detected, and an object in an invalid state is searched for from an array (t-1) stored within a program, as illustrated in Figure 13.

[0129] Here, the flow switches to S205 when there is an invalid value (= space in the array) (YES in S204), while the flow switches to S206 when there is no invalid value (NO in S204). In S205, a newly registered Kalman filter is used to track a new detection moving object from the array to a location with an invalid value, the internal state x and the error covariance matrix P are initialized, and the array state is then set to the start of tracking.

[0130] For the initialization of the internal state, the global coordinate system of an observation value is substituted as follows. A velocity component is equal to 0. A predefined value, for example 0,1, is substituted for the error covariance matrix P. For example, 0 is substituted for the interpolation count Intp.

[0131]

[0132] Equation (15) ^world' yworld r 0,0) (15)

[0133] S206 is the processing when there is no space in the array. When there is no In place in the table, a tracked moving object that is considered the most useless is removed according to a predetermined criterion, and a newly detected moving object is recorded (recording a new detecting moving object) in its place.

[0134] In the present embodiment, firstly, a Euclidean distance between the

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145] A newly detected moving object and a vehicle are compared to the Euclidean distance between the tracked moving object recorded in the table and the vehicle. If the moving object with the greatest Euclidean distance is the tracked moving object recorded in the table, the tracked moving object furthest from the vehicle is determined to have the lowest collision probability and is removed as the tracked moving object deemed the least useful. The newly detected moving object is then recorded using the same processing as S205. When the furthest object is a newly detected moving object, the stream terminates without recording that moving object. Here, the table deletion condition is not limited to the magnitude of a Euclidean distance, and a condition other than a Euclidean distance, for example the state (such as being interpolated) of a tracked moving object, can be applied. In S207, it is determined whether to perform interpolation processing on a Kalman filter that is not associated with an observation value. This determination is made by an interpolation count. In this embodiment, the flow moves to S208 when an interpolation count is, for example, equal to or greater than three, while the flow moves to S209 when an interpolation count is less than three. In S208, because an interpolation count has exceeded a threshold, the tracking target is determined to have been lost sight of for a reason such as the tracking target moving outside of a capture range, and the state of a relevant Kalman filter in the table is changed from the currently interpolating state to the invalid state, so that the tracking target is not targeted for tracking. In S209, the process of adjusting the predicted value of the tracked object predicted in S103 to an updated value is performed. In this case, the interpolation count is increased by one. Equation (16) x + = x~ (16) Equation (17) P+ = P- (17) Equation (18) Intp ~ Intp +1 (18) In S210, the processing is determined by the state of a Kalman filter. When the state is being tracked, the flow moves to S211-a, while the flow moves to S211-b in the case of the start of tracking, candidate for tracking and interpolation states.

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157] In S211-a and S211-b, the internal state is updated based on an observation value associated with a Kalman filter. First, a Kalman gain K is calculated using an observation error R', global coordinates z of a lead error covariance matrix P-, an observation matrix H, and an observation value. Here, I represents a unitary matrix. Equation (19) H — (1 0 ° (19) VO 1 0 07 1 J Equation (20) K = P" * H! * (H* Pi *H( + RT1 (20) Equation (21) x^ = Xf + K(z — H * xf) (21) Equation (22) Pt ~ (J — K * H) * P^ (22) In S211-a, the processing ends. In S211-b, the flow moves to S212. A start-of-travel state is updated to a candidate-travel state when the processing so far has been implemented, the value in the array is updated to a current-travel state when a state is an interpolation state and the initial velocity has already been calculated, and the value in the array is updated to a candidate-travel state when the initial velocity has not been calculated. In S212, it is determined whether to calculate a velocity where 0 is substituted at the beginning of the tracking of a Kalman filter. In the present embodiment, the condition for calculating an initial velocity is that the Kalman filter is updated three or more times, and that a certain amount of movement or more has been made from a starting tracking position. Here, a threshold for a certain movement or more is determined based on a region determination result of an observation value implemented in S102. In this embodiment, the distance is assumed to be 1.5 m for detection in region A2, and 1.0 m for detection in region A3. An equation for estimating an initial velocity is presented below. In the equation, n is a tracking count, xworid and yworid are global coordinate values ​​of an associated observation value during tracking, and framerate is an image frequency calculated from a capture cycle. In the present

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168] In this implementation, the frame rate is 30 fps. Equation (23) “%worldi- j ) VX ~ ~ n * fr amenity Equation (24) Xi=i(ytwn'd> “ yworldi-'i ) py - —--—---—— n * f ramer ate (23) (24) The value calculated here is substituted for vx, vy of x+ to produce an initial velocity. After calculating the initial velocity, a Kalman filter in a candidate tracking state is transformed into a tracking state. A series of processing steps in S108 are performed for all Kalman filters and new sensing moving objects. In S109, the position of the tracked moving object is sent to the external system 40. In this embodiment, a maximum of three tracked moving objects are emitted. Among the moving objects being tracked, track candidates, and interpolated, those with a high collision probability are presented in ascending order of the Euclidean distances between a vehicle and a tracking target. An object in a tracking start state may be subject to instantaneous misdetection and is therefore not emitted. According to one embodiment of the invention, when the observation value in the image 20 detected by the object position detection unit 11 is associated with a region with a high possibility of not including the entire moving object among the plurality of regions, the moving object is detected by an alternative. The observation value in image 20 detected by the object position detection unit 11 is associated with a region with a high possibility of not including the entire moving object among the plurality of regions, and at some point during the detection of the moving object by the alternative, the detection is carried out within a region of image 20 calculated according to a predictive value of the state prediction unit 15. According to one embodiment of the invention, when the observation value in the image 20 detected by the object position detection unit 11 is associated with a region greater than a vanishing point of the image 20 among the plurality of regions, the position of the moving object is suppressed. According to one embodiment of the invention, the state prediction unit 15 is configured to predict the observation value using motion information of the moving body, not on image coordinates 20 but on a coordinate system in which a point at the start of the system is an origin, and State update unit 16 is configured to update the value observation on the coordinate system predicted by the state prediction unit 15, and tracks a target moving object.

[0169] According to one embodiment of the invention, in a state estimator comprising the state prediction unit 15 and the state update unit 16, when a tracking target has an internal state including a velocity, the state update unit 16 is configured to calculate a velocity of movement of the tracking target, based on an observation value associated with each state estimator.

[0170] According to one embodiment of the invention, when the observation value is not obtained by a state estimator comprising the state prediction unit 15 and the state update unit 16, an internal state and an error covariance matrix are updated by the state update unit 16, using a prediction result from the state prediction unit 15.

[0171] Although the embodiment according to this disclosure has been described above, this disclosure is not limited to the embodiment described above. For example, the embodiment described above is a detailed description of the present invention, and it is not necessary for all the components described to be included. Furthermore, it is also possible to add components to the components in other embodiments. In addition, it is possible to add, delete, or replace some of the components. LIST OF REFERENCES

[0172] 10: External recognition device

[0173] 11: Object position detection unit

[0174] 12: Region determination unit

[0175] 13: Unit for establishing observation errors

[0176] 14: Association Unit

[0177] 15: State prediction unit

[0178] 16: State Update Unit

[0179] 20: Image

[0180] 30: Movement Information

Claims

Demands

1. External recognition device (10) configured to detect and track a moving object around a moving body equipped with a camera, the external recognition device comprising: - an object position detection unit (11) configured to detect a position of the moving object as an observation value, based on an image (20) captured by the camera; - a region determination unit (12) configured to determine a region to which the moving object belongs in the image (20), based on the observation value detected by the object position detection unit (11); - an observation error definition unit (13) configured to calculate an error relative to an observation value, based on a result of a determination by the region determination unit (12);- a state prediction unit (15) configured to predict a state of the moving object at a current time, based on the observation value up to a previous time which is a time prior to the current time and an error calculated by the observation error establishment unit (13); - an association unit (14) configured to associate the state of the moving object at the current time predicted by the state prediction unit (15) with the observation value detected by the object position detection unit (11); and - a state update unit (16) configured to update the state of the moving object, based on a result of the association by the association unit (14).

2. External recognition device according to claim 1, wherein the region determination unit (12) is configured to determine, using the image (20) divided into a plurality of regions, to which of the plurality of regions belongs the observation value in the image (20) detected by the object position detection unit (11), on the basis of physical parameters which are a mounting position and a mounting orientation of the camera relative to the moving body, and a lens distortion.

3. External recognition device according to claim 1 or claim 2, wherein when the observation value in the image (20) detected by the object position detection unit (11) is associated with a region with a high possibility of not including the entire moving object among the plurality of regions, the moving object is detected by an alternative.

4. External recognition device according to claim 3, wherein the observation value in the image (20) detected by the object position detection unit (11) is associated with a region having a high possibility of not including the entire moving object among the plurality of regions, and at some point during the detection of the moving object by the alternative, the detection is carried out within a region of the image (20) calculated according to a predictive value of the state prediction unit (15).

5. External recognition device according to any one of claims 1 to 4, wherein when the observation value in the image (20) detected by the object position detection unit (11) is associated with a region above a vanishing point of the image (20) among the plurality of regions, the position of the moving object is suppressed.

6. External recognition device according to any one of claims 1 to 5, wherein the observation error setting unit (13) is configured to: - add an assumed error in the observation value in the image (20) detected by the object position detection unit (11) to the observation value in the image (20) in each of the directions up, down, right and left of the image (20), - project the observation value in the image (20) to which the error is added onto a moving body coordinate system using a pinhole model, and - then, define, as observation error, an average of the absolute values ​​of the differences in each direction with respect to a position where the observation value in the image (20) is projected onto a moving body coordinate system.

7. External recognition device according to any one of claims 1 to 6, wherein the association unit (14) is configured to - associate, using a Mahalanobis distance, the state of the moving object at the current time predicted by the state prediction unit (15) with the observation value detected by the object position detection unit (11), and - verify, by a chi-square test, an association result by the association unit (14).

8. External recognition device according to any one of claims 1 to 7, wherein the state prediction unit (15) is configured to predict the observation value using motion information of the moving body, not on image coordinates (20) but on a coordinate system in which a point at system start-up is an origin, and the state update unit (16) is configured to update the observation value on the coordinate system predicted by the state prediction unit (15), and tracks a target moving object.

9. External recognition device according to any one of claims 1 to 8, wherein in a state estimator comprising the state prediction unit (15) and the state update unit (16), when a tracking target has an internal state comprising a velocity, the state update unit (16) is configured to calculate a velocity of movement of the tracking target, based on an observation value associated with each state estimator.

10. External recognition device according to any one of claims 1 to 9, wherein when the observation value is not obtained by a state estimator comprising the state prediction unit (15) and the state update unit (16), an internal state and an error covariance matrix are updated by the state update unit (16), using a prediction result from the state prediction unit (15).

Images