Object recognition device and object recognition method

Abstract

The purpose of the present invention is to accurately estimate the shape and position of a vehicle when a vehicle corner cannot be observed. This object recognition device comprises: a detection unit that detects an object using external information acquired by an external sensor; a shape information management unit that manages shape information indicating the shape of the object; a shape estimation unit that estimates the shape of the object on the basis of the shape information and partial shape information indicating the shape of at least a part of the object derived from the external information; and a position estimation unit that estimates the position of the object using the estimated shape of the object.

Description

Object recognition device and object recognition method

[0001] The present invention relates to an object recognition device and an object recognition method for recognizing the position of another vehicle in the vicinity of a vehicle.

[0002] In order to realize preventive safety functions and autonomous driving, it is effective to detect surrounding objects using on-board sensors and perform recognition processing such as image processing and point cloud processing. When the relative positions of the vehicle and other vehicles change, the observation point changes, for example, from the front of the vehicle to the side of the vehicle, making the position estimation unstable. Therefore, a position estimation method that takes the observation point into account is required.

[0003] In Patent Document 1, the solution section of the abstract explains that "an object recognition device 3 for recognizing other vehicles present around a vehicle determines whether or not there is a partially recognized vehicle among a plurality of recognized other vehicles that is partially hidden by the other vehicle present in front, and if a partially recognized vehicle is present and a corner of the partially recognized vehicle can be recognized, the longitudinal length and lateral length of the partially recognized vehicle are set to predetermined longitudinal and lateral lengths, and the center position of the partially recognized vehicle is calculated based on the corner of the partially recognized vehicle." As such, a method is disclosed in which observation points are identified by observing the corners of a vehicle, and the vehicle position is estimated based on the corners.

[0004] Japanese Patent Application Laid-Open No. 2022-011745

[0005] By using the method described in Patent Document 1, it is possible to easily calculate the center position of a partially recognized vehicle that is partially hidden by another vehicle in front of it, based on the corners of the partially recognized vehicle. However, there is a problem in that accurate estimation is impossible if the corners cannot be observed, such as when vehicles are approaching each other, such as when passing each other.

[0006] In view of the above, an object of the present invention is to provide an object recognition device and an object recognition method that can accurately estimate the shape and position of a vehicle even when the vehicle angle cannot be observed.

[0007] In order to solve the above problem, the object recognition device of the present invention is an object recognition device that includes a detection unit that detects an object using external world information acquired by an external world sensor, a shape information management unit that manages shape information that represents the shape of the object, a shape estimation unit that estimates the shape of the object based on the shape information and partial shape information that represents the shape of at least a part of the object obtained from the external world information, and a position estimation unit that estimates the position of the object using the estimated shape of the object.

[0008] According to the present invention, when the vehicle angle cannot be observed, the shape and position of the vehicle can be accurately estimated.

[0009] FIG. 1 is a functional block diagram showing the configuration of an object recognition device according to an embodiment; FIG. 2 is a diagram for explaining a database held by a shape information management unit; FIG. 3 is a processing flow diagram by an object recognition device according to an embodiment; FIG. 4 is a diagram explaining a passing scene; FIG. 5 is a diagram explaining partial shape extraction by a partial shape extraction unit; FIG. 6 is a diagram explaining a method for determining corners using principal component analysis; FIG. 7 is a diagram explaining a method for determining corners using principal component analysis; FIG. 8 is a diagram explaining shape matching by a shape matching unit; FIG. 9 is a diagram explaining a scene in which shape information is updated; FIG. 10 is a diagram explaining updating of matching shape information; FIG. 11 is a diagram illustrating characteristic shapes of vehicles; FIG. 12 is a diagram showing an example of matching based on feature amounts.

[0010] Hereinafter, embodiments of the present invention will be described in detail.

[0011] FIG. 1 is a functional block diagram showing the configuration of an object recognition device 1 according to one embodiment of the present invention. The object recognition device 1 shown here is an in-vehicle device connected to an external sensor 2 such as a LiDAR, and is a device that recognizes the type and position of objects near the host vehicle V0, particularly the type and position of other vehicles, based on the output of the external sensor 2. Specifically, the object recognition device 1 is a computer equipped with an arithmetic unit, a storage device, etc., and operates as various functional units described below by the arithmetic unit executing a predetermined control program. The external sensor 2 is, for example, a LiDAR disposed at the four corners of the host vehicle V0: the front left, front right, rear left, and rear right.

[0012] Here, the shape of a vehicle viewed from above is generally rectangular regardless of the type of vehicle (passenger car, truck, motorcycle, etc.). Therefore, by recognizing or estimating the corners of the vehicle, the position of the vehicle can be easily calculated using the corners as a reference. In general, data-driven methods such as deep learning require a large amount of calculation, which slows down the processing speed of the object recognition device 1. In contrast, a position estimation method using corners as a reference requires a small amount of calculation, which can prevent the recognition process from slowing down. Therefore, in the object recognition device 1 of this embodiment, the shape of a vehicle viewed from above is approximated as a rectangle, and vehicle type determination and position determination are easily performed under this assumption.

[0013] 1, the object recognition device 1 includes an object detection unit 10, a shape information management unit 11, a shape estimation unit 12, and a position estimation unit 13. The shape estimation unit 12 also includes a partial shape extraction unit 12a, a corner portion determination unit 12b, and a shape matching unit 12c. Each unit will be described in detail below.

[0014] <<Object Detector 10>> The object detector 10 detects the type of object using the measurement information of the external sensor 2. The object to be detected is the type of vehicle, such as a passenger car, truck, or motorcycle.

[0015] <<Shape Information Management Unit 11>> The shape information management unit 11 manages object shape information as a database. This database may be stored in a storage device inside the vehicle, or may be provided outside the vehicle and the database information may be acquired via communication. In Fig. 2, the shape information management unit 11 stores information for each vehicle type to be detected in databases such as a passenger car database 11a, a truck database 11b, and a motorcycle database 11c, and shape information for each vehicle type is registered in each database.

[0016] The shape information includes information about the vehicle's longitudinal length (hereinafter referred to as vehicle length), its transverse length (hereinafter referred to as vehicle width), and matching shape information with the partial shape information. The matching shape information may be a contour, a three-dimensional voxel, a 3D CAD model, or a feature calculated using a mathematical model. This shape information may be set in advance, or may be newly created based on the detection results of the object detection unit 10 or information obtained during past tracking of the vehicle. The shape information may also be updated based on the detection results of the object detection unit 10. The updating method will be described later.

[0017] <<Shape Estimation Unit 12 >> The shape estimation unit 12 estimates the object shape using partial shape information representing the shape of at least a portion of the object detected by the object detection unit 10 and shape information managed by the shape information management unit 11.

[0018] The partial shape extraction unit 12a calculates partial shape information from the object information detected by the object detection unit 10. The partial shape information may be a part of a contour, a part of a three-dimensional voxel, a part of a 3D CAD model, or a part of a feature calculated by a mathematical model.

[0019] The corner determination unit 12b determines whether the partial shape information calculated by the partial shape extraction unit 12a includes a corner of the object. A corner refers to a vertex and two different half lines extending from the vertex. Specifically, if another vehicle V1 is present in front of the host vehicle V0, the corner refers to the right or left end of the rear face of the other vehicle V1. If another vehicle V1 is present behind the host vehicle V0, the corner refers to the right or left end of the front face of the other vehicle V1.

[0020] When the corner part determination unit 12b determines that a corner part is not included, the shape matching unit 12c matches the partial shape information with the matching shape information in the database. The matching flow will be explained below. First, it selects which vehicle in the database the partial shape information should be matched with. This selection is made, for example, using an index that indicates the degree of agreement between the partial shape information and the matching shape information. By referring to the database, the longitudinal length and vehicle width of the oncoming vehicle V1 are estimated. Next, the position of at least one corner part of the vehicle is estimated by matching the partial shape information with the matching shape information.

[0021] <<Position Estimation Unit 13>> The position estimation unit 13 estimates the position of the vehicle using the vehicle shape estimated by the shape estimation unit 12. Using the corners estimated by the shape matching unit 12c as a reference, the remaining corners of the rectangle are calculated by using the vehicle length and vehicle width as the vehicle width information and vehicle length information held in the shape information management unit 11.

[0022] <Operation Example of Object Recognition Device 1> Hereinafter, an operation example of the object recognition device 1 of this embodiment will be described in detail with reference to the flowchart of FIG. 3 . The object recognition device 1 executes this routine each time it receives data from the external sensor 2. In the following operation example, the external sensor 2 is a LiDAR, and the object detection unit 10 performs vehicle detection using a 3D point cloud d3. The shape information management unit 11 manages shape information of passenger cars, trucks, and motorcycles in a database, and pre-sets vehicle width, vehicle length, and matching shape information as shape information. The matching shape information is the outline of the vehicle as seen from the side. The operation scene is assumed to be a scene in which a host vehicle V0 traveling from left to right in the figure and an oncoming vehicle V1 pass each other, as shown in FIG. 4 .

[0023] In step P1, the object detection unit 10 of the object recognition device 1 performs vehicle recognition processing by clustering based on the data of the 3D point cloud d3. Specifically, point clouds with close Euclidean distances between them are clustered. Next, the width, length, and height information of the clustered point cloud is used to determine whether the clustered object is a vehicle. Specifically, it is determined to be a vehicle if the width, length, and height are within a certain threshold range.

[0024] In step P2, the partial shape extraction unit 12a of the object recognition device 1 calculates a partial shape from the point cloud clustered in step P1. The partial shape is the outline of the oncoming vehicle V1 as seen from the side. Figure 5 shows an example of calculating the partial shape. In Figure 5, reference numeral 50 denotes the three-dimensional point cloud d3 clustered in step P1, and reference numeral 51 denotes the outline of the two-dimensional point cloud d2a calculated in step P2.

[0025] A specific method for calculating the partial shape will now be described. First, the three-dimensional point cloud d3 clustered in step P1 is vertically projected onto a plane defined so as to coincide with the side of the vehicle, and converted into a two-dimensional point cloud d2a. Next, the projected two-dimensional point cloud d2a is used to determine a polygon that contains all of the point clouds by Delaunay triangulation, and this polygon is defined as the contour 51 of the two-dimensional point cloud d2a. This contour 51 of the point cloud is defined as the partial shape.

[0026] In step P3, the corner determination unit 12b of the object recognition device 1 determines whether the corners of the vehicle recognized in step P1 have been recognized. A specific determination method will be described. First, the 3D point cloud d3 clustered in step P1 is projected perpendicularly onto the ground and converted into a 2D point cloud d2b. Next, principal component analysis is performed on the projected 2D point cloud d2b to calculate a first principal component 61 and a second principal component 62.

[0027] 6A shows an example of the results of principal component analysis when a corner is recognized, and FIG. 6B shows an example of the results when a corner is not recognized. As can be seen from comparing the two figures, when a corner is included in the observation location, the variance of the direction of the second principal component 62 is larger than when a corner is not included. Therefore, as in FIG. 6A , when the variance of the point cloud in the direction of the second principal component 62 is larger than a threshold, it is determined that a corner is recognized, and when the variance is smaller than the threshold, it is determined that a corner is not recognized, as in FIG. 6B .

[0028] If the corner of the other vehicle V1 can be recognized as shown in Figure 6A, the process proceeds to step P5. On the other hand, if the corner of the other vehicle V1 cannot be recognized as shown in Figure 6B, the process proceeds to step P4. Here, it is assumed that the corner cannot be recognized, and the process proceeds to step P4.

[0029] In step P4, the shape matching unit 12c matches the partial shape calculated in step P2 with pre-stored matching shape information. A specific processing method will be described. First, the vehicle type is determined from the size of the partial shape. Specifically, the vehicle height calculated from the partial shape is compared with the height of a truck, passenger car, or motorcycle calculated from the matching shape information, and the vehicle with the closest height is determined as the type of the oncoming vehicle V1. In this example, it is assumed that the vehicle is determined to be a passenger car. This makes it possible to estimate the length and width of the oncoming vehicle V1 by referencing passenger car information in the database. This makes it possible to determine the rectangular shape of the oncoming vehicle V1 as viewed from above. Next, the partial shape is matched with the matching shape information.

[0030] An example of matching is shown in Figure 7. In Figure 7, 70 indicates the matching shape information of a passenger car, 71 the windshield portion of the partial shape and matching shape information, and 72 the corner portion of the vehicle. As shown in Figure 7, the matching shape information and the partial shape are displayed superimposed, and the partial shape is moved in the longitudinal direction of the vehicle. At this time, matching is performed by searching for the point where the inclined shape of the vehicle's windshield portion included in the partial shape most closely matches the inclined shape of the windshield portion 71 of the matching shape information. Finally, the front tip of the vehicle in the matching shape information is set as corner portion 72, and the position of the corner portion of the other vehicle V1 is calculated.

[0031] In step P5, the position estimation unit 13 calculates the position of the vehicle based on the position of the corner estimated in step P4. Specifically, the remaining corner of the vehicle is calculated as the vehicle length and width estimated in step P4. The center of the estimated rectangle is calculated as the vehicle center position. Note that if step P5 is reached without passing through step P4, the position of the other vehicle V1 can be calculated based on the observed corner.

[0032] The object recognition device 1 of this embodiment described above can provide the following effects.

[0033] (1) When the corners of another vehicle cannot be recognized, the corners can be calculated using the partial shape and the shape information stored in the object information management unit. This makes it possible to estimate the corners even in situations where the corners of vehicles cannot be recognized, such as when passing each other, and to use this information as a basis for estimating the position and shape of the other vehicle.

[0034] (2) By estimating the position of other vehicles using the corners found, it is possible to accurately calculate the position of other vehicles even in situations where corners cannot be estimated. In addition, while general object recognition methods using machine learning require a large amount of training data, this method can calculate the vehicle position without requiring training data.

[0035] (3) The shape information management unit 11 stores shape information for passenger cars, trucks, and motorcycles. Because the size and shape characteristics of vehicles vary greatly depending on the type of vehicle being recognized, recognition processing that targets only one type of vehicle reduces the accuracy of position estimation. However, by storing multiple types of shape information and using them appropriately depending on the type of vehicle being recognized, as in the object recognition device 1, it becomes possible to estimate the position of other vehicles with high accuracy.

[0036] (4) By storing vehicle length, vehicle width, and matching shape information as vehicle shape information managed by the shape information management unit 11, if at least one corner of an object can be recognized or estimated, vehicle position can be easily and highly accurately estimated using the corner as a reference. This reduces the processing load for object recognition and prevents slowing down of processing speed. Furthermore, by storing matching shape information, when a corner cannot be recognized, it is possible to match it with a partial shape, estimate the position of the corner, and accurately calculate the position of another vehicle.

[0037] (5) By storing contour information of an object viewed from the side as matching shape information, it is possible to perform matching and calculate the corners of other vehicles when only the side of the other vehicle is observed, such as when passing each other. Furthermore, because contour information is two-dimensional information, the amount of data is smaller than that of three-dimensional CAD information or three-dimensional voxel information. Therefore, it is possible to reduce the number of object recognition processing failures compared to when storing matching shape information with a large amount of data.

[0038] <Database Update Process> Here, the process of updating the shape information in the database based on information obtained from observation will be described. The assumed scene is a time before the passing of the vehicle in FIG. 4 occurs, and the host vehicle V0 and the other vehicle V1 are positioned facing each other as shown in FIG. 8. Reference numeral 80 in FIG. 8 indicates a 3D point cloud d3 detected by the external sensor 2 of the host vehicle V0. The object type is then estimated using a method similar to that of step P1 and determined to be a passenger car. First, the passenger car shape information in the database is duplicated and registered in the database as passenger car A. At this time, the vehicle length, vehicle width, and matching shape information are inherited from the information of the original. Next, the values ​​in the database are updated. As for the vehicle width, an accurate value can be observed from the positional relationship between the vehicles, so the vehicle width of passenger car A in the database is updated to the vehicle width obtained from observation. As for the vehicle length, an accurate value cannot be observed from the positional relationship between the vehicles, so it is not updated.

[0039] A portion of the matching shape information is updated using the method shown in FIG. 9. Reference numeral 90 in FIG. 9 indicates the clustered point cloud. First, a partial shape is extracted using the same method as in step P2. Reference numeral 91 in FIG. 9 indicates the extracted partial shape. Next, the matching shape information of passenger car A is matched with the partial shape using the same method as in step P4. Finally, the portion of the matching shape information in the database that has been obtained as the partial shape is replaced.

[0040] As described above, the shape information managed by the shape information management unit 11 is updated using the object information detected by the detection unit. This makes it possible to estimate the vehicle position with higher accuracy compared to when position estimation is performed using pre-set shape information. Furthermore, by updating the shape information through observation, it becomes possible to estimate the vehicle position with higher accuracy even when only a small amount of shape information is stored in advance. This makes it possible to prevent the object recognition device 1 from consuming too much memory and from slowing down its processing speed.

[0041] The matching of contours described so far focuses on the shape of the windshield, but in the present invention, it is also possible to determine which part of the vehicle shape to focus on for matching based on point cloud information obtained from observation. A specific implementation method will now be described. Consider the scene in Figure 8 where two vehicles are facing each other.

[0042] First, the characteristic shape of the vehicle is identified as shown in FIG. 10 . In FIG. 10 , 100 indicates a portion of the partial shape with a small curvature, and 101 indicates a portion of the partial shape with a large curvature. Here, it is assumed that the portion with a large curvature is useful for matching. Next, the portion with a large curvature such as 101 is stored together with matching shape information in a database. Finally, when matching the partial shape with the matching shape information, weighting is performed to emphasize the degree of match of the portion with a large curvature. This allows the position of the oncoming vehicle V1 to be estimated with higher accuracy than when weighting is not performed. Furthermore, there are usually an infinite number of vehicle shapes, and it is difficult to set matching weights for all of them in advance. However, according to this embodiment, weighting can be performed while the vehicle is traveling without setting matching weights in advance.

[0043] In addition to the above-described embodiment, the present invention can switch the matching method depending on the distance between the host vehicle V0 and the oncoming vehicle V1. Consider a case where the vehicles are so close to each other that their contours cannot be detected. In this case, the contour matching method described above cannot accurately estimate the vehicle positions. Therefore, when the vehicles are close to each other, matching based on feature amounts is performed, as shown in FIG. 11 . The feature amounts may be the uneven shape of the vehicles or the brightness values ​​of the point cloud obtained from the vehicles. From now on, matching will be performed using the uneven shape as the feature amount. First, before the vehicles are close to each other, the feature amounts of the side of the vehicles are calculated, and the corresponding parts of the matching shape information in the database are stored in correspondence with each other.

[0044] Reference numeral 110 in FIG. 11 indicates a particularly large portion of the calculated unevenness. When vehicles are close to each other, feature quantities are also calculated for the partial shape. Reference numeral 111 in FIG. 11 indicates the partial shape, and reference numeral 112 indicates a particularly large portion of the unevenness included in the partial shape. In this case, matching is performed so that the uneven shape included in the partial shape matches the uneven shape portion in the database. By using this in conjunction with the aforementioned contour matching method, it is possible to improve the accuracy of vehicle position estimation regardless of the distance between the host vehicle V0 and the oncoming vehicle V1.

[0045] In addition to the above-described embodiments, the present invention allows for the use of vehicle speed information for matching. Consider a case where a vehicle is tracked and the vehicle speed at a time prior to the present can be estimated. In this case, by assuming that the amount of change in speed from the previous time to the present is within a certain range, the current vehicle position can be narrowed down using the speed and the operating cycle of the object recognition device 1. Using this, matching is performed within the narrowed range of vehicle positions. This reduces the search range for matching, thereby preventing processing slowdowns.

[0046] Although the present invention has been described above with reference to one embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0047] REFERENCE SIGNS LIST 1 object recognition device 10 object detection unit 11 shape information management unit 12 shape estimation unit 12a partial shape extraction unit 12b corner part determination unit 12c shape matching unit 13 position estimation unit 2 external sensor

Claims

1. An object recognition device comprising: a detection unit that detects an object using external world information acquired by an external sensor; a shape information management unit that manages shape information representing the shape of the object; a shape estimation unit that estimates the shape of the object based on the shape information and partial shape information that represents at least a part of the shape of the object obtained from the external world information; and a position estimation unit that estimates the position of the object using the estimated shape of the object.

2. An object recognition device as described in claim 1, wherein the detection unit detects an object using a point cloud acquired by the external sensor, and the shape estimation unit compares the shape information with partial shape information that represents the shape of at least a portion of the object as a point cloud, to estimate the shape of the object.

3. An object recognition device as described in claim 1, wherein the shape estimation unit determines the position of at least one corner of a rectangle when the object is represented as a rectangle, and estimates the shape of the object using the position of the corner.

4. An object recognition device as described in claim 1, wherein the shape estimation unit estimates the shape of the object using the shape information and the partial shape information when the corners of the rectangle when the object is represented as a rectangle are not included in the object detected by the detection unit.

5. An object recognition device as described in claim 1, wherein the shape information management unit stores at least one of preset shape information and shape information acquired by the external sensor, and updates the stored shape information using the shape information acquired by the external sensor.

6. An object recognition device according to claim 1, wherein the shape information management unit holds contour information of an object as viewed from the side as the shape information.

7. An object recognition method executed by a computing device, comprising: a detection step of detecting an object using external world information acquired by an external sensor; a shape estimation step of estimating a shape of the object based on shape information representing the shape of the object and partial shape information representing at least a part of the shape of the object obtained from the external world information; and a position estimation step of estimating a position of the object using the estimated shape of the object.

Images