Object recognition device and object recognition method

Abstract

To perform accurate shape and position estimation of a vehicle when an angle of the vehicle cannot be observed.SOLUTION: An object recognition device comprises: a detection unit that detects an object using external field information acquired by an external field sensor; a shape information management unit that manages shape information which represents a shape of an object; a shape estimation unit that estimates the shape of the object based on the shape information and partial shape information representing the shape of at least a part of the object obtained by the external field information; and a position estimation unit that estimates a position of the object using the estimated shape of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object recognition device and an object recognition method for recognizing the position of another vehicle in the vicinity of a vehicle. [Background technology]

[0002] In order to realize preventive safety functions and autonomous driving, it is effective to detect surrounding objects using on-board sensors and perform recognition processing such as image processing and point cloud processing. When the relative positions of the vehicle and other vehicles change, the observation point changes, for example, from the front of the vehicle to the side of the vehicle, making the position estimation unstable. Therefore, a position estimation method that takes the observation point into account is required.

[0003] In Patent Document 1, the solution section of the abstract explains that "an object recognition device 3 for recognizing other vehicles present around a vehicle, which determines whether or not there is a partially recognized vehicle among multiple recognized other vehicles that is partially hidden by the other vehicle present in front, and if a partially recognized vehicle is present and the corners of the partially recognized vehicle can be recognized, the longitudinal length and lateral length of the partially recognized vehicle are set to predetermined longitudinal and lateral lengths, and the center position of the partially recognized vehicle is calculated based on the corners of the partially recognized vehicle." As such, the method discloses a method of identifying observation points by observing the corners of a vehicle and estimating the vehicle position based on the corners. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-011745 Summary of the Invention [Problem to be solved by the invention]

[0005] By using the method described in Patent Document 1, it is possible to easily calculate the center position of a partially recognized vehicle that is partially hidden by another vehicle in front, based on the corners of the partially recognized vehicle. However, there is a problem in that accurate estimation is impossible if the corners cannot be observed, such as when vehicles approach each other, such as when passing each other.

[0006] In view of the above, an object of the present invention is to provide an object recognition device and an object recognition method that can accurately estimate the shape and position of a vehicle even when the vehicle angle cannot be observed. [Means for solving the problem]

[0007] In order to solve the above problem, the object recognition device of the present invention is an object recognition device that includes a detection unit that detects an object using external world information acquired by an external world sensor, a shape information management unit that manages shape information that represents the shape of the object, a shape estimation unit that estimates the shape of the object based on the shape information and partial shape information that represents the shape of at least a part of the object obtained from the external world information, and a position estimation unit that estimates the position of the object using the estimated shape of the object. [Effects of the Invention]

[0008] According to the present invention, when the vehicle angle cannot be observed, the shape and position of the vehicle can be accurately estimated. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram showing the configuration of an object recognition device according to an embodiment. [Figure 2] FIG. 4 is a diagram for explaining a database held by a shape information management unit. [Figure 3] Processing flow diagram of an object recognition device according to an embodiment [Figure 4] A diagram explaining the passing scene [Figure 5] 5A to 5C are diagrams for explaining partial shape extraction by a partial shape extraction unit. [Figure 6A]10A and 10B are diagrams illustrating a method for determining corner portions using principal component analysis. [Figure 6B] 10A and 10B are diagrams illustrating a method for determining corner portions using principal component analysis. [Figure 7] 5A and 5B are diagrams illustrating shape matching by a shape matching unit. [Figure 8] FIG. 10 is a diagram illustrating a scene in which shape information is updated. [Figure 9] FIG. 10 is a diagram illustrating updating of matching shape information. [Figure 10] 1 is a diagram illustrating a characteristic shape of a vehicle; [Figure 11] FIG. 10 is a diagram showing an example of matching based on feature amounts. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail.

[0011] FIG. 1 is a functional block diagram showing the configuration of an object recognition device 1 according to an embodiment of the present invention. The object recognition device 1 shown here is an in-vehicle device connected to an external sensor 2 such as a LiDAR, and is a device that recognizes the type and position of objects near the host vehicle V0, particularly the type and position of other vehicles, based on the output of the external sensor 2. Specifically, the object recognition device 1 is a computer equipped with an arithmetic unit, a storage device, etc., and operates as various functional units described below by the arithmetic unit executing a predetermined control program. The external sensor 2 is, for example, a LiDAR disposed at the four corners of the host vehicle V0: the front left, front right, rear left, and rear right.

[0012] Here, the shape of a vehicle viewed from above is generally rectangular regardless of the type of vehicle (passenger car, truck, motorcycle, etc.). Therefore, by recognizing or estimating the corners of the vehicle, the position of the vehicle can be easily calculated using the corners as a reference. In general, data-driven methods such as deep learning require a large amount of calculation, which slows down the processing speed of the object recognition device 1. In contrast, a position estimation method using corners as a reference requires a small amount of calculation, which can prevent the recognition process from slowing down. Therefore, in the object recognition device 1 of this embodiment, the shape of a vehicle viewed from above is approximated as a rectangle, and vehicle type determination and position determination are easily performed under this assumption.

[0013] <Object recognition device 1> 1, object recognition device 1 includes object detection unit 10, shape information management unit 11, shape estimation unit 12, and position estimation unit 13. Shape estimation unit 12 also includes partial shape extraction unit 12a, corner portion determination unit 12b, and shape matching unit 12c. Each unit will be described in detail below.

[0014] <<Object detection unit 10>> The object detection unit 10 detects the type of object using the measurement information of the external sensor 2. The object to be detected is the type of vehicle, such as a passenger car, truck, or motorcycle.

[0015] <<Shape information management section 11>> The shape information management unit 11 manages the shape information of objects as a database. This database may be stored in a storage device inside the vehicle, or may be provided outside the vehicle and the information from the database may be acquired via communication. In Fig. 2, the shape information management unit 11 stores the shape information for each vehicle type to be detected in databases such as a passenger car database 11a, a truck database 11b, and a two-wheeled vehicle database 11c, and each database stores shape information for each vehicle type.

[0016] The shape information includes information on the length of the vehicle in the longitudinal direction (hereinafter referred to as vehicle length), information on the length in the lateral direction (hereinafter referred to as vehicle width), and matching shape information with the partial shape information. The matching shape information may be a contour, a 3D voxel, a 3D CAD model, or a feature calculated by a mathematical model. This shape information may be set in advance, or may be newly created based on the detection results of the object detection unit 10 or information obtained during past tracking of the vehicle. The shape information may also be updated based on the detection results of the object detection unit 10. The updating method will be described later.

[0017] <<Shape estimation unit 12>> The shape estimation unit 12 estimates the shape of an object using partial shape information representing the shape of at least a part of the object detected by the object detection unit 10 and shape information managed by the shape information management unit 11 .

[0018] The partial shape extraction unit 12a calculates partial shape information from the object information detected by the object detection unit 10. The partial shape information may be a part of a contour, a part of a 3D voxel, a part of a 3D CAD model, or a part of a feature calculated by a mathematical model.

[0019] The corner determination unit 12b determines whether the partial shape information calculated by the partial shape extraction unit 12a includes a corner of the object. A corner refers to a vertex and two different half lines extending from the vertex. Specifically, if another vehicle V1 is present in front of the host vehicle V0, the corner refers to the right or left end of the rear face of the other vehicle V1. If another vehicle V1 is present behind the host vehicle V0, the corner refers to the right or left end of the front face of the other vehicle V1.

[0020] When the corner determination unit 12b determines that a corner is not included, the shape matching unit 12c matches the partial shape information with the matching shape information in the database. The matching flow will be explained below. First, it selects which vehicle in the database the partial shape information should be matched with. This is done, for example, using an index that indicates the degree of agreement between the partial shape information and the matching shape information. By referring to the database, the longitudinal length and vehicle width of the oncoming vehicle V1 are estimated. Next, the position of at least one corner of the vehicle is estimated by matching the partial shape information with the matching shape information.

[0021] <<Position estimation unit 13>> The position estimation unit 13 estimates the position of the vehicle using the vehicle shape estimated by the shape estimation unit 12. Using the corners estimated by the shape matching unit 12c as a reference, the remaining corners of the rectangle are calculated by using the vehicle length and width as the vehicle width information and vehicle length information held in the shape information management unit 11.

[0022] <Example of operation of object recognition device 1> Hereinafter, an example of the operation of the object recognition device 1 of this embodiment will be described in detail with reference to the flowchart of FIG. 3. The object recognition device 1 executes this routine each time it receives data from the external sensor 2. In the following example of operation, the external sensor 2 is a LiDAR, and the object detection unit 10 performs vehicle detection using a 3D point cloud d3. The shape information management unit 11 manages shape information of passenger cars, trucks, and motorcycles in a database, and pre-sets vehicle width, vehicle length, and matching shape information as shape information. The matching shape information is the outline of the vehicle as seen from the side. The operating scene is assumed to be a scene in which a host vehicle V0 traveling from left to right in the figure and an oncoming vehicle V1 pass each other, as shown in FIG. 4.

[0023] In step P1, the object detection unit 10 of the object recognition device 1 performs vehicle recognition processing by clustering based on the data of the 3D point cloud d3. Specifically, point clouds with close Euclidean distances between them are clustered. Next, the width, length, and height information of the clustered point cloud is used to determine whether the clustered object is a vehicle. Specifically, it is determined to be a vehicle if the width, length, and height are within a certain threshold range.

[0024] In step P2, the partial shape extraction unit 12a of the object recognition device 1 calculates a partial shape from the point cloud clustered in step P1. The partial shape is the outline of the oncoming vehicle V1 as seen from the side. Fig. 5 shows an example of calculating the partial shape. In Fig. 5, 50 indicates the three-dimensional point cloud d3 clustered in step P1, and 51 indicates the outline of the two-dimensional point cloud d2a calculated in step P2.

[0025] A specific method for calculating the partial shape will now be described. First, the 3D point cloud d3 clustered in step P1 is projected perpendicularly onto a plane defined to coincide with the side of the vehicle, and converted into a 2D point cloud d2a. Next, the projected 2D point cloud d2a is used to determine a polygon that contains all of the point clouds by Delaunay triangulation, and this is taken as the outline 51 of the 2D point cloud d2a. This outline 51 of the point cloud is taken as the partial shape.

[0026] In step P3, the corner determination unit 12b of the object recognition device 1 determines whether the corners of the vehicle recognized in step P1 have been recognized. A specific determination method will be described. First, the 3D point cloud d3 clustered in step P1 is projected perpendicularly onto the ground and converted into a 2D point cloud d2b. Next, principal component analysis is performed on the projected 2D point cloud d2b to calculate a first principal component 61 and a second principal component 62.

[0027] Figure 6A shows an example of the results of principal component analysis when a corner is recognized, and Figure 6B shows an example of the results when a corner is not recognized. As can be seen from comparing the two figures, when a corner is included in the observation location, the variance of the direction of the second principal component 62 is larger than when a corner is not included. Therefore, as in Figure 6A, when the variance of the point cloud in the direction of the second principal component 62 is larger than the threshold, it is determined that a corner is recognized, and when the variance is smaller than the threshold, it is determined that a corner is not recognized, as in Figure 6B.

[0028] If the corner of the other vehicle V1 can be recognized as shown in Figure 6A, the process proceeds to step P5. On the other hand, if the corner of the other vehicle V1 cannot be recognized as shown in Figure 6B, the process proceeds to step P4. Here, it is assumed that the corner cannot be recognized, and the process proceeds to step P4.

[0029] In step P4, the shape matching unit 12c matches the partial shape calculated in step P2 with pre-stored matching shape information. The specific processing method will be described below. First, the type of vehicle is determined from the size of the partial shape. Specifically, the vehicle height calculated from the partial shape is compared with the height of a truck, passenger car, or motorcycle calculated from the matching shape information, and the vehicle with the closest height is determined as the type of oncoming vehicle V1. In this case, it is assumed that the vehicle is determined to be a passenger car. This makes it possible to estimate the length and width of the oncoming vehicle V1 by referring to passenger car information in the database. This makes it possible to determine the rectangular shape of the oncoming vehicle V1 when viewed from above. Next, the partial shape is matched with the matching shape information.

[0030] An example of matching is shown in Figure 7. In Figure 7, 70 indicates the matching shape information of a passenger car, 71 the windshield portion of the partial shape and matching shape information, and 72 the corner of the vehicle. As shown in Figure 7, the matching shape information and partial shape are displayed superimposed, and the partial shape is moved in the longitudinal direction of the vehicle. At this time, matching is performed by searching for the point where the inclined shape of the vehicle's windshield portion included in the partial shape most closely matches the inclined shape of the windshield portion 71 of the matching shape information. Finally, the front tip of the vehicle in the matching shape information is set as corner portion 72, and the position of the corner of the other vehicle V1 is calculated.

[0031] In step P5, the position estimation unit 13 calculates the position of the vehicle based on the position of the corner estimated in step P4. Specifically, the remaining corner of the vehicle is calculated as the vehicle length and width estimated in step P4. The center of the estimated rectangle is calculated as the vehicle center position. Note that if step P5 is reached without passing through step P4, the position of the other vehicle V1 can be calculated based on the observed corner.

[0032] The object recognition device 1 of this embodiment described above can provide the following effects.

[0033] (1) When the corners of another vehicle cannot be recognized, the corners can be calculated using the partial shape and the shape information stored in the object information management unit. This makes it possible to estimate the corners even in situations where the corners of vehicles cannot be recognized, such as when passing each other, and to use this information as a basis for estimating the position and shape of another vehicle.

[0034] (2) By estimating the position of other vehicles using the corners found, it is possible to accurately calculate the position of other vehicles even in situations where corners cannot be estimated. In addition, while general object recognition methods using machine learning require a large amount of training data, this method can calculate the vehicle position without requiring training data.

[0035] (3) The shape information management unit 11 stores shape information for passenger cars, trucks, and motorcycles. Because the size and shape characteristics of vehicles vary greatly depending on the type of vehicle being recognized, recognition processing that targets only one type of vehicle reduces the accuracy of position estimation. However, by storing multiple types of shape information and using them appropriately depending on the type of vehicle being recognized, as in the object recognition device 1, it becomes possible to estimate the position of other vehicles with high accuracy.

[0036] (4) By storing vehicle length, vehicle width, and matching shape information as vehicle shape information managed by the shape information management unit 11, if at least one corner of an object can be recognized or estimated, vehicle position can be easily and highly accurately estimated using the corner as a reference. This reduces the processing load for object recognition and prevents slowing down of processing speed. Furthermore, by storing matching shape information, when a corner cannot be recognized, it is possible to match it with a partial shape, estimate the position of the corner, and accurately calculate the position of another vehicle.

[0037] (5) By storing contour information of an object viewed from the side as matching shape information, it is possible to perform matching and calculate the corners of other vehicles when only the side of the other vehicle is observed, such as when passing each other. In addition, because contour information is two-dimensional information, the amount of data is smaller than that of 3D CAD information or 3D voxel information. Therefore, compared to storing matching shape information with a large amount of data, it is possible to reduce the number of object recognition processing failures.

[0038] <Database update process> Here, we will explain the process of updating the shape information in the database based on information obtained from observations. The assumed scene is a time before the passing of Figure 4 occurs, and the host vehicle V0 and the other vehicle V1 are positioned facing each other as shown in Figure 8. Reference numeral 80 in Figure 8 indicates the 3D point cloud d3 detected by the external sensor 2 of the host vehicle V0. The object type is then estimated using the same method as in step P1 and determined to be a passenger car. First, the passenger car shape information in the database is duplicated and registered in the database as passenger car A. At this time, the vehicle length, vehicle width, and matching shape information are inherited from the original information. Next, the values ​​in the database are updated. As for the vehicle width, an accurate value can be observed from the positional relationship between the vehicles, so the vehicle width of passenger car A in the database is updated to the vehicle width obtained from observations. As for the vehicle length, an accurate value cannot be observed from the positional relationship between the vehicles, so it is not updated.

[0039] A portion of the matching shape information is updated using the method shown in Figure 9. 90 in Figure 9 shows the clustered point cloud. First, a partial shape is extracted using the same method as in step P2. 91 in Figure 9 shows the extracted partial shape. Next, the matching shape information of passenger car A is matched with the partial shape using the same method as in step P4. Finally, the portion of the matching shape information in the database that has been obtained as the partial shape is replaced.

[0040] As described above, the shape information managed by the shape information management unit 11 is updated using the object information detected by the detection unit. This makes it possible to estimate the vehicle position with higher accuracy compared to when position estimation is performed using pre-set shape information. Furthermore, by updating the shape information through observation, it becomes possible to estimate the vehicle position with higher accuracy even when only a small amount of shape information is stored in advance. This makes it possible to prevent the object recognition device 1 from consuming too much memory and from slowing down its processing speed.

[0041] The matching of contours explained so far is performed by focusing on the shape of the windshield, but in this invention, it is also possible to determine which part of the vehicle shape to focus on for matching based on point cloud information obtained from observation. A specific implementation method will be explained. Consider the scene in Figure 8 where two vehicles are facing each other.

[0042] First, the characteristic shape of the vehicle is identified as shown in FIG. 10. In FIG. 10, 100 indicates a portion of the partial shape with a small curvature, and 101 indicates a portion of the partial shape with a large curvature. Here, it is assumed that the portion with a large curvature is useful for matching. Next, the portion with a large curvature such as 101 is stored together with the matching shape information in the database. Finally, when matching the partial shape with the matching shape information, weighting is performed to emphasize the degree of match of the portion with a large curvature. This allows the position of the oncoming vehicle V1 to be estimated with higher accuracy than when weighting is not performed. Furthermore, there are usually an infinite number of vehicle shapes, and it is difficult to set matching weights for all of them in advance. However, according to this embodiment, weighting can be performed while the vehicle is traveling without setting matching weights in advance.

[0043] In addition to the above-described embodiment, the present invention can switch the matching method depending on the distance between the host vehicle V0 and the oncoming vehicle V1. Consider a case where the vehicles are so close to each other that their contours cannot be detected. In this case, the contour matching method described above cannot accurately estimate the vehicle position. Therefore, when the vehicles are close to each other, matching based on feature amounts is performed, as shown in FIG. 11. The feature amount may be the uneven shape of the vehicle, or the brightness value of the point cloud obtained from the vehicle. From now on, matching will be performed using the uneven shape as the feature amount. First, before the vehicles are close to each other, the feature amount of the side of the vehicle is calculated, and the part that corresponds to the feature amount in the matching shape information in the database is stored in correspondence with the calculated feature amount.

[0044] In Figure 11, 110 indicates a particularly large portion of the calculated unevenness. When vehicles are close to each other, feature amounts are also calculated for the partial shape. In Figure 11, 111 indicates the partial shape, and 112 indicates a particularly large portion of the unevenness included in the partial shape. In this case, matching is performed so that the uneven shape included in the partial shape matches the uneven shape portion in the database. By using this in conjunction with the contour matching method described above, it is possible to improve the accuracy of vehicle position estimation regardless of the distance between the host vehicle V0 and the oncoming vehicle V1.

[0045] In addition to the above-described embodiments, the present invention allows for the use of vehicle speed information for matching. Consider a case where a vehicle is tracked and the vehicle speed at a time prior to the present can be estimated. In this case, by assuming that the amount of change in speed from the previous time to the present is within a certain range, the current vehicle position can be narrowed down using the speed and the operating cycle of the object recognition device 1. Using this, matching is performed within the narrowed range of vehicle positions. This reduces the search range for matching, thereby preventing processing slowdowns.

[0046] Although the present invention has been described above with reference to one embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0047] 1 Object recognition device 10 Object detection unit 11 Shape information management department 12 Shape estimation section 12a Partial shape extraction part 12b Corner part determination section 12c Shape Matching Section 13 Position estimation part 2. External sensors

Claims

1. A detection unit that detects an object using external world information acquired by an external world sensor; a shape information management unit that manages shape information representing a shape of an object; a shape estimation unit that estimates a shape of the object based on the shape information and partial shape information that represents a shape of at least a part of the object obtained from the external environment information; a position estimation unit that estimates a position of the object using the estimated shape of the object; An object recognition device comprising:

2. The object recognition device according to claim 1, The detection unit detects an object using a point cloud acquired by the external sensor, The shape estimation unit compares the shape information with partial shape information that represents a shape of at least a portion of the object as a point cloud to estimate the shape of the object.

3. The object recognition device according to claim 1, The shape estimation unit determines the position of at least one corner of a rectangle when the object is represented as a rectangle, and estimates the shape of the object using the position of the corner.

4. The object recognition device according to claim 1, An object recognition device, wherein the shape estimation unit estimates the shape of the object using the shape information and the partial shape information when the corners of the rectangle when the object is represented as a rectangle are not included in the object detected by the detection unit.

5. The object recognition device according to claim 1, An object recognition device, wherein the shape information management unit stores at least one of predetermined shape information and shape information acquired by the external sensor, and updates the stored shape information using the shape information acquired by the external sensor.

6. The object recognition device according to claim 1, The shape information management unit holds contour information of an object as viewed from the side as the shape information.

7. 1. A method for object recognition executed by a computing device, comprising: A detection step of detecting an object using external information acquired by an external sensor; a shape estimation step of estimating a shape of the object based on shape information representing a shape of the object and partial shape information representing a shape of at least a part of the object obtained from the external environment information; a position estimation step of estimating a position of the object using the estimated shape of the object; The object recognition method comprises:

Images