OBJECT RECOGNITION METHOD AND OBJECT RECOGNITION DEVICE

Abstract

This object recognition method comprises generating clusters at a fixed cycle by clustering distance measurement points acquired from an object detection sensor, correlating the clusters on an image captured by an imaging means, setting a region on an image to which a cluster corresponds as a processing region, extracting a feature point from the image in the set processing region, calculating the movement speed of the extracted feature point, and determining whether a cluster from a different time point is a cluster of the same object on the basis of the difference in position between two feature points corresponding to the cluster from the different time and the difference in movement speed between the two feature points corresponding to the cluster from the different time.

Description

[Technical field]

[0001] The present invention relates to an object recognition method and an apparatus for recognizing objects existing around a vehicle. [Background technology]

[0002] Conventionally, an object detection device for detecting objects around a vehicle is disclosed in Patent Document 1. In the object detection device disclosed in Patent Document 1, measurement points acquired by a LIDAR are clustered into one or more clusters, and the clusters are fitted to a shape model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-129446 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional object detection device, when fitting a cluster to a shape model, the moving speed of the cluster is used to determine whether or not to fit the cluster to the shape model. However, even if the measurement points are on the same object, the locations that are clustered may differ at different times, making it difficult to accurately measure the moving speed of the cluster.

[0005] For example, when the rear of the vehicle is hidden in the shadow of a building, the measurement points in the front part of the vehicle are clustered, but when the front of the vehicle is hidden in the shadow of a building, the measurement points in the rear part of the vehicle are clustered. Therefore, even if it is the same vehicle, different points are clustered at different times, so it is impossible to accurately measure the moving speed of the cluster.

[0006] As described above, conventional object detection devices have the problem that they cannot accurately measure the moving speed of a cluster, and therefore cannot accurately recognize objects around the vehicle as the same object at different points in time.

[0007] Therefore, the present invention has been proposed in consideration of the above-mentioned situation, and has an object to provide an object recognition method and device therefor that can accurately recognize objects around a vehicle as the same object at different points in time. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, an object recognition method and an object recognition device according to an embodiment of the present invention clusters distance measurement points acquired from an object detection sensor to generate clusters at a constant cycle. Then, the clusters are associated with an image captured by an imaging means, an area on the image to which the clusters correspond is set as a processing area, feature points are extracted from the image within the set processing area, and the movement speed of the extracted feature points is calculated. As a result, it is determined whether the clusters at different times are clusters of the same object based on the difference in position between two feature points corresponding to the clusters at different times and the difference in movement speed between two feature points corresponding to the clusters at different times. Effect of the Invention

[0009] According to the present invention, objects around a vehicle can be recognized as being exactly the same object at different times. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a vehicle equipped with an object recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram for explaining a method of determining the same object by the object recognition device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining a method of determining the same object by the object recognition device according to the first embodiment. [Figure 4A]FIG. 4A is a flowchart showing a processing procedure of the object recognition processing by the object recognition device according to the first embodiment. [Figure 4B] FIG. 4B is a flowchart showing the processing procedure of the object recognition processing by the object recognition device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of an evaluation matrix generated by the object recognition device according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of an evaluation matrix generated by the object recognition device according to the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining clustering by the object recognition device according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an evaluation matrix generated by the object recognition device according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of an evaluation matrix generated by the object recognition device according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining clustering by a conventional object detection device. [Figure 11] FIG. 11 is a block diagram showing the configuration of a vehicle equipped with an object recognition device according to the second embodiment. [Figure 12] FIG. 12 is a diagram for explaining a method of determining the same object by the object recognition device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [First embodiment] A first embodiment of the present invention will now be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and detailed description will be omitted.

[0012] [Configuration of a vehicle equipped with an object recognition device] Fig. 1 is a block diagram showing the configuration of a vehicle equipped with an object recognition device according to this embodiment. As shown in Fig. 1, the vehicle 100 includes an object recognition device 1, an object detection sensor 3, and a camera 5. When the vehicle 100 starts traveling, the object recognition device 1 detects objects present around the vehicle 100 using distance measurement points detected by the object detection sensor 3 and images captured by the camera 5. Then, the object recognition device 1 determines whether the detected object is the same object at different points in time, and tracks the detected object.

[0013] The object detection sensor 3 detects objects present around the vehicle 100, and detects ranging points that measure the distance to each point on the detected object. The detected ranging points are output to the object recognition device 1 and recorded in a storage device such as a memory (not shown). The object detection sensor 3 may be any sensor that can detect the distance to an object present around the vehicle 100. For example, a LiDAR (Light Detection and Ranging), a laser radar, a stereo camera, or a ToF (Time of Flight) camera can be used.

[0014] The camera 5 is an imaging means having an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 5 is mounted on the vehicle 100 and captures an image of the surroundings of the vehicle 100. The captured image is output to the object recognition device 1 and recorded in a storage device such as a memory (not shown).

[0015] The object recognition device 1 is mounted on a vehicle 100 and recognizes objects existing around the vehicle 100. In particular, the object recognition device 1 clusters the distance measurement points acquired from the object detection sensor 3 to generate clusters at a constant period, and sets a processing area by corresponding the clusters to an image captured by the camera 5. Then, feature points are extracted from the image in the processing area, the moving speed of the feature points is calculated, and based on the difference in position and the difference in moving speed between two feature points corresponding to clusters at different times, it is determined whether the clusters at different times are clusters of the same object. The object recognition device 1 includes a clustering unit 11, a processing area setting unit 13, a feature point extraction unit 15, a moving speed calculation unit 17, and a same object determination unit 19.

[0016] The clustering unit 11 acquires distance measurement points detected by the object detection sensor 3, and generates clusters at a fixed period by performing clustering processing on the acquired distance measurement points. In the clustering processing, among the acquired distance measurement points, multiple distance measurement points that exist within a predetermined distance from each other are grouped (clustered) to generate clusters.

[0017] The processing area setting unit 13 acquires an image of the surroundings of the vehicle 100 captured by the camera 5, matches the clusters generated by the clustering unit 11 to the acquired image, and sets the area on the image to which the cluster corresponds as the processing area.

[0018] 2, for example, a plurality of ranging points 21a detected by the object detection sensor 3 at time t=k are clustered to generate a cluster C1. The generated cluster C1 is then superimposed on an image 23a captured by a camera 5, and the area on the image 23a on which the cluster C1 is superimposed is set as a processing area 25a. Similarly, at time t=k+1, a cluster C10 generated from the ranging point 21b is superimposed on the image 23b to set a processing area 25b.

[0019] The feature point extraction unit 15 extracts feature points from an image in a processing area set by the processing area setting unit 13. For example, the feature point extraction unit 15 extracts feature points 27a, 27b with high feature amounts from processing areas 25a, 25b of images 23a, 23b in FIG.

[0020] A feature point is, for example, an edge point that has a larger difference in luminance or hue with the surrounding pixels than other parts of the image, or a point that has a characteristic relationship with the surrounding pixels in terms of change in luminance or hue. In other words, a feature point means a pixel or a set of pixels that has a unique characteristic that can be distinguished from other pixels. The method of extracting such feature points on an image is a generally known method, and any known method can be applied as appropriate, without being limited to the above.

[0021] The moving speed calculation unit 17 calculates the moving speed of the feature points extracted by the feature point extraction unit 15. Specifically, the moving speed calculation unit 17 matches the extracted feature points with feature points on the image of the previous frame by template matching or the like, and calculates an optical flow from the coordinates of the matched feature points. The optical flow is a vector representation of the movement of an object in an image. The moving speed calculation unit 17 then averages the velocity vectors of the optical flow within the processing area to calculate the moving speed of the feature points. Note that the method of calculating the optical flow is a generally known technique, and will not be described in detail here.

[0022] For example, as shown in Fig. 2, feature points 27b extracted from image 23b at time t=k+1 are matched with feature points 27a extracted from image 23a at time t=k, and optical flow 29b is calculated from the coordinates of the matched feature points. Then, velocity vectors of multiple optical flows 29b in processing area 25b are averaged to calculate a velocity vector (vx, vy). This velocity vector is the moving velocity of the feature point corresponding to cluster C10 at time t=k+1.

[0023] The same object determination unit 19 determines whether the clusters at different times are clusters of the same object based on the difference in the positions of two feature points corresponding to the clusters at different times and the difference in the moving speeds of two feature points corresponding to the clusters at different times.

[0024] For example, a case will be described in which it is determined whether clusters C1, C2, and C3 detected at time t=k and clusters C10, C20, and C30 detected at time t=k+1 are the same object, as shown in Fig. 3. The same object determination unit 19 moves the feature points in the clusters C1, C2, and C3 by the time difference between time t=k and time t=k+1 according to the moving speed calculated by the moving speed calculation unit 17, and obtains clusters C1', C2', and C3'.

[0025] Then, the same object determination unit 19 sets combinations between the clusters C1', C2', C3' and the clusters C10, C20, C30, calculates the degree of match for each combination, and determines whether or not the clusters are of the same object based on the degree of match.

[0026] The degree of coincidence f indicates the degree of coincidence between clusters at different points in time, and can be calculated by the following formula (1). f = a × (difference in the positions of feature points corresponding to clusters) 2 + b × (difference in moving speed of feature points corresponding to clusters) 2 (1) where a and b are coefficients.

[0027] In this formula for the degree of match f, a first evaluation point is calculated based on the difference in the positions of two feature points corresponding to clusters at different times, and a second evaluation point is calculated based on the difference in the movement speeds of two feature points corresponding to clusters at different times.The degree of match is then calculated based on these first and second evaluation points.As shown in formula (1), the degree of match is calculated using the difference in the positions of the feature points and the difference in the movement speeds, so the smaller the degree of match, the higher the degree of match between two clusters at different times.

[0028] Specifically, the first evaluation point is the square of the difference in the positions of two feature points corresponding to clusters at different times, and the second evaluation point is the square of the difference in the moving speeds of two feature points corresponding to clusters at different times. Then, a first coefficient a to be multiplied by the first evaluation point and a second coefficient b to be multiplied by the second evaluation point are set, and the value obtained by multiplying the first evaluation point by the first coefficient a and the value obtained by multiplying the second evaluation point by the second coefficient b are added together to calculate the degree of match f. Note that the first evaluation point and the second evaluation point are respectively the position and the speed, and therefore have different units. Therefore, the first coefficient a and the second coefficient b are set so as to normalize the difference in units.

[0029] For example, in the case of a combination of clusters C1' and C10 in FIG. 3, first, the difference between the position of the feature point corresponding to cluster C1' and the position of the feature point corresponding to cluster C10 is calculated. Similarly, the difference between the moving speed of the feature point corresponding to cluster C1' and the moving speed of the feature point corresponding to cluster C10 is calculated, and the difference is squared, multiplied by coefficients a and b, and added, to calculate the degree of match f. As a result, if the degree of match f is sufficiently small, it can be determined that clusters C1 and C10 at different times are clusters of the same object. However, the detailed determination method will be described later.

[0030] The object recognition device 1 is a controller configured with general-purpose electronic circuits including a microcomputer, a microprocessor, and a CPU, and peripheral devices such as a memory, and is installed with a computer program for executing object recognition processing. Each function of the object recognition device 1 can be implemented by one or more processing circuits. The processing circuits include, for example, programmed processing devices including electrical circuits, and also include devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions described in the embodiments.

[0031] [Object recognition processing] Next, an object recognition process executed by the object recognition device 1 according to this embodiment will be described. 4A and 4B are1 is a flowchart showing the procedure of object recognition processing by the object recognition device 1 according to this embodiment. The object recognition processing starts when the vehicle 100 starts.

[0032] figure To 4A As shown, in step S10 at time t=k-1, the camera 5 captures an image of the surroundings of the vehicle and outputs the image at time t=k-1. The captured image is output to a storage device such as a memory of the object recognition device 1.

[0033] In step S11, the feature point extraction unit 15 extracts feature points from the image captured in step S10. For example, as shown in FIG.

[0034] At time t=k, in step S20, the clustering unit 11 acquires the ranging points detected by the object detection sensor 3, and generates clusters by performing a clustering process on the acquired ranging points. For example, as shown in Fig. 2, at time t=k, a cluster C1 is generated from the ranging point 21a.

[0035] In step S21, the camera 5 captures an image of the surroundings of the vehicle and outputs an image at time t=k.

[0036] In step S22, the processing area setting unit 13 acquires an image captured by the camera 5 at time t=k, associates the clusters generated in step S20 with the acquired image, and sets the area on the image to which the clusters correspond as a processing area. For example, as shown in Fig. 2, a processing area 25a is set in the image 23a at time t=k.

[0037] In step S23, the feature point extraction unit 15 extracts feature points from the image in the processing region set in step S22. For example, as shown in Fig. 2, feature point 27a is extracted from processing region 25a of image 23a.

[0038] In step S24, the moving speed calculation unit 17 matches the extracted feature points with feature points on the image of the previous frame by template matching, etc. For example, as shown in Fig. 2, the feature point 27a extracted from the image 23a at time t = k is matched with the feature point 27 extracted from the image 23 at time t = k-1.

[0039] In step S25, the moving speed calculation unit 17 calculates an optical flow from the coordinates of the feature points matched in step S24. For example, as shown in Fig. 2, an optical flow 29a is calculated from the coordinates of the matched feature points 27 and 27a.

[0040] In step S26, the moving speed calculation unit 17 averages the speed vectors of multiple optical flows 29a in the processing area 25a to calculate a speed vector (vx, vy). This speed vector is the moving speed of the feature point corresponding to the cluster C1 at time t=k.

[0041] Next, moving to time t=k+1 shown in Fig. 4B, in step S30, the clustering unit 11 acquires the ranging points detected by the object detection sensor 3, and generates clusters by performing clustering processing on the acquired ranging points. For example, as shown in Fig. 2, at time t=k+1, a cluster C10 is generated from the ranging point 21b.

[0042] In step S31, the camera 5 captures an image of the surroundings of the vehicle and outputs an image at time t=k+1.

[0043] In step S32, the processing area setting unit 13 acquires an image captured by the camera 5 at time t=k+1, associates the clusters generated in step S30 with the acquired image, and sets the area on the image to which the clusters correspond as a processing area. For example, as shown in Fig. 2, a processing area 25b is set in the image 23b at time t=k+1.

[0044] In step S33, the feature point extraction unit 15 extracts feature points from the image in the processing region set in step S32. For example, as shown in Fig. 2, feature point 27b is extracted from processing region 25b of image 23b.

[0045] In step S34, the moving speed calculation unit 17 matches the extracted feature points with feature points on the image of the previous frame by template matching, etc. For example, as shown in Fig. 2, the feature points 27b extracted from the image 23b at time t = k+1 are matched with the feature points 27a extracted from the image 23a at time t = k.

[0046] In step S35, the moving speed calculation unit 17 calculates an optical flow from the coordinates of the feature points matched in step S34. For example, as shown in Fig. 2, an optical flow 29b is calculated from the coordinates of the matched feature points 27a and 27b.

[0047] In step S36, the moving speed calculation unit 17 averages the speed vectors of the optical flows 29b in the processing area 25b to calculate a speed vector (vx, vy). This speed vector is the moving speed of the feature point corresponding to the cluster C10 at time t=k+1.

[0048] In step S37, the same object determination unit 19 moves the cluster from one frame before according to the moving speed calculated in step S26, and obtains the cluster after the movement. For example, as shown in Fig. 3, the feature points in the clusters C1, C2, and C3 at time t=k are moved by the time difference between t=k and t=k+1 according to the moving speed, and the clusters C1', C2', and C3' after the movement are obtained.

[0049] In step S38, the same object determination unit 19 determines whether the clusters at different times are clusters of the same object. First, the same object determination unit 19 sets combinations between the clusters at different times, and calculates the degree of match for each set combination. Specifically, it sets combinations between the clusters moved in step S37 and the clusters of the current frame, and calculates the degree of match for each combination. For example, as shown in FIG. 3, it sets combinations between the clusters C1', C2', and C3' after the movement and the clusters C10, C20, and C30 of the current frame, and calculates the degree of match for each combination.

[0050] The degree of match f can be calculated by the above formula (1). To specifically describe formula (1), the degree of match between cluster Ci and cluster Cj can be expressed by the following formula (2). f(Ci, Cj) = a(xi-xj) 2 +a(yi-yj) 2 +b(vxi-vxj) 2 +b(vyi-vyj) 2 (2)

[0051] For example, for the combination of clusters C1' and C10, the difference between the positions of the feature points corresponding to cluster C1' and those corresponding to cluster C10 is calculated, and the difference between the movement speeds of the feature points corresponding to cluster C1' and those corresponding to cluster C10 is calculated. Then, these differences are squared, multiplied by coefficients a and b, and added together to calculate the degree of match f.

[0052] Once the degree of matching is calculated in this way, the same object determination unit 19 then generates an evaluation matrix using the degree of matching calculated for each set combination. For example, as shown in Fig. 5, combinations between clusters C1', C2', and C3' after movement and clusters C10, C20, and C30 of the current frame are set, and an evaluation matrix is ​​generated using the degree of matching calculated for each combination. The numerical values ​​constituting the evaluation matrix in Fig. 5 are the degrees of matching. The smaller the degree of matching, the higher the possibility that the combined clusters are clusters of the same object.

[0053] Here, the same object determination unit 19 sets a sufficient threshold value, which is a degree of matching at which two combined clusters from different times are considered to be clusters of the same object. Therefore, a combination having a degree of matching that is less than the sufficient threshold value and satisfies the sufficient threshold value can be determined to be clusters of the same object.

[0054] In addition, the same object determination unit 19 sets a necessary threshold value that is at least the degree of matching required for two combined clusters taken at different times to be clusters of the same object. Therefore, a combination having a degree of matching that is equal to or greater than this necessary threshold value but does not satisfy the necessary threshold value can be determined to not be a cluster of the same object.

[0055] For example, in the case of FIG. 5, the sufficient threshold is set to 20, and the necessary threshold is set to 50. Therefore, as indicated by circles, the combination of C1' and C10 and the combination of C2' and C20 are less than the sufficient threshold, so these combinations are extracted and determined to be clusters of the same object. As a result, the clusters C10 and C20 in the current frame are determined to be the same object as the clusters C1 and C2 in the previous frame, and each inherits the IDs (identification numbers) of the clusters C1 and C2 in the previous frame. Therefore, the object recognition device 1 can track the clusters C10 and C2 in the current frame from the clusters C1 and C2 in the previous frame.

[0056] On the other hand, cluster C30 in the current frame cannot inherit the ID from the cluster in the previous frame because there is no combination that satisfies the required threshold value, so a new ID is assigned to cluster C30.

[0057] There may also be cases where the number of clusters in the current frame does not match the number of clusters in the previous frame. For example, as shown in Figure 6, there may be one cluster in the previous frame and three clusters in the current frame. In this case, only cluster C10 is sufficiently below the threshold, so it will inherit the ID of cluster C1 in the previous frame, and new IDs will be assigned to clusters C20 and C30.

[0058] Furthermore, in the clustering process, a cluster of the same object may be split into multiple clusters. For example, as shown in Fig. 7, at time t=k, one cluster C1 is generated for the vehicle, but at time t=k+1, two clusters C10 and C20 are generated for the vehicle because the vehicle is covered by a shadow of a building or the like.

[0059] In such a case, the combination of C1' and C10 and the combination of C1' and C20 are sufficiently below the threshold, as shown in Fig. 8. In this case, the clusters of C10 and C20 are merged into one cluster, and the ID of C1 is inherited.

[0060] There may also be a case where the number of clusters in the current frame is large. For example, as shown in Fig. 9, there may be three clusters in the previous frame and five clusters in the current frame. In this case, the combination of C1' and C10 and the combination of C1' and C20 are sufficiently below the threshold, so the IDs are determined. Therefore, the combination including C1', C10, and C20 is deleted.

[0061] Next, among the remaining combinations, the combination including C30 is equal to or greater than the necessary threshold, so a new ID is assigned. Then, the remaining combinations are less than the necessary threshold, so the combination with the smallest total degree of matching is selected. Specifically, the matching degree of 30 for the combination of C2' and C40 and the matching degree of 45 for the combination of C3' and C50 are added together to get 75. On the other hand, the matching degree of 40 for the combination of C3' and C40 and the matching degree of 40 for the combination of C2' and C50 are added together to get 80. Therefore, as shown by the circles, the combinations of C2' and C40 and C3' and C50 have smaller total degrees, so these combinations are extracted. As a result, cluster C40 will inherit the ID of cluster C2 from one frame before, and cluster C50 will inherit the ID of cluster C3 from one frame before.

[0062] In this manner, in step S38, the same object determination section 19 calculates the degree of coincidence, generates an evaluation matrix, and determines whether the clusters at different times are clusters of the same object.

[0063] In step S40, the same object determination unit 19 determines whether the power supply to the vehicle 100 has been turned off, and if the power supply to the vehicle 100 has not been turned off, the process returns to step S20 in Fig. 4A to continue the object recognition process. On the other hand, if the power supply to the vehicle 100 has been turned off, the object recognition process according to this embodiment is terminated.

[0064] [Variation 1] In the above-described embodiment, as shown in formula (1), the degree of match f is calculated based on the difference in positions of feature points corresponding to clusters and the difference in moving speed. However, the degree of match may be calculated using IoU (Intersection over Union).

[0065] IoU is an index that indicates the degree to which two regions overlap, and is expressed by the following formula (3). IoU = Area of ​​Intersection / Area of ​​Union (3) That is, when calculating the cluster coincidence f, the logical product of the areas of two processing regions set in clusters at different times is divided by the logical sum of the areas of two processing regions set in clusters at different times. The logical product of the areas is the area of ​​the part on the image where two clusters at different times exist in common, and the logical sum of the areas is the area of ​​the part on the image where at least one of the two clusters at different times exists.

[0066] Therefore, the degree of coincidence f may be calculated using the IoU according to the following formula (4). f = a × (difference in the positions of feature points corresponding to clusters) 2 + b × (difference in moving speed of feature points corresponding to clusters) 2 +c×(1-IoU) (4) where a, b, and c are coefficients.

[0067] That is, in the formula for the degree of agreement f, the third evaluation score is calculated using the value (IoU) obtained by dividing the logical product of the areas of two processing regions set in clusters at different times by the logical sum of the areas of the two processing regions.Then, the value obtained by multiplying this third evaluation score by the third coefficient c is added to the first evaluation score and the second evaluation score to calculate the degree of agreement f.

[0068] [Variation 2] In the above-described embodiment, the degree of coincidence is calculated to generate an evaluation matrix, and it is determined whether or not clusters at different times are the same object. However, it is also possible to make the determination using a simpler method.

[0069] Specifically, if the difference in the positions of two feature points corresponding to clusters at different times is less than a predetermined value, and the difference in the moving speeds of two feature points corresponding to clusters at different times is less than a predetermined value, it may be determined that the clusters at different times are clusters of the same object.

[0070] For example, the difference between the position of the feature point corresponding to cluster C1' after moving cluster C1 of the previous frame and the position of the feature point corresponding to cluster C10 of the current frame is calculated, and it is determined whether it is equal to or less than a preset position threshold. Similarly, the difference between the moving speed of the feature point corresponding to cluster C1' and the moving speed of the feature point corresponding to cluster C10 is calculated, and it is determined whether it is equal to or less than a preset speed threshold. Then, if both the difference in position and the difference in moving speed are equal to or less than a predetermined threshold, it is determined that clusters C1 and C10 at different times are clusters of the same object.

[0071] [Effects of the first embodiment] As described above in detail, in the object recognition device 1 according to this embodiment, the distance measurement points acquired from the object detection sensor 3 are clustered to generate clusters at a constant cycle, and the clusters are associated with the images captured by the camera 5 to set a processing area. Then, feature points are extracted from the images in the set processing area, the moving speeds of the feature points are calculated, and based on the difference in position and the difference in moving speed between two feature points corresponding to clusters at different times, it is determined whether the clusters at different times are clusters of the same object. As a result, in this embodiment, the moving speeds of the feature points are used to determine whether the clusters are clusters of the same object, so that objects around the vehicle can be accurately recognized as the same object at different times.

[0072] Conventionally, the moving speed of clusters is detected using a Kalman filter when detecting objects. However, even if the measurement points are on the same object, the locations clustered at different times may differ, making it difficult to accurately measure the moving speed of the clusters.

[0073] For example, as shown in Figure 10, at time t = 0, the front part of the vehicle is clustered, at time t = 1, the lower half of the vehicle is clustered, and at time t = 2, the rear part of the vehicle is clustered. In this way, different parts of the same vehicle are clustered at different times, so the clusters change over time. Therefore, since the clusters are not constant, it was difficult to accurately measure the moving speed.

[0074] In contrast, in this embodiment, the moving speed of the feature points corresponding to the clusters is used instead of the moving speed of the clusters. Since the moving speed of the feature points can be accurately calculated using optical flow, in this embodiment, the accurate moving speed can be used to accurately recognize objects around the vehicle as the same object at different times.

[0075] Furthermore, for example, assuming a case where an object is emerging from behind an object, it is possible to accurately recognize that the object is the same object before the entire shape of the object becomes capable of being measured by the object detection sensor 3, so that the object emerging from behind an object can be recognized as the same object at an early stage.

[0076] In addition, in the object recognition device 1 according to this embodiment, a first evaluation point is calculated based on the difference in the positions of two feature points corresponding to clusters at different times, and a second evaluation point is calculated based on the difference in the moving speeds of two feature points corresponding to clusters at different times. Then, a degree of agreement is calculated based on the first evaluation point and the second evaluation point, and it is determined whether the clusters at different times are clusters of the same object based on the degree of agreement. As a result, since the degree of agreement is calculated using the difference in the positions of the feature points and the difference in the moving speeds of the feature points, it is possible to accurately determine whether the clusters at different times are identical. Therefore, it is possible to accurately recognize objects around the vehicle as the same object at different times.

[0077] Furthermore, in the object recognition device 1 according to this embodiment, the first evaluation point is the square of the difference between the positions of two feature points corresponding to clusters at different times, and the second evaluation point is the square of the difference between the moving speeds of two feature points corresponding to clusters at different times. This allows the degree of match to be calculated using the square of the difference between the positions of the feature points and the difference between the moving speeds, so that it is possible to accurately determine whether the clusters at different times match. Therefore, it is possible to accurately recognize objects around the vehicle as the same object at different times.

[0078] Furthermore, in the object recognition device 1 according to this embodiment, a first coefficient to be multiplied by the first evaluation point and a second coefficient to be multiplied by the second evaluation point are set, and the degree of match is calculated by adding the value obtained by multiplying the first evaluation point by the first coefficient and the value obtained by multiplying the second evaluation point by the second coefficient. As a result, even if the units of the first evaluation point and the second evaluation point are different, the first and second evaluation points can be adjusted to be normalized by multiplying them by the first and second coefficients.

[0079] Furthermore, in the object recognition device 1 according to this embodiment, a third evaluation point is calculated using a value obtained by dividing the logical product of the areas of two processing regions set in clusters at different times by the logical sum of the areas of the two processing regions. Then, a value obtained by multiplying this third evaluation point by a third coefficient is further added to calculate the degree of match. This makes it possible to calculate a more accurate degree of match using IoU. Therefore, it is possible to more accurately recognize objects around a vehicle as being the same object at different times.

[0080] In addition, in the object recognition device 1 according to the present embodiment, combinations are set between clusters at different times, and the degree of coincidence is calculated for each combination to generate an evaluation matrix. Then, a degree of coincidence at which the two combined clusters at different times are considered to be clusters of the same object is set as a sufficient threshold, and combinations that satisfy the sufficient threshold are extracted from the evaluation matrix and determined to be clusters of the same object. In this way, since the evaluation matrix is ​​used, the best combination of clusters can be extracted regardless of the order of determining whether or not the sufficient threshold is satisfied. In particular, if it is determined in order for each combination whether or not the sufficient threshold is satisfied, a combination that is not the best may be extracted first, and the best combination may not be extracted. However, if the evaluation matrix is ​​used for the determination, the best combination can be reliably extracted.

[0081] Furthermore, in the object recognition device 1 according to the present embodiment, when the difference in the positions and the difference in the moving speeds of two feature points corresponding to clusters at different times are equal to or less than a predetermined value, the clusters at different times are determined to be clusters of the same object. This makes it possible to determine whether clusters are clusters of the same object without calculating the degree of coincidence, thereby significantly reducing the processing load.

[0082] [Second embodiment] A second embodiment of the present invention will now be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and detailed description will be omitted.

[0083] As shown in FIG. 11, the object recognition device 1 according to this embodiment is different from the first embodiment in that a stereo camera 51 is provided instead of the camera 5, and a self-position detection unit 53 and a map database 55 are further provided.

[0084] By providing the stereo camera 51, it becomes possible to capture a stereoscopic image. In the first embodiment, the two-dimensional position and moving speed on the image were obtained, but in this embodiment, the three-dimensional position and moving speed including the depth direction can be obtained.

[0085] The self-position detection unit 53 may acquire the current location of the vehicle 100 from a GPS receiver mounted on the vehicle 100, or may detect the self-position of the vehicle 100 by collating targets around the vehicle detected by the object detection sensor 3 with map information stored in the map database 55. Alternatively, the self-position detection unit 53 may calculate the moving direction and moving distance of the vehicle from a reference position (e.g., the position where the vehicle started traveling) based on the steering angle and traveling distance (odometry) of the vehicle, and detect the self-position of the vehicle 100 based on the reference position, the moving direction, and the moving distance.

[0086] Furthermore, the self-position detection unit 53 is connected to a group of sensors (not shown) mounted on the vehicle 100, and can acquire the behavior of the vehicle 100. For example, the self-position detection unit 53 is connected to an accelerator sensor, a steering sensor, a brake sensor, a vehicle speed sensor, an acceleration sensor, a wheel speed sensor, and the like, and acquires sensor values ​​output from these sensor groups.

[0087] The map database 55 is a database that stores map information, and the stored map information is highly accurate map information that includes at least information such as the positions of roads, the number of lanes, and the positions of sidewalks.

[0088] When performing clustering processing to generate clusters, the clustering unit 11 converts the coordinate system of the cluster from the vehicle coordinate system to the map coordinate system by using the self-position and map information detected by the self-position detection unit 53. The map coordinate system is a coordinate system in which a map is viewed from above, as shown in Fig. 12.

[0089] The processing area setting unit 13 matches the clusters generated by the clustering unit 11 to a three-dimensional image of the surroundings of the vehicle 100 captured by the stereo camera 51, and sets the area on the image to which the clusters correspond as the processing area.

[0090] The feature point extraction unit 15 extracts feature points from a three-dimensional image within the processing area set by the processing area setting unit 13. Since feature points are extracted from a three-dimensional image, the three-dimensional positions (coordinates) of the feature points are detected.

[0091] Since the three-dimensional coordinates of the feature point have been detected, the movement speed calculation unit 17 calculates the three-dimensional optical flow. Then, the movement speed calculation unit 17 calculates the three-dimensional movement speed of the feature point by averaging the velocity vector (vx, vy, vz) of the three-dimensional optical flow within the processing area. The movement speed calculation unit 17 may also calculate the three-dimensional position and movement speed of the feature point by matching the left and right images of the stereo camera 51 with the previous and next frames.

[0092] Furthermore, since the vehicle 100 is moving, the movement speed calculation unit 17 calculates the absolute speed by subtracting the behavior of the vehicle 100 during its movement from the calculated movement speed of the feature point. For example, the absolute speed of the feature point is calculated by subtracting the speed or yaw rate of the vehicle 100 from the movement speed of the feature point.

[0093] The same object determination unit 19 moves the cluster from one frame before according to the calculated absolute speed, and obtains the cluster after the movement. For example, as shown in Fig. 12, the feature points in cluster C1 at time t=k are moved by the time difference between t=k and t=k+1 according to the absolute speed, and a cluster C1' after the movement is obtained.

[0094] Then, the same object determination unit 19 sets a combination between the cluster C1' after the movement shown in FIG. 12 and the cluster C10 of the current frame, and calculates the degree of match for this combination.

[0095] The degree of match f can be expressed in three dimensions as shown in the following equation (5). f(Ci, Cj) = a(xi-xj) 2 +a(yi-yj) 2 +a(zi-zj) 2 +b(vxi-vxj) 2 +b(vyi-vyj) 2 +b(vzi-vzj) 2 (5)

[0096] Once the degree of coincidence f has been calculated in this manner, the same object determination section 19 determines whether or not the clusters at different times are clusters of the same object, using the same method as in the first embodiment.

[0097] [Variations] When performing clustering processing to generate clusters, the clustering unit 11 may determine whether the generated cluster corresponds to an object on the roadway or to an object on the sidewalk. After generating a cluster, the clustering unit 11 determines whether the cluster corresponds to an object on the sidewalk or to an object on the roadway, using the self-position detected by the self-position detection unit 53 and the positions of the roadway and the sidewalk included in the map information.

[0098] Then, when it is determined that the cluster corresponds to an object on the sidewalk, the cluster is considered to correspond to a pedestrian, so the position is more important than the speed. Therefore, the same object determination unit 19 sets the coefficient a multiplied by the position difference in the formula of the degree of coincidence f with more importance than the coefficient b. Specifically, the smaller the value of the degree of coincidence f, the higher the possibility of determining that the clusters are of the same object, so the coefficient a is set smaller than the coefficient b.

[0099] On the other hand, if it is determined that the cluster corresponds to an object on the roadway, the cluster is considered to correspond to a vehicle, so the speed is more important than the position. Therefore, the same object determination unit 19 sets the coefficient b multiplied by the difference in moving speed in the formula of the degree of agreement f with more importance than the coefficient a. Specifically, the smaller the value of the degree of agreement f, the higher the possibility of being determined as a cluster of the same object, so the coefficient b is set smaller than the coefficient a.

[0100] [Effects of the second embodiment] As described above in detail, the object recognition device 1 according to this embodiment obtains the three-dimensional positions of feature points and calculates the three-dimensional movement speeds of the feature points using the stereo camera 51. This makes it possible to determine whether or not a cluster is of the same object using the three-dimensional positions and movement speeds of the feature points, and therefore makes it possible to more accurately recognize objects around the vehicle as the same object at different points in time.

[0101] In addition, the object recognition device 1 according to the present embodiment determines whether a cluster corresponds to an object on a sidewalk or to an object on the roadway, based on the detected vehicle's own position and map information storing at least the roadway position and the sidewalk position. If the cluster corresponds to an object on the sidewalk, the first coefficient a is set with more importance than the second coefficient b, and if the cluster corresponds to an object on the roadway, the second coefficient b is set with more importance than the first coefficient a. As a result, if the cluster corresponds to an object on the sidewalk, it is considered that the cluster corresponds to a pedestrian, so by placing importance on the first coefficient a, it is possible to calculate the degree of match by placing importance on the difference in position. On the other hand, if the cluster corresponds to an object on the roadway, it is considered that the cluster corresponds to a vehicle, so by placing importance on the second coefficient b, it is possible to calculate the degree of match by placing importance on the difference in moving speed. Therefore, it is possible to accurately calculate the degree of match according to the object to which the cluster corresponds.

[0102] The above-described embodiment is merely an example of the present invention, and the present invention is not limited to the above-described embodiment, and various modifications can be made to the design and other aspects of the present invention without departing from the technical concept of the present invention. [Explanation of symbols]

[0103] 1 Object recognition device 3. Object detection sensor 5. Camera 11 Clustering Department 13 Processing area setting section 15 Feature point extraction unit 17 Movement speed calculation section 19 Same object determination unit 51 Stereo Camera 53 Self-location detection unit 55 Map Database 100 vehicles

Claims

1. An object recognition method for recognizing objects present around a vehicle equipped with an object detection sensor and an imaging means, comprising: acquiring distance measurement points obtained by measuring distances to each point on an object existing around the vehicle using the object detection sensor, and clustering the acquired distance measurement points to generate clusters at a constant cycle; acquiring an image of the surroundings of the vehicle captured by the imaging means, matching the cluster to the acquired image, and setting an area on the image to which the cluster corresponds as a processing area; Extracting feature points from the image within the set processing area; Calculating the moving speed of the extracted feature points; An object recognition method that determines whether clusters at different times are clusters of the same object based on the difference in position of two feature points corresponding to the clusters at different times and the difference in movement speed of two feature points corresponding to the clusters at different times.

2. 2. The object recognition method according to claim 1, further comprising the steps of: calculating a first evaluation point based on a difference in the positions of two feature points corresponding to the clusters at the different times; calculating a second evaluation point based on a difference in the moving speeds of the two feature points corresponding to the clusters at the different times; calculating a degree of match indicating a degree of match between the clusters at the different times based on the first evaluation point and the second evaluation point; and determining whether the clusters at the different times are clusters of the same object based on the degree of match.

3. 3. The object recognition method according to claim 2, wherein the first evaluation point is a square of a difference in positions of two feature points corresponding to the clusters at different times, and the second evaluation point is a square of a difference in moving speeds of two feature points corresponding to the clusters at different times.

4. The object recognition method according to claim 3, further comprising the steps of: setting a first coefficient to be multiplied by the first evaluation point; and setting a second coefficient to be multiplied by the second evaluation point; and calculating the degree of match by adding together a value obtained by multiplying the first evaluation point by the first coefficient and a value obtained by multiplying the second evaluation point by the second coefficient.

5. The object recognition method according to claim 4, further comprising the steps of: calculating a third evaluation point using a value obtained by dividing the logical product of the areas of two processing areas set in the clusters at different times by the logical sum of the areas of the two processing areas; and further adding a value obtained by multiplying the third evaluation point by a third coefficient to calculate the degree of similarity.

6. Combinations are set between the clusters at different times, and the degree of agreement is calculated for each combination set to generate an evaluation matrix; A sufficient threshold is set to a degree of similarity at which the two combined clusters from different points in time are considered to be clusters of the same object; The object recognition method according to any one of claims 2 to 5, further comprising extracting combinations that satisfy the sufficient threshold value from the evaluation matrix and determining that the combinations are clusters of the same object.

7. 2. The object recognition method according to claim 1, further comprising the step of determining that the clusters at different times are clusters of the same object if a difference in the positions of two feature points corresponding to the clusters at different times is equal to or less than a predetermined value and a difference in the moving speeds of two feature points corresponding to the clusters at different times is equal to or less than a predetermined value.

8. the imaging means is a stereo camera, The object recognition method according to any one of claims 1 to 7, further comprising obtaining a three-dimensional position of the feature point and calculating a three-dimensional moving speed of the feature point.

9. Detecting a vehicle's own position; determining whether the cluster corresponds to an object on a sidewalk or a roadway based on the detected self-location and map information storing at least the location of the roadway and the location of the sidewalk; if the cluster corresponds to an object on a sidewalk, the first coefficient is set to be more important than the second coefficient; 5. The object recognition method according to claim 4, wherein when the cluster corresponds to an object on a roadway, the second coefficient is set with more importance than the first coefficient.

10. An object recognition device including a controller that recognizes objects present around a vehicle equipped with an object detection sensor and an imaging means, The controller: acquiring distance measurement points obtained by measuring distances to each point on an object existing around the vehicle using the object detection sensor, and clustering the acquired distance measurement points to generate clusters at a constant cycle; acquiring an image of the surroundings of the vehicle captured by the imaging means, matching the cluster to the acquired image, and setting an area on the image to which the cluster corresponds as a processing area; Extracting feature points from the image within the set processing area; Calculating the moving speed of the extracted feature points; An object recognition device that determines whether clusters at different times are clusters of the same object based on the difference in position of two feature points corresponding to the clusters at different times and the difference in movement speed of two feature points corresponding to the clusters at different times.

Images