Object recognition device

Abstract

Object detection device (1, 2, 3), with: a LIDAR (LD, 20) configured to emit a laser beam around a vehicle (M1) and to detect a laser beam reflected in an environment of the vehicle (M1); a processing unit (PU, 30, 40, 70) configured to perform clustering processing of a detected point group from the LIDAR (LD, 20) based on 3D position data of the corresponding detected point, including time-of-flight data of the laser beam; and a color image acquisition section (10) configured to acquire a color image (CIMG) of the environment, where in clustering processing the processing unit (PU, 30, 40, 70) is configured to select a first acquisition point corresponding to a first coordinate (CD). DP1 (x, y)) with a second detection point corresponding to a second coordinate (CD DP2(x, y)) to join if a joinable condition exists in the first and second coordinates (CD DP1 (x, y), CD DP2 (x, y)) on a distance image (DIMG) represented by the 3D position data, where the connectable condition is satisfied if there is a distance (dC) between the first and second coordinates (CD). DP1 (x, y), CD DB2 (x, y)) greater than or equal to a predetermined distance (dC) TH ) is a difference (dT) in the flight time data between the first and second recording points that is smaller than a predetermined difference (dT). TH ) is, and a condition for a coordinate is met, where the condition for a coordinate is satisfied if a route coordinate (CD) L (x, y)), which specifies a coordinate that lies on a route between the first and second coordinate (CD DP1 (x, y), CD DP2 (x, y)) is located, an unrecorded coordinate (CDBR (x, y)) is where the flight time data was not calculated, and a color pixel value that specifies a pixel value of each coordinate that defines the route coordinate (CD). L (x, y)) overlaps, a preset pixel value of the black type.

Description

[0001] The present disclosure relates to an object recognition device installed in a vehicle. background

[0002] JP 2012-145 444 A discloses an object detection device in which clustering processing is performed using a data point group acquired by a LiDAR (Laser Imaging Detection and Ranging) system. The data point group consists of 3D positional data (i.e., X, Y, and Z positional data) according to a number of acquired points. The clustering processing involves comparing the 3D data between the acquired points, thereby grouping acquired points that can be recognized as a single object. List of state of the art

[0003] The following is a list of patent literature that the applicant has identified as prior art of the present invention. JP 2012 - 145 444 A JP 2018 - 048 949 A

[0004] The publication EP 3 396 408 A1 discloses a data fusion system that fuses LiDAR and camera data for an automated vehicle. The system comprises a camera, a LiDAR, and a controller. The camera captures an image of an object near a host vehicle. The LiDAR detects the distance and direction to the object based on a reflected light signal. The controller communicates with the camera and the LiDAR. The controller is configured to determine the object's reflection characteristics based on the image and the reflected signal, and to adjust the LiDAR's detection characteristics if the object's reflection characteristics make it difficult for the LiDAR to determine the distance and direction to the object.

[0005] The publication DARAEI, MH [et al.]: Region segmentation using LiDAR and camera. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), 2017, pp. 1-6; discloses a sensor fusion-based segmentation approach that generates dense depth regions referred to as supersurfaces. This method aggregates both a point cloud from a LiDAR and a camera image to perform a supersegmentation of the three-dimensional scene into piecewise planar surfaces using a multi-label Markov random field (MRF). Summary

[0006] The LIDAR uses the time of flight (TOF) of the laser beam to generate the Z-position data (i.e., the TOF data). Therefore, if the time of flight is not measured, the Z-position data is not obtained. If the Z-position data is not obtained, an empty area appears in a data point group. In this case, the empty area is not subject to clustering processing.

[0007] Two reasons are assumed if the flight time is not measured. One is that there is no object to reflect the emitted light of the laser beam. The other is that there was a weakly reflective object that absorbed the emitted light (for example, a black object). Therefore, in the latter case, the clustering processing is not performed correctly, and there is a probability that the detection of the weakly reflective object is performed incorrectly.

[0008] One objective of the present disclosure is to provide an object detection device capable of detecting the weakly reflective object that absorbs the laser beam emitted by the LIDAR.

[0009] A first aspect is an object recognition device that solves the above problem and has the following features.

[0010] The object detection device includes a LIDAR and a processing unit.

[0011] The LIDAR is configured to emit a laser beam around a vehicle and to detect a laser beam that is reflected in the vehicle's surroundings.

[0012] The processing unit is configured to perform clustering processing of the group of captured LIDAR points based on 3D position data of the corresponding captured point, including time-of-flight data of the laser beam.

[0013] The object recognition device also includes a color image acquisition section.

[0014] The color image acquisition section is configured to acquire a color image of the environment.

[0015] In clustering processing, the processing unit is configured to link a first acquisition point corresponding to a first coordinate with a second acquisition point corresponding to a second coordinate if a connectable condition or linking condition is met in the first and second coordinates with respect to a distance image represented by the 3D position data.

[0016] The connectable condition is met if a distance between the first and second coordinates is greater than or equal to a predetermined distance, a difference in the flight time data between the first and second acquisition points is less than a predetermined difference, and a condition for one coordinate is met.

[0017] The condition for a coordinate is met if a route coordinate, which specifies a coordinate that lies on a route between the first and second coordinates, is an uncaptured coordinate where the flight time data has not been calculated, and a color pixel value, which specifies a pixel value from each coordinate that overlaps the route coordinate, is a preset pixel value of the black type.

[0018] A second aspect continues to include the following features from the first aspect.

[0019] The processing unit is still configured to perform interpolation processing before executing clustering processing.

[0020] Interpolation processing is a process for interpolating the flight time data into the uncaptured coordinate.

[0021] In interpolation processing, the processing unit takes data for interpolation as the flight time data of an uncaptured coordinate if an applicable condition is met.

[0022] The applicable condition is met if the color pixel value of the uncaptured coordinate, after interpolation, is the preset pixel value of the black type.

[0023] A third aspect further includes the following features from the second aspect.

[0024] The data for interpolation are the flight time data at a coordinate with the shortest distance from the uncaptured coordinate in a negative direction on a vertical axis of the distance image.

[0025] A fourth aspect further includes the following characteristics from the second aspect.

[0026] The object recognition system also includes a position information acquisition section and a database.

[0027] The position information retrieval section is configured to retrieve position information about the vehicle.

[0028] The database is configured to store terrain information.

[0029] In interpolation processing, the processing unit is further configured to specify a road surface coordinate, which indicates a coordinate corresponding to a road surface, from coordinates of the distance image, based on the position information and the terrain information.

[0030] The data for interpolation are the flight time data at the road surface coordinate with the shortest distance from the uncaptured coordinate in a negative direction in a vertical axis of the distance image.

[0031] According to the first aspect, the first and second acquisition points are linked if the linkable condition is met. This linkable condition allows the first and second acquisition points to be linked even if there is an unacquired coordinate between them. This simplifies grouping these acquired points into the same cluster during clustering processing. Therefore, it is possible to suppress misidentification of the weakly reflecting object.

[0032] According to the second aspect, interpolation processing is performed before clustering processing. In interpolation, if the applicable condition is met, the data for interpolation is used as the flight time data in the unaccounted coordinate. Therefore, it is possible to reduce the computational load for clustering processing.

[0033] According to the third aspect, the flight time data at the coordinate with the shortest distance from the untracked coordinate in the negative direction on the vertical axis of the distance image are used as the data for interpolation. Therefore, it is possible to improve the accuracy of the clustering processing.

[0034] According to the fourth aspect, the flight time data at the road surface coordinate with the shortest distance from the uncaptured coordinate in a negative direction on a vertical axis of the distance image are used as the data for interpolation. Therefore, it is possible to improve the accuracy of the clustering processing. Brief description of the drawings Fig. Figure 1 is a diagram to explain the acceptance of the present application; Fig. Figure 2 is a schematic diagram showing an image of the rear section of a vehicle traveling ahead, whose body color is the white type; Fig. Figure 3 is a schematic diagram showing an image of the rear section of the vehicle ahead, whose body is the black type; Fig. Figure 4 is a diagram to represent a condition for a coordinate; Fig. Figure 5 is a diagram to represent the condition for a coordinate; Fig. Figure 6 is a block diagram showing a configuration example of an object detection device according to a first embodiment; Fig. Figure 7 is a flowchart to illustrate an example of processing that is performed in a processing unit; Fig. Figure 8 is a diagram to explain a second example of interpolation processing according to a second embodiment; Fig. Figure 9 is a block diagram showing the configuration example of the object recognition device according to the second embodiment; Fig. Figure 10 is a flowchart illustrating the example of the processing that is performed in the processing unit; Fig. Figure 11 is a diagram to explain the interpolation processing according to a third embodiment; and Fig. Figure 12 is a block diagram to show the configuration example of the object recognition device according to the third embodiment. Description of the exemplary implementations

[0035] Embodiments of the present disclosure are described below with reference to the attached drawings. It should be understood, however, that even if the number, quantity, amount, range, or other numerical characteristic of an element is mentioned in the following description of the embodiments, the present disclosure is not limited to the mentioned numerical property unless expressly stated or theoretically defined. Furthermore, structures, steps, or the like described in connection with the following embodiments are not necessarily essential to the embodiments of the present disclosure unless expressly stated or theoretically defined. 1. First embodiment

[0036] First, a first embodiment of the present disclosure is described with reference to Fig. 1 to 7 described. 1.1 Assumption

[0037] Fig. Figure 1 is a diagram illustrating an assumption of the present disclosure. A vehicle M1 is in Fig. Figure 1 shows a LiDAR LD mounted on vehicle M1. The LiDAR LD emits a laser beam around vehicle M1 and detects a laser beam reflected from its surroundings. When the laser beam is detected by the LiDAR LD, a data point group is obtained. This data point group consists of 3D position data (i.e., X, Y, and Z coordinates) based on a number of detected points. A time reference for the 3D position data and position data from the LiDAR LD relative to this time reference (for example, latitude and longitude data) can be added to the data point group.

[0038] The vehicle M1 includes a processing unit (PU). The processing unit PU performs object recognition processing. This object recognition processing includes clustering processing. Clustering processing is a process for grouping the data point group into a plurality of clusters based on the 3D position data. The present disclosure is characterized in that the clustering processing is characterized by the execution of a process for linking at least two elements of data points. There is no specific limitation regarding the object recognition processing based on the clusters obtained by executing the clustering processing, and a known technique is employed.

[0039] In Fig. Figure 1 represents a road surface RS on which the vehicle M1 is driving. When the laser beam reflected by the road surface RS is detected by the LIDAR LD, the 3D position data is obtained according to the number of points detected on the road surface RS. Fig. Figure 1 shows a vehicle M2 in front of vehicle M1. Vehicle M2 can be moving or stationary. Vehicle M2 can be a moving body other than a vehicle, or it can be a stationary body. When the laser beam reflected by vehicle M2 is detected by the LIDAR LD, the 3D position data is obtained according to the number of points detected on vehicle M2. Each of the data points DP that are in Fig. Figure 1 shows a schematic representation of the recorded points of the road surface RS and the vehicle M2. 1.2 Problems with clustering processing

[0040] As described above, if the time of flight was not measured by the LIDAR LD, two reasons are assumed: one is that there was no object to reflect the emitted light of the laser beam; the other is that there was a weakly reflective object that absorbed the emitted light. Problems in the latter cases are addressed with reference to Fig. 2 and Fig. 3 described.

[0041] These figures schematically depict two types of images: a color CIMG image and a distance DIMG image of a rear section RE. M2of the vehicle M2. The CIMG color image is an RGB image acquired from a camera or image sensor. In the RGB image, a pixel value of a 2D position coordinate is represented by an RGB value. That is, a pixel value IC(x, y) of a coordinate CC(x, y) in the CIMG color image is expressed by the RGB value. It should be noted that the pixel value IC(x, y) corresponds to a "color pixel value" in this disclosure. The CIMG color image can be represented using a CMY value, a CMYK value, an HSV value, or an HLS value.

[0042] The distance image DIMG is an image represented by 3D position data. Each coordinate of the distance image DIMG is represented by the pixel value according to the Z-position data (i.e., TOF data) at the data point DP. Specifically, a pixel value ID(x, y) of a coordinate CD(x, y) in the distance image DIMG is represented by the pixel value according to the TOF. For example, the pixel value ID(x, y) is represented by a smaller pixel value when the TOF is shorter, while it is represented by a larger pixel value when the TOF is longer.

[0043] Fig. 2 shows the image of the rear section RE M2 If the body color of the vehicle M2 is white, as shown in the DIMG distance image, the laser beam will be focused on a surface of the rear section RE. M2Simply reflected. Therefore, a number of data points (DPs) can correspond to the captured points of the rear section (RE). M2 Therefore, in this situation, the data points (DPs) are simply grouped into the same cluster during clustering processing.

[0044] Fig. Figure 3 shows an image of the rear section RE M2 If the body color of vehicle M2 is black, as shown in the distance image DIMG, the data points DPs can only be located in a portion of the rear section RE. M2 Features such as a taillight (TL) and a bumper (RB) will be retained. This is because the laser beam passes through the rear section (RE). M2 is not reflected and the TOF was not measured.

[0045] In Fig. Three areas (hereinafter referred to as "unrecorded areas") BR1 to BR3 are shown, in which no Z-position data was obtained. The coordinate CD (x, y), which consists of the unrecorded areas BR, is referred to as an "unrecorded coordinate CD". BR (x, y)” is then assigned to each of the pixel values. BR (x, y) of the uncharted coordinates CD BR (x, y) represented by the RGB value of the black type. The recording of such uncaptured coordinates CD BR (x, y) in the distance image DIMG means that there is an empty area in the data point group. If the empty area is in the data point group, it separates the data points DPs according to the captured points of the rear section RE. M2 This makes it difficult to group the data points (DPs) into the same cluster. 1.3 Features of object recognition processing in the first embodiment

[0046] Therefore, if there is a weakly reflective object that absorbs the laser beam, the clustering processing could not be performed correctly, and this object could not be correctly detected. For this reason, in the clustering processing according to the first embodiment, a basic condition and a special condition are set as conditions for linking any two data points (DPs) in the distance image (DIMG) (connectable condition). For the sake of simplicity, the two data points (DPs) are referred to as "first data point DP1" and "second data point DP2" in the following description. 1.3.1 Basic condition

[0047] The basic condition is a condition regularly used to link the first data point DP1 with the second data point DP2. The basic condition consists of the following expressions (i) and (ii) for a first coordinate CD. DP1 (x, y) and a second coordinate CD DP2 (x, y). It should be noted that the first coordinate is CD. DP1 (x, y) is a coordinate CD(x, y) corresponding to the first data point DP1. The second coordinate CD DP2 (x, y) is the coordinate CD(x, y) corresponding to the second data point DP2. (i) A distance dC between the first coordinate CD DP1 (x, y) and the second coordinate CD DP2 (x, y) is less than a predetermined distance dC TH . (ii) A difference dT between the TOF data TD DP1 (x, y) and the TOF data TD DP2 (x, y) is less than a predetermined difference dT TH .

[0048] The predetermined distance dCTH is a threshold value that is preset based on a typical vehicle width (for example, approximately 2 m). The predetermined difference dT TH This is the threshold at which two TOFs are recognized as comparable. It should be noted that the TOF data TD DP1 (x, y) are the Z-position data in the first data point DP1. The TOF data TD DP2 (x, y) are the Z-position data in the second data point DP2. If expressions (i) and (ii) are satisfied simultaneously, then the basic condition is satisfied. 1.3.2 Special condition

[0049] The special condition is used to join the second data point DP2 and the first data point DP1, which could not be joined according to the basic condition. The special condition consists of the following expressions (iii) to (v) for the first coordinate CD. DP1 (x, y) and the second coordinate CD DP2 (x, y). (iii) The distance dC is equal to or greater than the predetermined distance dC TH . (iv) The difference dT is smaller than the predetermined difference tT TH . (v) A condition for a coordinate is satisfied.

[0050] If expressions (iii), (iv) and (v) are satisfied simultaneously, then the special condition is satisfied. Note that condition (iv) is the same as condition (ii).

[0051] The condition for the coordinate shown in expression (v) consists of the following expressions (v1) and (v2). (v1) A route coordinate CD L (x, y) corresponds to the unrecorded coordinate CD BR (x, y) (v2) When an alignment of the distance image DIMG and the color image CIMG is performed, all pixel values ​​ICs(x, y) correspond to the coordinate CC(x, y) that corresponds to the coordinate CD L(x, y) overlap, the predetermined pixel value of the black type.

[0052] The route coordinate CD L (x, y) is the coordinate CD(x, y) that lies on a path between the first coordinate CD DP1 (x, y) and the second coordinate CD DP2 (x, y). The number of route coordinates CD L(x, y) is at least 1. The "preset pixel value of black type" includes the RGB value of black type. The "RGB value of black type" refers to the RGB value where each value of the RGB value is less than the preset value (for example, each value is approximately 50 when the RGB value is represented in 256 gradations). If expressions (v1) and (v2) are satisfied simultaneously, then the condition for the coordinate is satisfied. If the CIMG color image is represented by a color system value different from the RGB value (for example, the CMYK value), then the "preset pixel value of black type" refers to the value of black type in that color system.

[0053] Fig. Figure 4 is a diagram to represent condition (v1). In Fig. 4 Some areas in the distance image DIMG are shown schematically. In the Fig. In the example shown, for simplicity, the coordinates CD(x, y) are represented by white, gray, and black. The coordinates CDs(x, y) expressed by white and gray (for example, coordinates CDs (i+2, j+1) and (i+2, j+4)) correspond to the data points DPs. The coordinates CDs(x, y) expressed by black (for example, coordinates CDs (i+2, j+2) and (i+6, j+2)) correspond to the unaccounted coordinate CD. BR (x, y).

[0054] The route coordinate CD L (x, y) is based on the shortest route SL between the first coordinate CD DP1 (x, y) and the second coordinate CD DP2 (x, y) is specified. Let the first coordinate be CD. DP1 (x, y) is the coordinate CD(i, j), and the second coordinate CD DP2 (x, y) the coordinate CD(i+2, j). In this case, the shortest route SL is specified as the path L1, which is in Fig. 4 is shown. Then the route coordinate is CD.L (x, y) the coordinate CD(i+1, j). Here, the coordinate CD(i+1, j) corresponds to the unmeasured coordinate CD. BR (x, y). Therefore, according to a combination of the coordinate CD(i, j) and the coordinate CD(i+2, j), expression (v1) is satisfied.

[0055] There could be more than one shortest route SL. In this case, if any of these satisfies expression (v1), then expression (v1) is satisfied. Let the first coordinate be CD. DP1 (x, y) is the coordinate CD(i+2, j+1) and the second coordinate CD DP2 (x, y) the coordinate CD(i+1, j+4). In this case, the shortest route SL is the path L2 or L3, which is in Fig. Figure 4 shows that the path L3, which contains the coordinate CD(i+2, j+4), does not satisfy v1. On the other hand, the coordinates (i+1, j+1), (i+1, j+2) and (i+1, j+3) correspond to the unaccounted-for coordinates CD. BR(x, y). Therefore, according to path L2, expression (v1) is satisfied. Therefore, according to the combination of coordinate CD(i+2, j+1) and coordinate CD(i+1, j+4), expression (v1) is satisfied.

[0056] Fig. Figure 5 is a diagram illustrating expression (v2). In Fig. 5 are the distance image DIMG and the color image CIMG, which are related to Fig. 3 are described, illustrated. Fig. Figure 5 shows a projected image PIMG, which projects the color image CIMG onto the distance image DIMG after alignment. As shown in this projected image PIMG, a slice of the rear section RE is visible. M2 Projected onto an unscanned area BR1. Here, the pixel value IC(x, y) of the disc does not correspond to the preset pixel value of the black type. Therefore, the pixel value IC(x, y) of the coordinate CC(x, y) is different from the coordinate CD. L(x, y) overlaps, the RGB value that is different from the black type. Consequently, expression (v2) is not satisfied.

[0057] A rear light TLs of the rear section RE is applied to an unmonitored area BR2. M2 and a painted surface of the car body is projected. Here, the pixel value IC(x, y) of the taillight TL does not correspond to the preset pixel value of the black type. Therefore, the pixel value IC(x, y) satisfies the coordinate CC(x, y), which corresponds to the coordinate CD. L (x, y) overlaps according to the taillight TL, not expression (v2). On the other hand, the pixel value IC(x, y) of the painted surface of the body corresponds to the preset pixel value of the black type. Therefore, according to the pixel value IC(x, y) of the coordinate CC(x, y), which is the coordinate CD L If (x, y) overlaps according to the painted surface of the body, expression (v2) is satisfied.

[0058] The painted surface of the car body is projected onto the unscanned area BR3. Therefore, the same conclusion is reached as for the painted surface of the car body projected onto the unscanned area BR2. 1.4 Effect

[0059] According to the characteristics described above, clustering processing is performed based on the combination of the basic condition and the special condition. Specifically, according to the special condition, even if there is an uncaptured coordinate CD, the processing is carried out. BR Since there is a relationship (x, y) between the first data point DP1 and the second data point DP2, it is possible to link data points DP1 and DP2. Therefore, in clustering processing, it is possible to simply group data points DP1 and DP2 into the same cluster. This also makes it possible to suppress misidentification of the weakly reflecting object. 1.5 Configuration examples of the object detection device

[0060] Fig. Figure 6 is a diagram showing a configuration example of an object detection device 1 according to the first embodiment. As in Fig. As shown in Figure 6, the object recognition device 1 comprises a color image acquisition section 10, a distance image acquisition section 20 and a processing unit 30.

[0061] Color image acquisition section 10 acquires the CIMG color image. Color image acquisition section 10 can be, for example, a camera or an image data sensor. Color image acquisition section 10 can also be an image processing unit separate from the camera or image data sensor. Color image acquisition section 10 transmits the CIMG color image to processing unit 30.

[0062] The range image acquisition section 20 acquires the DIMG range image. The range image acquisition section 20 is, for example, the LIDAR LD, which is in Fig. Figure 1 shows the distance image acquisition section 20, which can be an image processing unit separate from the LIDAR LD. The distance image acquisition section 20 transmits the distance image DIMG to the processing unit 30.

[0063] The processing unit 30 is a processing unit PU, which is located in Fig. Figure 1 shows the following. Typically, the processing unit 30 is a microcomputer comprising a processor, memory, and an input / output interface. The processing unit 30 receives various pieces of information via the input / output interface. The processing unit 30 then performs the object recognition processing based on the received data.

[0064] The processing unit 30 comprises an alignment section 31, a section 32 for identifying a specific coordinate, and a clustering section 33 as functions for performing the recognition processing. These function blocks, which are in Fig. The 6 shown are realized when the processor of the processing unit executes 30 control programs that are stored in the memory.

[0065] Alignment section 31 performs an alignment of the CIMG color image and the DIMG distance image. The alignment is carried out with high accuracy using, for example, the following information (i) to (iii). (i) Calibration parameters of the camera (or image sensor) and the LiDAR LD (ii) Procurement time t CIMG of the color image CIMG and procurement time t DIMG of the distance image DIMG (iii) Position of the camera (or image sensor) at the time of procurement t CIMG and position of the LIDAR LD at the procurement time t DIMG

[0066] Section 32, for the identification of a special coordinate, projects the color image CIMG onto the distance image DIMG after alignment. Section 32, for the identification of a special coordinate, identifies a special coordinate CP(x, y) on the distance image DIMG onto which the color image CIMG was projected (i.e., the projected images PIMG). The special coordinate CP(x, y) is a coordinate CP(x, y) on the projected image PIMG that satisfies the following expressions (i) and (ii). (i) The coordinate CP(x, y) corresponds to the unaccounted coordinate CD BR on the distance image DIMG (ii) The pixel value IC(x, y) is the preset pixel value of the black type on the color image CIMG.

[0067] Section 32, for the identification of the special coordinate, transmits the special coordinate CP. PE (x, y) to clustering section 33 if the special coordinate CP PE (x, y) is specified.

[0068] Clustering section 33 performs the clustering processing. This processing is responsible for linking at least two data points (DPs). During the linking process, it is determined whether the basic condition or the special condition is satisfied with respect to the first data point (DP1) and the second data point (DP2) on the distance image (DIMG). It should be noted that the determination of the condition for the coordinate contained in the special condition is based on the special coordinate (CP(x, y)) and the route coordinate (CD). L (x, y) is executed. If the special coordinate CP(x, y) of the route coordinate CD LIf (x, y) equals, the condition for the coordinate is satisfied. If the basic condition or the special condition is satisfied, the first data point DP1 and the second data point DP2 are joined. 1.6 Specific processing

[0069] Fig. Figure 7 is a flowchart illustrating an example of processing performed by processing unit 30. The routine shown in this diagram is executed repeatedly for each predetermined control cycle.

[0070] First, in step S11, the CIMG color image is acquired. Next, in step S12, the DIMG distance image is acquired. Finally, in step S13, the CIMG color image is projected onto the DIMG distance image. The alignment of the CIMG color image and the DIMG distance image is performed before the CIMG color image is projected.

[0071] Step S14 determines whether the basic condition is met. Step S15 determines whether the specific condition is met. If the assessment result of step S14 or S15 is positive, these data points are linked in step S16. If both assessment results of steps S14 and S15 are negative, these data points are not linked in step S17. All data points (DPs) on the distance image (DIMG) are checked against the basic or specific condition. This means that the assessment process, including steps S14 to S17, is repeated until assessment results are obtained for all data points (DPs). 2. Second embodiment

[0072] Next, a second embodiment will be described with reference to Fig. Sections 8 to 10 are described. It should be noted that descriptions that overlap with those of the first embodiment are omitted where necessary. 2.1 Features of object recognition processing according to the second embodiment

[0073] The object recognition processing of the second embodiment includes interpolation processing. This interpolation processing is a process for interpolating uncaptured Z-position data into the uncaptured coordinate CD. BR (x, y). Interpolation processing is performed before clustering processing. In interpolation processing, based on an applicable condition or application condition, it is determined whether the distance image DIMG, in which the uncaptured Z-position data is located in the uncaptured coordinate CD, should be used. BR (x, y) were interpolated, to be adopted or not.

[0074] For simplicity, the unrecorded Z-position data is stored in the unrecorded coordinate CD. BR (x, y) are referred to as "unrecorded data" in the following description. Furthermore, the Z-position data used for interpolating the unrecorded data are referred to as "data for interpolation." Furthermore, the distance image DIMG, expressed by the 3D position data in which the unrecorded data are interpolated, is referred to as an "interpolated image IIMG." Furthermore, the unrecorded coordinate CD BR (x, y) after interpolation of the unrecorded data as an “interpolated coordinate CD” IN (x, y)” is denoted. Furthermore, the data point DP is defined according to the interpolated coordinate CD. IN (x, y) is referred to as an “interpolated data point DPI”.

[0075] Now, interpolation methods for the unaccounted data will be explained. In a first example, the Z-position data corresponds to the coordinate CD(x, y) next to the unaccounted coordinate CD. BR (x, y) the data for interpolation. The "coordinate CD(x, y) next to the unrecorded coordinate CD" BR (x, y)” includes not only the coordinate CD(x, y) corresponding to the data point DP but also the interpolated coordinate CD IN (x, y). In the first case, all of the unaccounted coordinates can be CD. BR (x, y) by the interpolated coordinate CD IN (x, y) can be replaced. It should be noted that such an interpolation method is known.

[0076] A second example will be given with reference to Fig. 8 described. In Fig. Figure 8 shows the distance images DIMGs before and after interpolation. The distance image DIMG before interpolation is the same as the distance image DIMG that was interpolated. Fig. Figure 3 shows the second example. A search is performed for a data point DP in the negative y-axis (that is, a negative direction in the vertical axis) of the unscanned area BR. The unscanned data is then interpolated using the Z-position data of the first data point DP found in the search. The "first data point DP found in the search" is the data point DP with the shortest distance from the unscanned area BR in the negative y-axis direction. As described above, in the second example, the Z-position data of the first data point DP found in the search corresponds to the interpolation data.

[0077] In the Fig. In the example shown, all the unrecorded data in the unrecorded area BR1 are interpolated using the Z-position data (z1) of data point DP. All the unrecorded data in the unrecorded area BR2 are interpolated using the Z-position data (z2) of data point DP. All the unrecorded data in the unrecorded area BR3 are interpolated using the Z-position data (z3) of data point DP.

[0078] In fact, data point DP(z1) represents the Z-position data of the road surface RS directly beneath vehicle M2. Therefore, in this case, the interpolated data point DPI, which ensures the accuracy of the Z-position data, can be obtained. On the other hand, data points DP(z2) and DP(z3) represent the Z-position data on the painted surface of the vehicle body. In this case, it is simpler to link the data points DPs around the uncaptured area BR2 (or BR3) with the interpolated data point DPI.

[0079] During interpolation processing, it is further determined whether the data should be applied to the interpolated image IIMG for interpolation or not. This determination is made based on the following applicable condition or application condition.

[0080] Applicable condition or application condition: when the alignment of the distance image DIMG and the color image CIMG has been carried out, the pixel value IC(x, y) corresponds to the coordinate CD. IN (x, y) overlaps the preset pixel value of the black type.

[0081] If the applicable condition is met, the data is used for interpolation of the interpolated image IIMG. Otherwise, the data is discarded for interpolation. 2.2 Effect

[0082] According to the characteristics described above, the unacquired data is interpolated through interpolation processing. Therefore, it is possible to reduce the computational load of the clustering processing, which is performed after the interpolation. According to the second example of the interpolation method, it is possible to obtain the Z-position data of the road surface RS, which is located directly under vehicle M2. Therefore, it is possible to improve the accuracy of the clustering processing through the interpolated data point DPI, where the accuracy of the Z-position data is ensured. 2.3 Configuration example of the object detection device

[0083] Fig. Figure 9 is a diagram showing a configuration example of an object detection device 2 according to the second embodiment. As in Fig. As shown in Figure 9, the object recognition device 2 comprises the color image acquisition section 10, the distance image acquisition section 20 and the processing unit 40.

[0084] The hardware configuration of processing unit 40 is the same as that of processing unit 30, which is in Fig. Figure 6 shows that the function for performing object recognition processing, contained in processing unit 40, is the same as that contained in processing unit 30. Processing unit 40 includes a section 41 for interpolating uncaptured data as a configuration for performing interpolation processing. These function blocks, which are shown in Fig. The effects shown in 9 are realized when the processor of processing unit 40 executes a control program that is stored in the memory.

[0085] Section 41, on the interpolation of uncaptured data, performs the interpolation processing. In this processing, the uncaptured data is interpolated using the interpolation methods described above. The interpolation processing also determines whether the data is to be applied to the interpolated image IIMG for interpolation. This determination is made based on the special coordinate CP(x, y) and the applicable condition. As described above, the special coordinate CP(x, y) is specified in Section 32 for identifying a special coordinate. If the interpolated coordinate CD IN If (x, y) corresponds to the special coordinate CP(x, y), then the applicable condition is satisfied. 2.4 Specific processing

[0086] Fig. Figure 10 is a flowchart illustrating an example of interpolation processing performed by processing unit 40. The routine shown in this diagram is executed repeatedly according to a predetermined control cycle.

[0087] The processing of steps S21 and S22 is the same as that of steps S11 and S12, which are described in Fig. 7 are described. In step S23, the uncaptured data is interpolated. The interpolation method is as described above. The processing of step S24 is the same as that of step S13, which is described in Fig. 7 is described.

[0088] In step S25, it is determined whether the applicable condition is met. If the assessment result of step S25 is positive, then in step S26 the data for interpolation are applied to the interpolated image IIMG. If the assessment result of step S25 is negative, then the data for interpolation are discarded in step S27. The targets to be assessed with respect to the applicable condition are all of the interpolated data points (DPIs) on the interpolated image IIMG. That is, the assessment processing of steps S25 to S27 is repeated until the assessment result is obtained for all of the interpolated data points (DPIs). 3. Third example

[0089] Next, a third embodiment will be described with reference to Fig. 11 and Fig. 12 described. It should be noted that descriptions which overlap with those of the first or second embodiment are omitted where necessary. 3.1 Features of object recognition processing according to the third embodiment

[0090] Similar to the object recognition processing according to the second embodiment, the object recognition processing according to the third embodiment includes interpolation processing. However, in the interpolation processing according to the third embodiment, the Z-position data of the coordinate CD(x, y) corresponding to the road surface RS located directly beneath the vehicle M2 are specified based on position and terrain information of the vehicle M1. That is, the interpolation processing according to the third embodiment differs from that according to the second embodiment in the interpolation method.

[0091] For simplicity, a coordinate CD(x, y) corresponding to the road surface RS is referred to as a "road surface coordinate CD". RS (x, y)” in the following description.

[0092] The interpolation processing according to the third embodiment is described with reference to Fig. 11 described. In Fig. Figure 11 shows the distance images DIMGs before and after interpolation. The distance image DIMG before interpolation is the same as the distance image DIMG that was interpolated. Fig. 3 is shown. In which in Fig. In the example shown in Figure 11, the search for data point DP is performed in the negative y-axis direction of the uncaptured area BR until a data point DP corresponding to the road surface RS is found. The "data point DP corresponding to the road surface RS" is a data point DP with the shortest distance from the uncaptured area BR in the negative y-axis direction. The uncaptured data is then interpolated using the Z-position data of the data point DP corresponding to the road surface RS. As described above, in the third embodiment, the Z-position data of the data point DP corresponding to the road surface RS corresponds to the data for interpolation.

[0093] In the Fig. In example 11, the Z-position data of data point DP corresponding to the road surface RS is equal to z4. Thus, in this example, all the unrecorded data of the unrecorded areas BR1 to BR3 are interpolated using the same Z-position data (i.e., z4). 3.2 Effect

[0094] Based on the features described above, it is possible to achieve the same effects as in the second embodiment. That is, it is possible to reduce the computational load for the clustering processing performed after the interpolation processing. Additionally, since the interpolation is performed based on the Z-position data of the road surface RS, which is located directly beneath the vehicle M2, the accuracy of the clustering processing can be improved by the interpolated data point DP, where the accuracy of the Z-position data is ensured. 3.3 Configuration example of the object detection device

[0095] Fig. Figure 12 is a diagram showing a configuration example of the object detection device according to the third embodiment. As in Fig. As shown in Figure 12, the object recognition device 3 comprises the color image acquisition section 10, the distance image acquisition section 20, a position information acquisition section 50, a database 60 and a processing unit 70.

[0096] Position information acquisition section 50 obtains position information for vehicle M1. Position information acquisition section 50 is, for example, a GNSS receiver (GNSS "Global Navigation Satellite System"). Position information acquisition section 50 receives signals from three or more satellites. Position information acquisition section 50 calculates the position information of vehicle M1 based on the received signals. Position information acquisition section 50 transmits the position information to processing unit 70.

[0097] Database 60 is a database that stores terrain information. This terrain information is map information about the terrain. The terrain information includes the elevation Z of the road surface RS at each position [X, Y] on the map.

[0098] The hardware configuration of processing unit 70 is the same as that of processing unit 30, which is related to Fig. 6 is described. The function for performing object recognition processing, contained in processing unit 70, is the same as that contained in processing unit 30. Processing unit 70 includes a section 71 for interpolating uncaptured data as a configuration for performing interpolation processing. These function blocks, which are described in Fig. The functions shown in 12 are realized when the processor of processing unit 70 executes a control program that is stored in the memory.

[0099] Section 71, on the interpolation of unrecorded data, performs the interpolation processing. The interpolation processing performed by Section 71, on the interpolation of unrecorded data, is essentially the same as that performed by Section 41, on the interpolation of unrecorded data, which is described in Fig. As described in section 9, the process is carried out. That is, in the interpolation processing, the unrecorded data are interpolated using the interpolation method as described above. The interpolation processing also determines whether the z-position data (z RS ), to be applied to the interpolated image IIMG or not.

[0100] In step S11, a color image CIMG is acquired. In step S12, a distance image DIMG is acquired. In step S13, the color image CIMG is projected onto the distance image DIMG. The color image CIMG and the distance image DIMG are aligned before the color image CIMG is projected. In step S14, it is determined whether a basic condition is met. In step S15, it is determined whether a special condition is met. If an assessment result from step S14 or S15 is positive, then in step S16, a first data point and a second data point are linked on the distance image DIMG. If both assessment results from steps S14 and S15 are negative, the data points are not linked in step S17.

Claims

[1] Object detection device (1, 2, 3), with: a LIDAR (LD, 20) configured to emit a laser beam around a vehicle (M1) and to detect a laser beam reflected in an environment of the vehicle (M1); a processing unit (PU, 30, 40, 70) configured to perform clustering processing of a detected point group from the LIDAR (LD, 20) based on 3D position data of the corresponding detected point, including time-of-flight data of the laser beam; and a color image acquisition section (10) configured to acquire a color image (CIMG) of the environment, where in clustering processing the processing unit (PU, 30, 40, 70) is configured to select a first acquisition point corresponding to a first coordinate (CD). DP1 (x, y)) with a second detection point corresponding to a second coordinate (CD DP2(x, y)) to join if a joinable condition exists in the first and second coordinates (CD DP1 (x, y), CD DP2 (x, y)) on a distance image (DIMG) represented by the 3D position data, where the connectable condition is satisfied if there is a distance (dC) between the first and second coordinates (CD). DP1 (x, y), CD DB2 (x, y)) greater than or equal to a predetermined distance (dC) TH ) is a difference (dT) in the flight time data between the first and second recording points that is smaller than a predetermined difference (dT). TH ) is, and a condition for a coordinate is met, where the condition for a coordinate is satisfied if a route coordinate (CD) L (x, y)), which specifies a coordinate that lies on a route between the first and second coordinate (CD DP1 (x, y), CD DP2 (x, y)) is located, an unrecorded coordinate (CDBR (x, y)) is where the flight time data was not calculated, and a color pixel value that specifies a pixel value of each coordinate that defines the route coordinate (CD). L (x, y)) overlaps, a preset pixel value of the black type. [2] Object recognition device (1, 2, 3) according to claim 1, where the processing unit (PU, 30, 40, 70) is still configured to perform interpolation processing before executing clustering processing, where the interpolation processing is a processing to interpolate the flight time data in the unaccounted coordinate (CD). BR (x, y)) is, where in interpolation processing the processing unit (PU, 30, 40, 70) uses data for interpolation as the flight time data at the uncaptured coordinate (CD). BR (x, y)) is used when an applicable condition is met, where the applicable condition is met if the color pixel value of the uncaptured coordinate (CD) BR (x, y)) after interpolation is the preset pixel value of the black type. [3] Object recognition device (1, 2, 3) according to claim 2, wherein the data for interpolation are the flight time data at a coordinate with the shortest distance from the undetected coordinate (CD). BR (x, y)) in a negative direction in a vertical axis of the distance image (DIMG). [4] Object recognition device (1, 2, 3) according to claim 2, further comprising: a position information acquisition section (50) configured to acquire position information of the vehicle (M1); and a database (60) configured to store terrain information, where in the interpolation processing the processing unit (PU, 30, 40, 70) is still configured to provide a road surface coordinate (CD) RS (x, y)) specifying a coordinate corresponding to a road surface (RS) of distance image (DIMG) coordinates, based on the position information and the terrain information, where the data for interpolation are the flight time data at the coordinate of the road surface (CD). RS (x, y)) with the shortest distance from the unmeasured coordinate (CD BR (x, y)) in a negative direction in a vertical axis of the distance image (DIMG).

Images