Object detection device

Abstract

It enables early detection of distant moving objects and small moving objects. [Solution] The object detection device 50 includes a classification unit 114 that classifies point cloud data acquired by the lidar 5 into moving point cloud data whose absolute value of absolute movement speed is equal to or greater than a predetermined speed, and stationary point cloud data other than moving point cloud data; a storage unit 12 that stores the moving point cloud data in frame units; an offset processing unit 117 that offsets the position of each measurement point in the moving point cloud data included in past frames stored in the storage unit 12 based on the movement speed and direction of movement of each measurement point estimated based on the velocity information of each measurement point; a superposition unit 117 that superimposes the offset moving point cloud data of past frames onto the newly acquired point cloud data from the lidar 5; and a detection unit 118 that detects a moving object based on the moving point cloud data after the superposition processing.

Description

[Technical Field]

[0001] The present invention relates to an object detection device for detecting objects around a vehicle. [Background technology]

[0002] As an example of this type of device, there is a known device that detects moving objects using 3D point cloud data acquired by a lidar (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 7126633 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, if point cloud data is used directly for detecting moving objects, as in the apparatus described in Patent Document 1, there is a risk that the detection of distant or small moving objects may be delayed. [Means for solving the problem]

[0005] An object detection device according to one aspect of the present invention includes: a detector that acquires point cloud data including three-dimensional position information and velocity information indicating relative movement speed at measurement points on the surface of an object contained in three-dimensional space by irradiating electromagnetic waves into three-dimensional space and receiving reflected waves; a calculation unit that calculates the absolute movement speed of each of a plurality of measurement points corresponding to the point cloud data based on the velocity information; a classification unit that, when point cloud data is acquired by the detector, classifies the point cloud data into moving point cloud data whose absolute value of the absolute movement speed calculated by the calculation unit is equal to or greater than a predetermined speed, and stationary point cloud data other than moving point cloud data; a storage unit that stores the moving point cloud data classified by the classification unit for each point cloud frame containing moving point cloud data at the same time; and a processing unit that performs processing to detect a moving object moving in three-dimensional space. The processing unit performs an offset process to offset the position of each measurement point in the moving point cloud data contained in past point cloud frames stored in the memory unit, based on the moving speed and direction of each measurement point estimated based on the velocity information of each measurement point. When the moving point cloud data newly acquired by the detector is classified by the classification unit, an overlay process is performed to superimpose the moving point cloud data from the offset-processed past point cloud frames onto the moving point cloud data, and a moving object is detected based on the moving point cloud data after the overlay process. [Effects of the Invention]

[0006] According to the present invention, distant moving objects and small moving objects can be detected early. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing the main components of a vehicle control device including an object detection device according to an embodiment of the present invention. [Figure 2A] A diagram showing an example of a three-dimensional object contained in the three-dimensional space surrounding the vehicle. [Figure 2B] A plan view of the moving object in Figure 2A, seen from above. [Figure 3A] A diagram showing multiple pedestrians passing each other. [Figure 3B]A diagram showing an example of XYV data. [Figure 4A] A diagram showing an example of a three-dimensional space around the host vehicle. [Figure 4B] A diagram showing an example of XYV data corresponding to the three-dimensional space of FIG. 4A. [Figure 4C] A diagram showing an example of XYV data corresponding to the three-dimensional space of FIG. 4A. [Figure 5A] A diagram showing an example of a point cloud of a past frame superimposed on the XYV data of the current frame without offset processing. [Figure 5B] A diagram showing an example of superimposed XYV data. [Figure 6] A diagram for explaining the detection of a moving object. [Figure 7] A diagram for explaining the calculation of the movement vector of a moving object. [Figure 8] A flowchart showing an example of the process executed by the CPU of the controller in FIG. 1.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The object detection device according to the embodiments of the present invention can be applied to a vehicle having an automatic driving function, that is, an autonomous vehicle. Note that the vehicle to which the object detection device according to the present embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be any of an engine vehicle having an internal combustion engine (engine) as a driving source for traveling, an electric vehicle having a driving motor as a driving source for traveling, and a hybrid vehicle having an engine and a driving motor as driving sources for traveling. The host vehicle can travel not only in an automatic driving mode in which driving operation by a driver is unnecessary but also in a manual driving mode by the driver's driving operation.

[0009] When an autonomous vehicle is driving in autonomous driving mode (hereinafter referred to as autonomous driving or self-driving), it recognizes the external environment around the vehicle based on detection data from on-board detectors such as LiDAR (Light Detection and Ranging). Based on this recognition result, the autonomous vehicle generates a driving trajectory (target trajectory) for a predetermined time from the present moment and controls the driving actuators so that the vehicle drives along the target trajectory.

[0010] Figure 1 is a block diagram showing the main components of a vehicle control device 100, including an object detection device. This vehicle control device 100 includes a controller 10, a communication unit 1, a positioning unit 2, an internal sensor group 3, a camera 4, a lidar 5, and a driving actuator AC. The vehicle control device 100 also includes an object detection device 50, which constitutes part of the vehicle control device 100. The object detection device 50 detects objects around the vehicle based on detection data from the lidar 5.

[0011] Communication unit 1 communicates with various servers (not shown) via a network including wireless communication networks such as the Internet and mobile phone networks, and acquires map information, driving history information, and traffic information from the servers periodically or at arbitrary times. The network includes not only public wireless communication networks but also closed communication networks established for each designated management area, such as wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The acquired map information is output to storage unit 12, and the map information is updated. Positioning unit (GNSS unit) 2 has a positioning sensor that receives positioning signals transmitted from positioning satellites. Positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. Positioning unit 2 uses the positioning information received by the positioning sensor to measure the current position (latitude, longitude, altitude) of its own vehicle.

[0012] The internal sensor group 3 is a collective term for multiple sensors (internal sensors) that detect the vehicle's driving state. For example, the internal sensor group 3 includes a vehicle speed sensor that detects the vehicle's speed, acceleration sensors that detect the vehicle's longitudinal and lateral acceleration (lateral acceleration), a rotation speed sensor that detects the rotation speed of the drive source, and a yaw rate sensor that detects the rotational angular velocity of the vehicle's center of gravity around its vertical axis. Sensors that detect the driver's operations in manual driving mode, such as accelerator pedal operation, brake pedal operation, and steering wheel operation, are also included in the internal sensor group 3.

[0013] Camera 4 has an image sensor such as a CCD or CMOS and captures images of the area around the vehicle (front, rear, and sides). LiDAR 5 irradiates electromagnetic waves (reflected waves) into the three-dimensional space around the vehicle and detects the external conditions around the vehicle based on the reflected waves. More specifically, the electromagnetic waves (such as laser light) irradiated by LiDAR 5 are reflected back at a certain point (measurement point) on the surface of an object, and the distance from the laser source to that point, the intensity of the reflected electromagnetic waves, and the relative velocity of the object located at that measurement point are measured. The electromagnetic waves from LiDAR 5, which is mounted at a predetermined position (front) of the vehicle, are scanned horizontally and vertically around the vehicle (forward), so the position, shape, and relative speed of objects in front of the vehicle (moving objects such as other vehicles and stationary objects such as the road surface and structures) are detected. Objects detected by LiDAR 5, including people, are referred to as "objects". Therefore, moving objects include not only vehicles such as cars and bicycles in motion, but also people in motion (pedestrians, etc.). In the following, the above three-dimensional space will be represented by the X-axis along the direction of travel of the vehicle, the Y-axis along the width direction of the vehicle, and the Z-axis along the height direction of the vehicle. Therefore, the above three-dimensional space may be called the XYZ space.

[0014] Actuator AC is a drive actuator used to control the movement of the vehicle. When the drive source is an engine, actuator AC includes a throttle actuator that adjusts the opening degree (throttle opening) of the engine's throttle valve. When the drive source is a drive motor, the drive motor is included in actuator AC. Brake actuators that operate the vehicle's braking system and steering actuators that drive the steering system are also included in actuator AC.

[0015] The controller 10 is comprised of an electronic control unit (ECU). More specifically, the controller 10 includes a computer comprising an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as ROM or RAM, and other peripheral circuits (not shown) such as an I / O interface. While multiple ECUs with different functions, such as an engine control ECU, a drive motor control ECU, and a braking system ECU, can be provided separately, in Figure 1, for convenience, the controller 10 is shown as a collection of these ECUs.

[0016] The memory unit 12 stores highly accurate and detailed map information (referred to as high-precision map information). This high-precision map information includes road location information, road shape information (curvature, etc.), road gradient information, intersection and branching point location information, number of lanes (driving lanes), lane width and location information for each lane (lane center position and lane boundary information), location information of landmarks (traffic lights, signs, buildings, etc.) as map markers, and road surface profile information such as road surface irregularities. The memory unit 12 also stores various control programs, information such as thresholds used in the programs, and setting information for on-board detectors such as the lidar 5.

[0017] The calculation unit 11 has the following functional configuration: a data acquisition unit 111, an estimation unit 112, a calculation unit 113, a classification unit 114, a generation unit 115, a preliminary detection unit 116, an overlay unit 117, a detection unit 118, a vector calculation unit 119, a tracking unit 120, and a driving control unit 121.

[0018] As shown in Figure 1, the data acquisition unit 111, estimation unit 112, calculation unit 113, classification unit 114, generation unit 115, provisional detection unit 116, superposition unit 117, and detection unit 118 are included in the object detection device 50. Details of the data acquisition unit 111, estimation unit 112, calculation unit 113, classification unit 114, generation unit 115, provisional detection unit 116, superposition unit 117, and detection unit 118, as well as the vector calculation unit 119 and tracking unit 120 included in the object detection device 50 will be described later.

[0019] In automatic driving mode, the driving control unit 121 generates a target trajectory based on the external conditions around the vehicle, including the size, position, and relative speed of objects detected by the detection unit 118, and the movement trajectory of objects determined by the tracking unit 120. Specifically, the driving control unit 121 generates a target trajectory to avoid collisions or contact with objects or to follow those objects. The driving control unit 121 controls the actuator AC so that the vehicle travels along the target trajectory. Specifically, it controls the actuator AC along the target trajectory to adjust the accelerator opening and drive the braking and steering devices. In manual driving mode, the driving control unit 121 controls the actuator AC in response to driving commands (such as steering operations) from the driver acquired by the internal sensor group 3.

[0020] Details of the object detection device 50 will now be described. As described above, the object detection device 50 includes a data acquisition unit 111, an estimation unit 112, a calculation unit 113, a classification unit 114, a generation unit 115, a preliminary detection unit 116, an overlay unit 117, and a detection unit 118. The object detection device 50 further includes a lidar 5.

[0021] The data acquisition unit 111 acquires four-dimensional data (hereinafter referred to as point cloud data) as detection data for the LIDA 5. This data includes positional information indicating the three-dimensional position coordinates of measurement points on the surface of an object from which the reflected wave of the LIDA 5 can be obtained, and velocity information indicating the relative movement velocity of the measurement points. The point cloud data is acquired by the LIDA 5 on a frame-by-frame basis, specifically at predetermined time intervals (time intervals determined by the frame rate of the LIDA 5).

[0022] The estimation unit 112 estimates the absolute moving speed of the host vehicle based on the point cloud data acquired by the data acquisition unit 111. Here, the estimation of the absolute moving speed of the host vehicle by the estimation unit 112 will be described.

[0023] First, the estimation unit 112 extracts from the point cloud data acquired by the data acquisition unit 111 the point cloud data excluding the information of the measurement points corresponding to the three-dimensional object, that is, the point cloud data corresponding to the road surface around the host vehicle (hereinafter referred to as road surface point cloud data). The estimation unit 112 uses the following formula (i) to calculate the unit vector e i indicating the direction of the relative moving speed v i based on the position coordinates (x i , y i , z i ) included in the four-dimensional data (x i , y i , z i , v i ) of the road surface point cloud data, that is, the measurement points P i (i = 1, 2,..., n) corresponding to the road surface. i (i = 1, 2,..., n) of the four-dimensional data (x i , y i , z i , v i ) i y i i z i i v i i )(x i , y i , z i ) i y i i z i i Based on the position coordinates (x i , y i , z i ) i v i i e i

[0024]

Equation

[0025] Next, the estimation unit 112 estimates the vehicle's movement vector (movement speed (absolute movement speed) and direction of movement) Vself. Specifically, the estimation unit 112 sets a conversion formula for converting the relative movement speed vi of a measurement point Pi corresponding to the road surface into absolute movement speed as the objective function L, and solves an optimization problem to optimize the objective function L to approach zero. Since the measurement points Pi are measurement points on the road surface, the absolute speed of each of these measurement points should be zero. Therefore, by optimizing the objective function L to approach zero, the correct Vself can be estimated. Vself is expressed in terms of velocity components in the XYZ axis directions, as shown in equation (ii) below. The objective function L is expressed by equation (iii) below. By solving the above optimization problem, a Vself such that the right-hand side of equation (iii) is zero is searched for. Note that Vself may be initially set to zero, or the Vself estimated in the previous frame may be set. Alternatively, measurement points that are estimated to have an absolute speed of zero may be extracted by another method, and the extracted measurement points may be used as measurement points Pi.

[0026]

number

number

[0027] In equation (iii), A is a matrix of unit vectors ei of n measurement points corresponding to the road surface, and is expressed by equation (iv). Also in equation (iii), V is a 1×n matrix representing the velocity components (relative movement speed) of n measurement points Pi corresponding to the road surface, and is expressed by equation (v). The estimation unit 112 obtains Vself, obtained by solving the above optimization problem, as an estimated value of the absolute movement speed of the vehicle in the current frame.

[0028]

number

number

[0029] The calculation unit 113 calculates the absolute movement speed of all measurement points, more specifically, all measurement points including those corresponding to three-dimensional objects, based on the vehicle's movement vector Vself estimated by the estimation unit 112. Here, the calculated absolute movement speed of a measurement point is negative when the measurement point approaches the vehicle and positive when the measurement point moves away from the vehicle.

[0030] The classification unit 114 classifies the point cloud data acquired by the data acquisition unit 111 into moving point cloud data corresponding to measurement points where the absolute value of the absolute moving velocity calculated by the calculation unit 113 is equal to or greater than a predetermined velocity Th#V, and stationary point cloud data corresponding to measurement points where the absolute value is less than the predetermined velocity Th#V.

[0031] The generation unit 115 generates velocity-added data by adding the absolute movement speed calculated by the calculation unit 113 to the moving point cloud data. More specifically, the generation unit 115 adds the absolute movement speed corresponding to each measurement point to the position information of each measurement point included in the moving point cloud data. The generation unit 115 stores the generated velocity-added data in the storage unit 12 along with the frame ID of the current frame.

[0032] Figures 2A and 2B show examples of three-dimensional objects contained in the three-dimensional space surrounding a vehicle. Figure 2A shows a moving object (bicycle CY and the person riding bicycle CY RD) traveling in front of the vehicle in the direction of travel (X direction). Figure 2B shows a plan view of the moving object in Figure 2A, seen from above (Z direction). As shown in Figure 2B, the maximum size of the three-dimensional object in the X and Y directions (Xmax and Ymax) can be recognized even without information on the height direction (Z direction) of the object.

[0033] Therefore, the generation unit 115 may project each of the measurement points onto a plane to remove height information from the position information of each measurement point corresponding to the moving point cloud data, thereby converting the position information of each measurement point from three dimensions to two dimensions. Specifically, when the position coordinates of each measurement point are expressed in an XYZ coordinate system, the generation unit 115 may project each measurement point corresponding to the moving point cloud data onto an XY plane to convert the moving point cloud data into two-dimensional data (XY data) expressed in an XY coordinate system. In this case, the generation unit 115 generates three-dimensional velocity-added data (XYV data) by adding absolute movement velocity to the XY data. The following explanation will use the case where the velocity-added data generated by the generation unit 115 is XYV data as an example.

[0034] The preliminary detection unit 116 detects moving objects around the vehicle based on the XYV data generated by the generation unit 115. More specifically, the preliminary detection unit 116 performs clustering on the XYV data to detect the bounding box, which is the circumscribing region of the moving object, from the XY plane.

[0035] Incidentally, objects that are far away from the vehicle or are small in size may not be detected or may be detected late in the clustering process because the number of point clouds (number of measurement points) measured by the lidar is small. Therefore, in order to suppress such detection omissions and detection delays, the provisional detection unit 116 lowers the threshold for the number of measurement points to be considered as a point cloud and performs the clustering process. In the clustering process by the provisional detection unit 116 (hereinafter referred to as provisional clustering process), a value Th0 smaller than the threshold Th1 of the clustering process by the detection unit 118, which will be described later, is set as the threshold.

[0036] In the following, the detection of moving objects by the provisional detection unit 116 may be referred to as provisional detection of moving objects. Note that any of the following methods may be used for the clustering process performed by the provisional detection unit 116 and the detection unit 118: DBSCAN (Density-based spatial clustering of applications with noise), K-means, etc.

[0037] The provisional detection unit 116 detects the position and size of the moving object on the XY plane based on the position and size of the bounding box (circumscribing region) detected by the provisional clustering process. The provisional detection unit 116 stores information that can identify the measurement point cloud corresponding to the detected moving object, that is, the measurement point cloud included in the detected bounding box, in the storage unit 12 as a provisional detection result.

[0038] Here, we will explain the provisional detection of moving objects by the provisional detection unit 116. Figure 3A shows a scene in which multiple pedestrians are moving. Here, instead of detection data from a lidar mounted on a vehicle, we will use detection data from a lidar installed in the concourse AS shown in Figure 3A as an example. Figure 3A shows the concourse AS as seen from the lidar's perspective. Figure 3A shows pedestrians HM32 and HM34 moving (walking) in the same direction (X-axis direction) along the extension direction of the concourse AS, while pedestrians HM31, HM33, and HM35 are moving (walking) in the opposite direction. The absolute value of the absolute movement speed of pedestrians HM31 to HM35 is greater than or equal to a predetermined speed Th#V.

[0039] Figure 3B shows an example of 3D data (XYV data) generated by the generation unit 115. Figure 3B shows XYV data obtained by adding the absolute movement speeds of pedestrians HM31 to HM35 to the 2D data obtained by projecting the measurement point clouds (clusters) PC1 to PC5 corresponding to pedestrians HM31 to HM35 in Figure 3A onto the XY plane. In Figure 3B, the measurement point cloud for each pedestrian is drawn in a color corresponding to the absolute movement speed of each pedestrian. For the sake of simplicity, the absolute movement speeds of pedestrians HM31, HM33, and HM35 are assumed to be equal. Also, the absolute movement speeds of pedestrians HM32 and HM34 are assumed to be equal. Therefore, in Figure 3B, the measurement point clouds PC1, PC3, and PC5 corresponding to pedestrians HM31, HM33, and HM35 are drawn in the same color (black), while the measurement point cloud PC3 corresponding to pedestrians HM32 and HM34 is drawn in a different color (white).

[0040] Figure 3B also shows bounding boxes B1 to B5 detected by the provisional clustering process. Bounding boxes B1 to B5 correspond to the measurement point clouds PC1 to PC5, respectively. As described above, the provisional detection unit 116 performs clustering by lowering the threshold for the number of measurement points considered as a point cloud in order to suppress the failure to detect moving objects. Therefore, bounding box B5 corresponding to measurement point cloud PC5, which contains only a few measurement points (2 points in the figure), is also detected. Furthermore, in the clustering process for XYV data, velocity information is considered in the classification of measurement points, so measurement point clouds PC2 and PC3 corresponding to pedestrians HM32 and HM33 moving at different absolute speeds are not recognized as a single measurement point cloud even if they are close to each other, but are recognized as separate measurement point clouds. As a result, as shown in Figure 3B, bounding boxes B2 and B3 corresponding to measurement point clouds PC2 and PC3 are detected, respectively.

[0041] The superposition unit 117 estimates the amount of movement of the moving object detected by the provisional detection unit 116 from a past frame (for example, the previous frame) to the current frame. Based on the estimated amount of movement, the superposition unit 117 superimposes the XYV data of the current frame with the XYV data of the past frame. The XYV data generated by this superposition is called superimposed velocity data or superimposed XYV data.

[0042] Here, we will explain the generation of superimposed XYV data. First, the superimposing unit 117 estimates the movement vector (absolute movement velocity and direction of movement) of the moving object detected by the provisional detection unit 116 using the following equation (vi). Hereafter, the moving object detected by the provisional detection unit 116 may be referred to as the provisional detection object.

[0043]

number

[0044] In equation (vi), Vmodel is the movement vector of the measurement point corresponding to the tentatively detected object, and when there are m measurement points constituting the tentatively detected object, (v xi , v yi , v zi )(i=1,2,…,m). Vself is the movement vector of the vehicle estimated by the estimation unit 112 using equation (iii). A is a matrix of unit vectors ei of n measurement points corresponding to the road surface, used to estimate Vself. V is a 1×n matrix representing the velocity components (relative movement velocity) of n measurement points corresponding to the road surface, used to estimate Vself.

[0045] The superimposition unit 117 stores the calculated movement vector Vmodel in the storage unit 12, along with information (identifier) ​​that can identify the corresponding provisionally detected object, and associates it with the frame ID of the current frame. When multiple moving objects are detected by the provisional detection unit 116, the superimposition unit 117 calculates a movement vector corresponding to each moving object (provisionally detected object). Each calculated movement vector is stored in the storage unit 12 along with information (identifier) ​​that can identify the corresponding provisionally detected object.

[0046] Next, the superimposing unit 117 reads the movement vector of the provisionally detected object, which is stored in association with the frame ID of the past frame, from the storage unit 12. The superimposing unit 117 calculates the amount of movement of the provisionally detected object from the past frame to the current frame by multiplying the velocity component of the movement vector of the provisionally detected object read from the storage unit 12 by the elapsed time from the past frame to the current frame. Based on the calculated amount of movement and the direction indicated by the movement vector, the superimposing unit 117 performs an offset process to offset (translate) the measurement point cloud corresponding to the provisionally detected object, which is included in the XYV data of the past frame.

[0047] If the XYV data of a past frame contains measurement point clouds corresponding to multiple tentatively detected objects, an offset process is performed for each measurement point cloud corresponding to each tentatively detected object. The superposition unit 117 overlays the offset-processed measurement point clouds onto the XYV data of the current frame to generate superimposed XYV data.

[0048] The superimposing unit 117 may also generate superimposed XYV data by superimposing the measurement point clouds of multiple past frames onto the XYV data of the current frame. For example, if frame n (n: frame number) is the current frame, the superimposed XYV data may be generated by superimposing the measurement point clouds of the provisionally detected objects contained in the XYV data of frames n-1, n-2, and n-3 onto the XYV data of frame n.

[0049] In this case, the superimposed part 117 uses the above equation (vi) to determine the movement vector mv of the provisionally detected object corresponding to frames n-1, n-2, and n-3, respectively. n-1 ,mv n-2 ,mv n-3 The superimposed part 117 calculates the calculated movement vector mv n-1 ,mv n-2 ,mv n-3 Multiply each velocity component by the travel time T, T×2, and T×3 to obtain the amount of movement MA of the provisionally detected object between each frame, between frame n and frames n-1, n-2, and n-3. n-1 MA n-2 MA n-3 Calculate each of them.

[0050] The superimposed part 117 is the movement vector mv n-1 ,mv n-2 ,mv n-3 The direction and the amount of movement MA n-1 MA n-2 MA n-3Based on this, an offset process is performed on each of the measurement point clouds of the provisionally detected objects corresponding to frames n-1, n-2, and n-3. The superposition unit 117 superimposes the offset-processed measurement point clouds of the provisionally detected objects in frames n-1, n-2, and n-3 onto the XYV data of frame n to generate superimposed XYV data.

[0051] If the vehicle 101 is in motion, the superimposing unit 117 further performs an offset rotation process to offset and rotate the measured point cloud of the provisionally detected object included in the XYV data of past frames, based on the azimuth angle difference and movement vector of the vehicle 101 between frames. The movement vector of the vehicle 101 represents the direction of movement of a representative point (such as the center of gravity) of the vehicle 101 between frames and the speed of movement in that direction. The azimuth angle difference of the vehicle 101 is the angle difference between the azimuth in the current frame and the azimuth (direction of travel) of the vehicle 101 in past frames. The azimuth angle difference and movement vector of the vehicle 101 can be estimated by performing a predetermined scan matching process and superimposing the static point cloud data of past frames onto the static point cloud data of the current frame.

[0052] Figure 4A shows an example of the three-dimensional space around the vehicle. In Figure 4A, the vertical direction (X-axis direction) represents the direction of movement (direction of travel) of the vehicle 101. The horizontal direction (left-right direction in the figure) and the height direction (from back to front in the figure) relative to the X-axis direction represent the Y-axis and Z-axis directions.

[0053] Figure 4A shows the vehicle 101 and the objects surrounding it at time t1. Objects SB1 to SB6 are stationary objects, and objects MB1 and MB2 are moving objects. Stationary objects SB1 to SB6 include the road surface on which the vehicle 101 is traveling, structures such as walls and median strips installed on the side of the road, and other vehicles parked on the shoulder of the road. Moving objects include other vehicles and pedestrians. For the sake of simplicity, the vehicle 101 is assumed to be stationary.

[0054] Figures 4B and 4C show examples of XYV data corresponding to the three-dimensional space in Figure 4A, generated by the generation unit 115. Note that the XYV data generated by the generation unit 115 does not actually include the measurement point clouds N1 to N6 corresponding to the stationary objects SB1 to SB6, but these measurement points are shown in Figures 4B and 4C for clarity. The same applies to Figures 5A, 5B, 6, and 7, which will be described later.

[0055] Figure 4B shows the XYV data for a past frame (the frame at time t1). Figure 4C shows the XYV data for the current frame (the frame at the present time (time t2), after frame time T has elapsed from time t1). In Figures 4B and 4C, the vertical direction (X-axis direction) represents the direction of movement (direction of travel) of the vehicle 101. The horizontal direction (left-right direction in the figure) relative to the X-axis direction represents the Y-axis direction.

[0056] In Figures 4B and 4C, regions N1 to N6 schematically represent the measurement point clouds corresponding to stationary objects SB1 to SB6, or more precisely, the positions and sizes of the measurement point clouds. Regions M11 and M12 in Figure 4B schematically represent the measurement point clouds corresponding to moving objects MB1 and MB2, or more precisely, the positions and sizes of the measurement point clouds. Regions M13 and M14 in Figure 4B represent measurement points or measurement point clouds that do not correspond to either of the moving objects MB1 or MB2, and represent noise. The arrows mv11 to mv14 in Figure 4B schematically represent the movement vector Vmodel of the measurement point clouds M1 to M4 estimated by equation (vi) above.

[0057] Regions M21 and M22 in Figure 4C schematically represent the measurement point clouds corresponding to the moving objects MB1 and MB2, or more precisely, the position and size of the measurement point clouds. Region M23 in Figure 4C represents measurement points or measurement point clouds that do not correspond to either of the moving objects MB1 or MB2, and represents noise.

[0058] Figures 5A and 5B show examples of measurement point clouds from past frames superimposed on the XYV data of the current frame. Figure 5A shows an example of the current frame's XYV data in which the measurement point clouds M11-M14 of moving objects, which are included in the XYV data of past frames, are superimposed without offset processing. Figure 5B shows an example of the current frame's XYV data in which the measurement point clouds M11-M14 of moving objects, which are included in the XYV data of past frames, are superimposed after offset processing, i.e., superimposed XYV data generated by the superposition unit 117.

[0059] In Figure 5A, the region M11p to M14p, indicated by the dashed line, schematically represents the measured point cloud M11 to M14 of past frames superimposed on the current frame. In Figure 5B, the region M11o to M14o, indicated by the dashed line, schematically represents the measured point cloud M11 to M14 of past frames superimposed on the current frame. The measured point cloud M11o to M14o in Figure 5B is offset in the direction of the movement vectors mv11 to mv14 by an amount of movement obtained by multiplying the velocity component of the movement vectors mv11 to mv14 by the frame time T, relative to the measured point cloud M11p to M14p in Figure 5A.

[0060] Furthermore, since each measurement point in the XYV data of past frames has velocity information indicating relative movement speed, the movement speed and direction of each measurement point can be estimated based on that velocity information. Therefore, the superimposing unit 117 may offset each measurement point in the XYV data of the previous frame based on the velocity information that each measurement point possesses. The superimposing unit 117 may then overlay the offset measurement points onto the XYV data of the current frame to generate superimposed XYV data.

[0061] Figure 6 is a diagram illustrating the detection of moving objects by the detection unit 118. The detection unit 118 performs clustering on the superimposed XYV data (Figure 5B) generated by the superimposition unit 117 to detect moving objects around the vehicle.

[0062] In the superimposed XYV data in Figure 5B, the measurement point cloud M11o overlaps with the measurement point cloud M21. Therefore, in this clustering process, measurement point clouds M11o and M21 are recognized as a single measurement point cloud. As a result, even if the number of measurement points in either measurement point cloud M11o or measurement point cloud M21 is less than the threshold Th1, if the total number of measurement points is equal to or greater than the threshold Th1, a bounding box B10 enclosing both measurement point clouds is detected, as shown in Figure 6. Similarly, a bounding box B20 enclosing measurement point clouds M12o and M22 is detected.

[0063] The detection unit 118 detects the position and size of the moving object (objects MB1 and MB2 in Figure 4A) in three-dimensional space (XYZ space) based on the position and size of the detected bounding boxes B10 and B20.

[0064] As described above, by offsetting the measurement point cloud of the moving object tentatively detected in the previous frame and overlaying it with the XYV data of the current frame (Figure 5B), the number of measurement points corresponding to the moving object can be increased. As a result, even moving objects that are far away or small in size, for which there are not enough corresponding measurement points in a single frame, can be detected early. In addition, as shown in the measurement point clouds M13o and M14o in Figure 6, measurement point clouds from past frames superimposed on the current frame that are not included in any bounding box can be identified as noise.

[0065] The vector calculation unit 119 calculates the movement vector of the moving object detected by the detection unit 118. First, the vector calculation unit 119 performs a determination process to identify the same object between frames (between past frames and the current frame) for the moving object detected by the detection unit 118. In this determination process, it is determined that the measurement point cloud of the current frame included in the bounding box detected by the detection unit 118 (hereinafter referred to as the current measurement point cloud) and the measurement point cloud of past frames superimposed on the current frame (hereinafter referred to as the superimposed measurement point cloud) correspond to the same moving object. In the example in Figure 6, it is determined that the current measurement point cloud M21 and the superimposed measurement point cloud M11o included in bounding box B10 correspond to the same moving object. Also, it is determined that the current measurement point cloud M22 and the superimposed measurement point cloud M12o included in bounding box B20 correspond to the same moving object.

[0066] The vector calculation unit 119 calculates the motion vector of the moving object detected by the detection unit 118 based on the result of the above determination process. Figure 7 is a diagram illustrating the calculation of the motion vector of the moving object by the vector calculation unit 119.

[0067] First, the vector calculation unit 119 overlays the XYV data of past frames onto the XYV data of the current frame, as shown in Figure 7. In Figure 7, the XYV data of past frames (frame at time t1) is shown as a dashed line, and the XYV data of the current frame (frame at time t2) is shown as a solid line. If the vehicle 101 is in motion, the vector calculation unit 119 performs an offset rotation process on the XYV data of past frames based on the movement vector of the vehicle 101 and the difference in azimuth angle before overlaying.

[0068] The measurement point clouds M11p to M14p in Figure 7 are the measurement point clouds M11 to M14 of past frames superimposed on the current frame. The representative points G11p, G12p, G21, and G22 represent the centroids of the measurement point clouds M11p, M12p, M21, and M22. The representative points G11p, G12p, G21, and G22 may be other than the centroids.

[0069] The vector calculation unit 119 calculates the movement vector MV1 based on the positional relationship between the representative point G21 of the measurement point cloud M21 and the representative point G11p of the measurement point cloud M11p, which is determined to correspond to the same moving object as the measurement point cloud M21. The movement vector MV1 indicates the movement speed and direction of the moving object corresponding to the measurement point cloud M11p and the measurement point cloud M21 from past frames to the current frame.

[0070] Similarly, the vector calculation unit 119 calculates the movement vector MV2 of the moving object corresponding to the measured point clouds M12p and M22 based on the positional relationship between the representative points G12p and G22. The vector calculation unit 119 stores the calculated movement vectors MV1 and MV2 in the storage unit 12 along with information (identifiers) that can identify the corresponding moving object.

[0071] The tracking unit 120 tracks a moving object (objects MB1 and MB2 in Figure 4A) when the movement vectors (movement vectors MV1 and MV2 in Figure 7) corresponding to the moving object detected by the detection unit 118 are stored in the storage unit 12. Based on the movement vectors of the tracked object (hereinafter referred to as the tracked object), the tracking unit 120 determines the movement trajectory of the tracked object between frames (between past frames and the current frame). While the tracked object is detected by the detection unit 118, the tracking unit 120 tracks the movement path of the tracked object by accumulating the movement trajectory determined for each frame.

[0072] Figure 8 is a flowchart showing an example of the processing performed by the arithmetic unit 11 of the controller 10 in Figure 1 according to a predetermined program. The processing shown in this flowchart is repeated at predetermined intervals while the vehicle control device 100 is running. More specifically, it is repeated at intervals according to the frame rate of the rider 5.

[0073] First, in step S1, the external conditions around the vehicle are detected. Specifically, an illumination command is sent to the rider 5, and point cloud data (detection data) is acquired that includes position information and speed information of the measurement point where the reflected electromagnetic wave emitted from the rider 5 in response to the illumination command is obtained. In step S2, the point cloud data acquired in step S1 is classified into moving point cloud data and stationary point cloud data.

[0074] Next, the processes in steps S3 to S8 are executed on the moving point cloud data. Although the controller 10 also performs predetermined processing on the stationary point cloud data, the explanation is omitted here.

[0075] In step S3, absolute movement speed is added to the positional information of each measurement point included in the moving point cloud data to generate velocity-added data (XYV data). In step S4, a preliminary clustering process is performed on the XYV data generated in step S3. This allows for the preliminary detection of moving objects around the vehicle 101. The preliminary detection results of the moving objects are stored in the storage unit 12 along with the generated XYV data, associated with the frame ID of the current frame. The preliminary detection results include information that allows for the identification of the measurement point cloud corresponding to the preliminaryly detected moving object.

[0076] In step S5, the movement vector of the moving object provisionally detected in step S4 is estimated using equation (vi) above. The estimated movement vector is stored in the storage unit 12 as part of the provisional detection result, along with information (identifier) ​​that can identify the corresponding moving object.

[0077] In step S6, based on the provisional detection result of the previous frame stored in the memory unit 12, the measurement point cloud corresponding to the previously provisionally detected moving object is offset and superimposed on the XYV data (XYV data of the current frame) generated in step S3. More specifically, based on the provisional detection result, the measurement point cloud corresponding to the previously provisionally detected moving object is obtained from the XYV data of the previous frame, and the obtained measurement point cloud is offset based on the movement vector of that moving object. Then, the offset measurement point cloud is superimposed on the XYV data of the current frame. This generates superimposed XYV data.

[0078] In step S7, clustering is performed on the superimposed XYV data generated in step S6. The positions and sizes of the bounding boxes detected by this clustering process are then used to determine the positions and sizes of moving objects around the vehicle.

[0079] Finally, in step S8, the movement vector of the moving object detected in step S7 is estimated (calculated). More specifically, it is calculated based on the position of the measurement point cloud of the current frame (current measurement point cloud) included in the bounding box detected in step S7, and the position of the measurement point cloud of the previous frame superimposed on the current frame (superimposed measurement point cloud) included in that bounding box in the previous frame.

[0080] As mentioned above, each measurement point in the XYV data has velocity information indicating the relative movement speed, so each measurement point may be offset based on that velocity information. That is, without performing the processing in steps S4 and S5, in step S6, each measurement point in the XYV data of the previous frame may be offset based on the velocity information that each measurement point has. Then, the offset measurement points may be superimposed on the XYV data generated in step S3 to generate superimposed XYV data.

[0081] According to the embodiments described above, the following effects and advantages are achieved. (1) The object detection device 50 includes a lidar 5 that acquires point cloud data including three-dimensional position information and velocity information indicating relative movement speed at measurement points on the surface of an object included in three-dimensional space by irradiating electromagnetic waves into three-dimensional space and receiving reflected waves; a calculation unit 113 that calculates the absolute movement speed of each of a plurality of measurement points corresponding to the point cloud data based on the velocity information; a classification unit 114 that, when point cloud data is acquired by the lidar 5, classifies the point cloud data into moving point cloud data for which the absolute value of the absolute movement speed calculated by the calculation unit 113 is equal to or greater than a predetermined speed, and stationary point cloud data other than moving point cloud data; a storage unit 12 that stores the moving point cloud data classified by the classification unit 114 on a frame basis, that is, for each point cloud frame that contains moving point cloud data at the same time; and a processing unit that performs processing to detect a moving object moving in three-dimensional space. The processing unit performs an offset process to offset the position of each measurement point in the moving point cloud data contained in past point cloud frames stored in the storage unit 12, based on the moving speed and direction of each measurement point estimated based on the velocity information of each measurement point. When the moving point cloud data newly acquired by the lidar 5 is classified by the classification unit, an overlay process is performed to superimpose the offset moving point cloud data of past frames onto the moving point cloud data, and moving objects are detected based on the moving point cloud data after the overlay process. This allows for the early detection of distant objects and small objects. Furthermore, the offset process described above allows for the accurate detection of moving objects that have moved a large amount between frames.

[0082] (2) When the moving point cloud data newly acquired by the lidar 5 is classified by the classification unit 114, the processing unit performs a provisional clustering process as a first clustering process on the moving point cloud data, setting the minimum number of points in the clusters to be detected to a first predetermined number. The processing unit also offsets the position of each measurement point belonging to a cluster included in the moving point cloud data of past frames based on the moving speed and direction of the cluster estimated based on the velocity information of each measurement point. The processing unit also superimposes the data of each measurement point belonging to a cluster from the offset-processed moving point cloud data of past frames onto the moving point cloud data classified by the classification unit 114. The processing unit also performs a clustering process as a second clustering process on the moving point cloud data after the superimposition process, setting the minimum number of points to a second predetermined number greater than the first predetermined number. Furthermore, the processing unit detects the position and size of moving objects in three-dimensional space based on the position and size of the clusters detected by the second clustering process. This enables early and accurate detection of distant objects and small objects.

[0083] (3) The lidar 5 is mounted on a moving object. The above speed information is first speed information, and the object detection device 50 further includes an estimation unit 112 as a speed acquisition unit that acquires second speed information indicating the absolute speed of the moving object. The calculation unit 113 calculates the absolute speed of each of the multiple measurement points corresponding to the point cloud data based on the first speed information and the second speed information. The classification unit 114 classifies the point cloud data into moving point cloud data for measurement points whose absolute value of the absolute speed calculated by the calculation unit 113 is equal to or greater than a predetermined speed. As a result, even when the lidar is mounted on a moving object, distant objects and small objects can be detected early.

[0084] The above embodiment can be modified into various forms. Modifications are described below. In the above embodiment, the lidar 5 as a detector is mounted on a vehicle and acquires point cloud data including three-dimensional position information and velocity information indicating relative movement speed at measurement points on the surface of objects included in the three-dimensional space by irradiating electromagnetic waves into the three-dimensional space around the vehicle and receiving reflected waves. However, the detector may be something other than a lidar. Specifically, it may be a 4D imaging radar that acquires four-dimensional information (point cloud data) of distance, azimuth angle, elevation angle, and relative movement speed at measurement points on the surface of objects included in the three-dimensional space by irradiating millimeter-wave radio waves and receiving reflected waves. Furthermore, the mobile body on which the detector is mounted may be something other than a vehicle, such as a self-propelled robot.

[0085] Furthermore, although the above embodiment uses an object detection device that constitutes part of the vehicle control device 100 as an example, the object detection device and the detector provided by the object detection device may be installed outside the vehicle or may be a stationary type.

[0086] Furthermore, in the above embodiment, the superposition unit 117 and the detection unit 118 act as processing units and perform the offset processing before performing the overlay processing. However, the processing units may also perform the overlay processing before performing the offset processing.

[0087] Furthermore, in the above embodiment, the generation unit 115 converts the moving point cloud data obtained by the classification unit 114 into two-dimensional data, adds absolute movement velocity to the two-dimensional moving point cloud data to generate three-dimensional velocity-added data (XYV data), and the provisional detection unit 116 and the detection unit 118 perform clustering processing on the XYV data to detect moving objects around the vehicle. However, if accuracy of the cluster size in three-dimensional space (XYZ space) is required, the clustering processing described above may be performed on the XYZ space. Specifically, the generation unit may add absolute movement velocity to the moving point cloud data obtained by the classification unit 114 to generate four-dimensional velocity-added data (hereinafter referred to as XYZV data). In this case, the provisional detection unit 116 and the detection unit 118 perform clustering processing on the XYZV data.

[0088] Furthermore, in the above embodiment, the estimation unit 112, which acts as the speed acquisition unit, selects a representative measurement point Pi from among the remaining measurement points after excluding the measurement point corresponding to the three-dimensional object from a plurality of measurement points, and estimates the absolute speed of the vehicle based on the position information and speed information of the representative measurement point extracted from the point cloud data acquired by the data acquisition unit 111, and acquires the estimation result as second speed information. However, the speed acquisition unit may also acquire the measurement result of the absolute speed of the vehicle acquired by a measuring instrument included in the internal sensor group 3 as second speed information. In this case, the object detection device 50 includes at least a vehicle speed sensor from the internal sensor group 3 as a measuring instrument. Alternatively, the speed acquisition unit may calculate and acquire the absolute speed of the vehicle based on the current position of the vehicle measured by the positioning unit 2. In this case, the object detection device 50 includes the positioning unit 2.

[0089] Furthermore, in the above embodiment, the driving control unit 121 controls the vehicle's driving to avoid collisions or contact with objects detected by the detection unit 118. However, the driving control unit 121 may also output information (such as image information) indicating the position and size of objects detected by the detection unit 118 as a detection result to a display device (not shown) or the like. In addition, the driving control unit 121 may also act as a notification unit to predict the possibility of collision or contact with a moving object based on the size, position, and speed of the moving object detected by the detection unit 118. When the possibility of collision or contact with a moving object is above a predetermined level, the vehicle control device 100 may notify the occupants of the vehicle of warning information (video information or audio information) regarding collisions or contact with moving objects detected by the detection unit 118 via a display or speaker (not shown) provided by the vehicle control device 100.

[0090] Furthermore, although the object detection device 50 was applied to an autonomous vehicle in the above embodiment, the object detection device 50 can also be applied to vehicles other than autonomous vehicles. For example, the object detection device 50 can also be applied to a manually driven vehicle equipped with ADAS (Advanced driver-assistance systems).

[0091] The above description is merely an example, and the present invention is not limited by the embodiments and modifications described above, as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and to combine modifications with each other. [Explanation of symbols]

[0092] 5 LIDA, 10 Controller, 11 Calculation Unit, 12 Memory Unit, 111 Data Acquisition Unit, 112 Estimation Unit, 113 Calculation Unit, 114 Classification Unit, 115 Generation Unit, 116 Provisional Detection Unit, 117 Superposition Unit, 118 Detection Unit, 119 Vector Calculation Unit, 120 Tracking Unit, 121 Driving Control Unit, 50 Object Detection Device, 100 Vehicle Control Device, AC Actuator

Claims

1. A detector that acquires point cloud data including three-dimensional position information and velocity information indicating relative movement speed at measurement points on the surface of an object included in the three-dimensional space by irradiating electromagnetic waves into a three-dimensional space and receiving the reflected waves, A calculation unit that calculates the absolute moving speed of each of the multiple measurement points corresponding to the point cloud data based on the speed information, When the point cloud data is acquired by the detector, a classification unit classifies the point cloud data into moving point cloud data where the absolute value of the absolute moving speed calculated by the calculation unit is equal to or greater than a predetermined speed, and stationary point cloud data other than the moving point cloud data. A storage unit that stores the moving point cloud data classified by the classification unit for each point cloud frame containing the moving point cloud data at the same time, The system includes a processing unit that performs processing to detect a moving object moving within the three-dimensional space, The aforementioned processing unit, An offset process is performed to offset the position of each measurement point in the moving point cloud data included in the past point cloud frame stored in the storage unit, based on the moving speed and direction of each measurement point estimated based on the velocity information of each measurement point. When the moving point cloud data newly acquired by the detector is classified by the classification unit, an overlay process is performed on the moving point cloud data, overlaying the moving point cloud data of the previous point cloud frame that has been offset. An object detection device characterized by detecting the moving object based on the moving point cloud data after the superposition process.

2. In the object detection device according to claim 1, The aforementioned processing unit, When the moving point cloud data newly acquired by the detector is classified by the classification unit, a first clustering process is performed on the moving point cloud data, setting the minimum number of points in the cluster to be detected to a first predetermined number. In the offset process, the positions of each measurement point belonging to the cluster included in the moving point cloud data of the past point cloud frame are offset based on the moving speed and direction of the cluster estimated based on the velocity information of each measurement point. In the superposition process, the data of each measurement point belonging to the cluster from the previously offsetted moving point cloud data of the point cloud frame is superimposed on the moving point cloud data classified by the classification unit. A second clustering process is performed on the moving point cloud data after the superposition process, in which the minimum number of points is set to a second predetermined number that is greater than the first predetermined number. An object detection device characterized by detecting the position and size of the moving object in the three-dimensional space based on the position and size of the clusters detected by the second clustering process.

3. In the object detection device according to claim 1, The aforementioned detector is an object detection device characterized by being mounted on a moving object.

4. In the object detection device according to claim 3, The speed information mentioned above is first speed information, The system further includes a speed acquisition unit that acquires second speed information indicating the absolute speed of the moving body, The calculation unit calculates the absolute movement speed of each of the multiple measurement points corresponding to the point cloud data based on the first speed information and the second speed information. The object detection device is characterized in that the classification unit classifies the point cloud data into the moving point cloud data, specifically the measurement points whose absolute value of the absolute moving velocity calculated by the calculation unit is equal to or greater than the predetermined velocity.

5. In the object detection device according to any one of claims 1 to 4, An object detection device characterized in that the detector is a lidar or radar.

Images