Target recognition device

Abstract

The objective of the present invention is to provide a target recognition device capable of stably determining whether a radar target acquired from a radar device and a camera target acquired from a camera device are the same target, regardless of the depth distance of the camera target. To this end, a fusion target generating unit (8) projects, onto an image captured by a camera device (3), radar target information, which is information relating to a radar target (12a, 14a) measured by a radar device (4), to determine a radar target projection position, which is the position of the radar target on the image, and compares the radar target projection position with a camera target position, which is the position, on the image, of a camera target (12b, 14b) included in the image, to determine whether the camera target (12b, 14b) and the radar target (12a, 14a) are the same target.

Description

Target recognition device

[0001] The present invention relates to a target recognition device that detects and recognizes a target based on radar information and image information.

[0002] In recent years, driving assistance devices such as an Advanced Emergency Braking System (AEBS) for collision mitigation braking and an Adaptive Cruise Control (ACC) for following a vehicle while maintaining a fixed distance between vehicles have been developed. It is important for these driving assistance devices to detect and recognize targets (e.g., a vehicle traveling ahead) that exist in front of the vehicle. Target recognition devices include those equipped with radar or lidar (LiDAR) that detect targets using electromagnetic waves such as millimeter waves or laser light, those equipped with a camera that detects targets by analyzing captured images, and those using sensor fusion including a camera with radar or lidar.

[0003] Cameras and radars use completely different distance measurement principles. Camera distance measurement utilizes the difference in the position of a target captured in an image in an image coordinate system. For example, a stereo camera with two synchronized cameras (left and right) calculates distance using triangulation, based on the parallax, which is the difference in pixel position of the target point in each image coordinate system between the two images. A monocular camera estimates distance based on the difference in the position of the target point in the image coordinate system between adjacent frames. Meanwhile, radar emits electromagnetic waves, such as millimeter waves, in a specified direction and receives the electromagnetic waves reflected by the target. A delay proportional to the distance occurs between the transmitted and received electromagnetic waves, and the distance to the target is calculated from this delay. Horizontal position is detected by converting the phase difference of the received waves between multiple receiving antennas spaced at a specified interval into a horizontal angle.

[0004] Radar generally has the advantage of being able to measure position information such as the distance to a target and its horizontal position, as well as its relative speed, and is also able to detect targets even in conditions where forward visibility is poor due to bad weather such as dense fog. However, it has the disadvantage of low horizontal resolution due to limitations on the number and spacing of multiple receiving antennas. Cameras generally have the advantages of high horizontal and vertical resolution on images, making it possible to detect the size of targets and having a wide field of view. However, because distance is calculated based on the position difference between the image coordinates described above, the accuracy of the distance to the target and its relative speed is poor. Furthermore, because they are easily affected by lighting conditions, their target detection ability is likely to decrease in bad weather or at night.

[0005] Because LIDAR uses near-infrared light, it is vulnerable to snow and fog, and its detection capabilities decrease in bad weather. It also faces the cost issue of being more expensive than cameras and millimeter-wave radar.

[0006] On the other hand, in sensor fusion, a radar target detected by a radar device is compared with a camera target detected by a camera, and if it is determined that the radar target and the camera target are the same target, the radar target and the camera target are treated as one fusion target, and if it is determined that the radar target and the camera target are not the same target, they are treated as separate fusion targets. Sensor fusion is expected to complement the disadvantages and advantages of both radar and camera (see, for example, Patent Document 1).

[0007] Japanese Patent Application Laid-Open No. 2007-218738

[0008] Because target detection by camera devices, such as stereo cameras, and target detection by radar devices each have their own unique distance measurement errors, it is difficult to determine whether the targets detected by each device are the same or different. This makes the determination process complicated, as it requires using not only the relative distance between the targets detected by each device, but also the relative speed relationship.

[0009] The inventors also discovered that conventional object position information acquisition means, when directly measuring observable quantities, convert into real-space position information measured in meters, generate errors dependent on the direct observation quantities of each object detection means. For example, even if the horizontal angle accuracy of an object in a captured image is very high, if there is an error in the depth distance to the object acquired from the image data, in real space, not only the error in the depth distance but also the error in horizontal length units will increase. This error tends to increase as the depth distance to the object increases. Furthermore, stereo cameras often produce a constant parallax error of less than one pixel as an offset. When this error is converted to real-space distance, it is almost insignificant at close ranges, but the distance error increases at long distances.

[0010] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a target recognition device that can stably determine whether a radar target acquired from a radar device and a camera target acquired from a camera device are the same target, regardless of the depth distance of the camera target.

[0011] In order to achieve the above object, the present invention provides a target recognition device including a camera device, a camera target acquisition unit that acquires, as camera target information, information including a camera target position, which is the position on an image of a camera target that is a target included in the image captured by the camera device, a radar device, a radar target acquisition unit that acquires, as radar target information, information about a radar target that is a target detected by the radar device, and a fusion target generation unit that fuses the camera target information and the radar target information to generate fusion target information, in which the fusion target generation unit projects the radar target information onto the image to determine a radar target projection position, which is the position of the radar target on the image, and determines whether the camera target and the radar target are the same target by comparing the camera target position and the radar target projection position.

[0012] According to the present invention, by projecting a radar target acquired from a radar device onto an image captured by a camera device and comparing it with the position of a camera target acquired from the camera device, it is possible to stably determine whether the radar target and the camera target are the same target regardless of the depth distance of the camera target.

[0013] Schematic diagram of a vehicle equipped with a target object recognition device according to a first embodiment of the present invention. Functional block diagram of a driving assistance device according to a first embodiment of the present invention. Flowchart showing the processing of a fusion target object generation unit according to a first embodiment of the present invention. Flowchart showing details of the same target object determination processing according to a first embodiment of the present invention. Diagram showing the state ahead of the vehicle. Diagram showing the same target object determination method according to the prior art and the first embodiment of the present invention. Functional block diagram of a driving assistance device according to a second embodiment of the present invention. Flowchart showing the processing of a fusion target object generation unit according to a second embodiment of the present invention. Diagram showing an example of a method in which the fusion target object generation unit according to the second embodiment of the present invention estimates the size of a target from the parallax of surrounding pixel positions of a radar target.

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same elements are designated by the same reference numerals, and duplicated explanations will be omitted as appropriate.

[0015] Fig. 1 is a schematic diagram of a vehicle 1 according to a first embodiment of the present invention. The vehicle 1 is equipped with a driving assistance device 2 (see Fig. 2). A stereo camera 3 serving as a camera device is attached to the front ceiling of the vehicle interior at a certain height from the road surface. A millimeter-wave radar 4 serving as a radar device is attached to a front structural member of the vehicle 1, such as a front grille or a front bumper.

[0016] FIG. 2 is a functional block diagram of a driving assistance device 2 mounted on a vehicle 1. The driving assistance device 2 includes a stereo camera 3, a millimeter-wave radar 4, a radar target acquisition unit 5, a camera target acquisition unit 6, a fusion target generation unit 8, a preceding vehicle detection unit 9, an adaptive cruise control (ACC) unit 10, and a controller unit 11. The functional units 3 to 11 constituting the driving assistance device 2 are electronic control devices configured by microcomputers equipped with a central processing unit (CPU), memory, programs, a timer, input / output units, etc., and are connected to a multiplexed communication line such as a controller area network (CAN) bus. The controller unit 11 functions as a bus controller that relays data between the functional units 3 to 10. Note that each of the functional units 3 to 10 does not need to be configured by a single microcomputer; two or more of the functional units may be integrated into a single microcomputer. The stereo camera 3, the millimeter wave radar 4, the radar target acquisition unit 5, the camera target acquisition unit 6, and the fusion target generation unit 8 constitute a target recognition device 100 in this embodiment.

[0017] The stereo camera 3 is composed of a pair of left and right cameras using solid-state imaging elements such as CMOS sensors. These left and right cameras are installed side by side and integrated at a fixed distance. The left and right cameras capture images of objects outside the vehicle from different viewpoints. The stereo camera 3 performs correction processing such as affine transformation on each image (reference image and comparison image) captured by the left and right cameras, and then performs parallax calculation by block matching using the two left and right images.

[0018] In disparity calculation using block matching, a window of 8 pixels x 8 pixels is set for the left and right images, and the pixel positions in the left and right images where small images within the set window match are searched for. The difference in pixel positions of matching points in the left and right images is the disparity. An image in which the calculated disparity is assigned to each pixel in the image is called a disparity image. SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference) is used as a function for determining the degree of match.

[0019] In addition, one of the left and right images is used as a reference image, and various image processing such as edge detection and segmentation is performed on the reference image. A three-dimensional object is detected from the parallax image and the image that has been subjected to various image processing, and the parallax and coordinates on the reference image at a point that represents the three-dimensional object, such as the center position of the three-dimensional object, are output to the camera target acquisition unit 6.

[0020] The reference image that has been subjected to various image processing processes is sent to the lane information acquisition unit 7. The lane information acquisition unit 7 detects white lines and the like painted on the road surface using a Hough transform or the like, and outputs the detected lines to the preceding vehicle detection unit 9 as lane data.

[0021] The camera target acquisition unit 6 assigns an ID to candidates for preceding vehicles among the three-dimensional objects detected by the stereo camera 3, and registers the parallax and coordinate position on the reference image. At this time, for targets that are thought to be the same as those in the previous frame, the target position information registered in the previous frame is used to calculate the target speed, which is also registered as target information and output to the fusion target generation unit 8 as camera target information.

[0022] The millimeter-wave radar 4 has a transmitter / receiver unit that transmits and receives millimeter waves at regular intervals within a predetermined scanning range in the horizontal direction of the vehicle 1. The distance from the vehicle to a three-dimensional object in front of the vehicle is measured from the time difference between the transmitted millimeter waves being reflected by a reflective object such as a three-dimensional object in front of the vehicle and returning. The speed is measured from the Doppler shift in the frequency of the transmitted and received waves. The receiver unit has multiple receiving antennas installed at predetermined intervals in the horizontal direction, and the horizontal distance is measured from the phase difference of the millimeter waves received by the receiving antennas. These are output to the radar target acquisition unit 5.

[0023] Since there may be multiple reflecting objects, the radar target acquisition unit 5 registers the ID, depth distance, horizontal distance, and speed as radar targets in descending order of reflection intensity. The registered target data is output to the fusion target generation unit 8 as radar target information.

[0024] The fusion target generation unit 8 compares the radar target information output from the radar target acquisition unit 5 with the camera target information output from the camera target acquisition unit 6, and determines whether the two are the same target or different targets. The fusion target generation unit 8 assigns a new fusion target ID to targets determined to be the same target or different targets, registers the fusion target information, and outputs it to the preceding vehicle detection unit 9.

[0025] The preceding vehicle detection unit 9 extracts a preceding vehicle traveling ahead of the own vehicle's lane from the lane data output from the lane information acquisition unit 7 and the fusion target information output from the fusion target generation unit 8, and outputs the inter-vehicle distance to the preceding vehicle and the relative speed between the preceding vehicle and the own vehicle to the ACC unit 10.

[0026] The ACC unit 10 outputs vehicle control signals such as an accelerator signal and a brake signal based on the inter-vehicle distance and relative speed to the preceding vehicle, which are outputs from the preceding vehicle detection unit 9 .

[0027] 3 is a flowchart showing the processing of the fusion target generation unit 8. First, the camera target information acquired by the camera target acquisition unit 6 is read (step S101). The camera target information includes the parallax, coordinate position (x, y) on the image, and relative speed for each preceding vehicle candidate ID. The maximum number of preceding vehicle candidates detected by the camera target acquisition unit 6 is predetermined (for example, a maximum of five). If the camera target ID number is I, the parallax Δcamera(I), coordinates on the image (xcamera(I), ycamera(I)), and relative speed v(I) are registered for each ID number.

[0028] Following step S101, the radar target information acquired by the radar target acquisition unit 5 is read (step S102). The radar target information includes the depth distance, horizontal distance, and relative speed for each preceding vehicle candidate ID. The maximum number of preceding vehicle candidates that the radar target acquisition unit 5 will detect is also predetermined (e.g., a maximum of seven vehicles). If the radar target ID number is J, the depth distance Z(J), horizontal distance X(J), and relative speed u(J) are registered for each ID number.

[0029] Following step S102, the depth distance Z(J) and horizontal distance X(J) are projected onto a reference image (e.g., the right image) (step S103). In the ACC application, since the target is limited to a preceding vehicle, the target height is set to the vehicle height and a predetermined value (e.g., 1 m) is used as the estimated value. When the focal length of the stereo camera is f [m], the pixel pitch p [m], the camera mounting height H [m], and the estimated vehicle height h [m] are assumed, and the origin of the reference image is the optical center, the position (xradar, yradar) on the image where the radar target is projected is given by: xradar(J) = (f / p) × X(J) / Z(J) (Equation 1) yradar(J) = (f / p) × (h-H) / Z(J) (Equation 2)

[0030] Following step S103, a parallax conversion value of the depth distance Z(J) of the radar target is calculated (step S104). When the baseline length of the stereo camera 3 is B [m], the parallax conversion value Δradar(J) [pixels] of the depth distance Z(J) of the radar target is given by Δradar(J)=f×B / (p×Z(J)) (Equation 3).

[0031] In the subsequent processing, radar target information and camera target information are compared in order and fused into a fusion target. For the sake of explanation, an example is shown in which radar targets are first assigned as fusion targets with priority, and then targets for which the same target was not found by the radar and camera are assigned as fusion targets. The priority can be determined by assigning camera targets with priority, or by prioritizing targets at close range.

[0032] Following step S104, the radar target ID of the radar target to be determined as the same target and the fusion target ID of the fusion target to be registered are each initialized to 1 (step S105).

[0033] Following step S105, it is determined whether the radar target ID of the currently targeted radar target is equal to or less than the maximum number of preceding vehicle candidates detected by the radar target acquisition unit 5 (step S106). If it is equal to or less than the maximum number, the process proceeds to step S107. If it exceeds the maximum number, all radar targets are assigned as fusion targets, and the process proceeds to step S115.

[0034] In step S107, the camera target ID of the camera target to be compared with the currently targeted radar target is initialized to 1.

[0035] Following step S107, it is determined whether the camera target ID of the camera target to be compared is equal to or less than the maximum number of preceding vehicle candidates detected by the camera target acquisition unit 6 (step S108). If it is equal to or less than the maximum number, the process proceeds to step S109. If it exceeds the maximum number, the comparison of the currently targeted radar target with all camera targets has been completed, and there is no camera target that is the same as the currently targeted radar target. Therefore, fusion target information is generated and registered based only on the radar target information (step S113).

[0036] In step S109, it is determined whether the currently targeted radar target and the camera target to be compared are the same target. Details of the same target determination process will be described later. If it is determined that they are the same target, the camera target ID is registered as already used (step S111), and fusion target information is generated and registered based on the radar target information and the camera target information (step S112). If it is determined in step S109 that they are not the same target, 1 is added to the camera target ID of the camera target to be compared (step S110) in order to compare with the next camera target, and the process returns to step S108.

[0037] When the registration of the fusion target is completed in step S112 or step S113, 1 is added to the radar target ID of the radar target to be determined as the same target and the fusion target ID of the fusion target to be registered (step S114), and the process returns to step S106.

[0038] In step S115 and after, a process is performed to assign a camera target that is not determined to be the same target as any of the radar targets to a fusion target. Note that a radar target that is not determined to be the same target as any of the camera targets has already been assigned to a fusion target in step S113.

[0039] First, in order to search for a camera target that has not been determined to be the same target as any radar target, the camera target ID of the camera target to be searched for is initialized to 1 (step S115).

[0040] Following step S115, it is determined whether the target camera target ID is equal to or less than the maximum number of preceding vehicle candidates detected by the camera target acquisition unit 6 (step S116). If the maximum number is exceeded, all camera targets are assigned to fusion targets, and the flow ends. If the maximum number is not exceeded, it is determined whether the camera target ID of the target camera target is an unused camera target ID that was not registered as used in step S111 (step S117). If the camera target ID is an unused camera target ID, fusion target information is generated and registered based only on the camera target information (step S118).

[0041] Following step S118, the fusion target ID of the fusion target to be registered is incremented by 1 (step S119), and it is determined whether the number of fusion target IDs to be registered is equal to or less than a predetermined maximum number (step S120). If the number exceeds the maximum number, the flow ends. If the number is equal to or less than the maximum number, the camera target ID of the camera target to be compared is incremented by 1 (step S121), and the process returns to step S116.

[0042] 4 is a flowchart showing the details of the same target determination process (step S109). First, it is determined whether the radar target and the camera target are close to each other on the reference image (step 201). A specific example of this determination method is described below.

[0043] A radar object with a radar object ID of J is projected to a position (xradar(J), yradar(J)) on the reference image according to Equation 1, and a camera object with a camera object ID of I is located at a position (xcamera(I), ycamera(I)) on the reference image. Therefore, it is determined whether the difference in the horizontal pixel positions of the radar object and the camera object on the reference image (Equation 4), or the sum of the difference in the horizontal pixel positions of the radar object and the camera object on the reference image and the difference in the vertical pixel positions of the radar object and the camera object on the reference image (Equation 5), is equal to or less than a predetermined threshold (first threshold). |xradar(J) - xcamera(I)| (Equation 4) |xradar(J) - xcamera(I)| + |yradar(J) - ycamera(I)| (Equation 5) The determination threshold in Equation 4 may be set to a pixel-converted value of the angle error θ of the millimeter-wave radar 4. Assuming that the angular error θ is θ, the pixel pitch p is θ, and the number of pixels n is θ, the following relationship holds: f×tanθ=n×p (Formula 6) Here, if the angular error θ of the millimeter-wave radar 4 is 1 degree, the focal length f=3.8 mm, and the pixel pitch p=3 μm, the pixel equivalent value of the angular error θ is approximately 20 pixels.

[0044] If the determination result in step S201 is NO, it is determined that the radar target and the camera target are not the same target (step S204), and the flow ends. If the determination result in step S201 is YES, it is determined whether the parallax converted value of the radar target and the parallax of the camera target are close to each other (step S202). A specific example of the determination method will be described below.

[0045] The depth distance Z(J) of a radar target having a radar target ID of J is converted into disparity using Equation 3. Therefore, it is determined whether the difference (Equation 7) between the disparity converted value Δradar(J) of the depth distance Z(J) and the disparity Δcamera(I) of a camera target having a camera target ID of I is equal to or less than a predetermined threshold (second threshold): |Δradar(J)−Δcamera(I)| (Equation 7) The determination threshold in Equation 7 may be set to, for example, about 0.5 pixels, since the disparity error is equal to or less than 1 pixel.

[0046] If the determination result in step S202 is YES, it is determined that the radar target and the camera target are the same target (step S203), and the flow ends. If the determination result in step S201 is NO, it is determined that the radar target and the camera target are not the same target (step S204), and the flow ends.

[0047] An example of the execution of the same target determination process will be described with reference to Figures 5 and 6. Figure 5 is a diagram showing the state ahead of the host vehicle. In the upper diagram of Figure 5, the host vehicle is traveling in a lane defined by a left boundary line 13l and a right boundary line 13r, and a preceding vehicle 12 is traveling at a distance in the same lane. A radar target 12a and a camera target 12b are detected for the preceding vehicle 12. In the lower diagram of Figure 5, the host vehicle is traveling in a lane defined by a left boundary line 15l and a right boundary line 15r, and a preceding vehicle 14 is traveling close to the same lane. A radar target 14a and a camera target 14b are detected for the preceding vehicle 14.

[0048] The left diagram of Figure 6 is a bird's-eye view of the situation in Figure 5, in which distant targets 12a, 12b and close targets 14a, 14b are plotted, and left boundary lines 13l, 15l and right boundary lines 13r, 15r of the own lane are drawn so that the positional relationship with each target can be seen. In the same target determination according to the prior art, the horizontal position and depth distance of the radar targets 12a, 14a are compared with the horizontal position and depth distance of the camera targets 12b, 14b in the diagram to determine whether the two targets are the same target.

[0049] The right diagram of Figure 6 shows the same target determination method according to this embodiment. In this embodiment, the closeness of the horizontal positions of each target is evaluated by the horizontal pixel position, and the closeness of the depth distance is evaluated by the parallax, so the horizontal axis of the diagram represents the horizontal pixel position and the vertical axis represents the parallax. Furthermore, as the above-mentioned same target determination criterion, the range where the horizontal pixel position is within 20 pixels and the parallax is within 0.5 pixels is shown by a dashed line.

[0050] In the prior art (left diagram in FIG. 6 ), the difference in horizontal position between the radar target 12 a and the camera target 12 b of the distant preceding vehicle 12 is larger than the difference in horizontal position between the radar target 14 a and the camera target 14 b of the close preceding vehicle 14, so the result of determining whether the radar targets 12 a, 14 a and the camera targets 12 b, 14 b are the same target is affected by the depth distance.

[0051] In contrast, in this embodiment (right diagram in FIG. 6 ), the horizontal target position including the horizontal angle error obtained from the radar device 4 is projected onto an image obtained from the camera device 3 with high precision and almost no horizontal angle error, so it is only necessary to consider the horizontal angle error of the radar device 4. As a result, the horizontal pixel position difference between the radar target 12a and the camera target 12b of a distant preceding vehicle becomes approximately the same as the horizontal pixel position difference between the radar target 14a and the camera target 14b of a close preceding vehicle. This makes it possible to determine with high precision whether the radar targets 12a, 14a and the camera targets 12b, 14b are the same target.

[0052] Furthermore, in the prior art, the difference in depth distance between the radar target 12a and the camera target 12b of the distant preceding vehicle 12 is greater than the difference in depth distance between the radar target 14a and the camera target 14b of the close preceding vehicle 14, so the result of determining whether the radar targets 12a, 14a and the camera targets 12b, 14b are the same target is affected by the depth distance.

[0053] In contrast, in this embodiment, the depth distance of the target obtained by the radar device 4 is converted into a position difference in image coordinates such as parallax, and this is compared with depth distance information expressed as a position difference in image coordinates obtained from image information of the camera device 3. This makes it possible to ignore errors of the radar device 4, which has high distance measurement accuracy, on the image, and to consider only errors of the camera device 3. As a result, the parallax difference between the radar target 12a and the camera target 12b of the distant preceding vehicle 12 becomes approximately the same as the parallax difference between the radar target 14a and the camera target 14b of the close preceding vehicle 14. This makes it possible to determine with high accuracy whether the radar targets 12a, 14a and the camera targets 12b, 14b are the same target.

[0054] (Summary) In this embodiment, the target recognition device 100 includes a camera device 3, a camera target acquisition unit 6 that acquires, as camera target information, information including camera target positions, which are positions on an image of camera targets 12 b, 14 b that are targets included in the image captured by the camera device 3, a radar device 4, a radar target acquisition unit 5 that acquires, as radar target information, information on radar targets 12 a, 14 a that are targets detected by the radar device 4, and a fusion target generation unit 8 that fuses the camera target information and the radar target information to generate fusion target information. In this embodiment, the fusion target generation unit 8 projects the radar target information onto the image to determine radar target projection positions, which are the positions of the radar targets 12 a, 14 a on the image, and compares the camera target positions with the radar target projection positions to determine whether the radar targets 12 a, 14 a and the camera targets 12 b, 14 b are the same target.

[0055] According to the present embodiment configured as described above, by projecting the radar targets 12a and 14a acquired from the radar device 4 onto the image captured by the camera device 3 and comparing the positions of the radar targets 12a and 14a with the positions of the camera targets 12b and 14b acquired from the camera device 3, it is possible to stably determine whether the radar targets 12a and 14a and the camera targets 12b and 14b are the same target, regardless of the depth distance of the camera targets 12b and 14b.

[0056] Furthermore, the fusion target generation unit 8 in this embodiment determines the horizontal positions of the radar targets 12a, 14a on the image based on the horizontal positions of the radar targets 12a, 14a measured by the radar device 4, and determines the vertical positions of the radar targets on the image based on the depth distances to the radar targets 12a, 14a and estimated values ​​of the heights of the radar targets 12a, 14a from the ground contact point measured by the radar device 4. This makes it possible to determine the horizontal and vertical positions of the radar targets 12a, 14a measured by the radar device 4 on the image.

[0057] In this embodiment, the camera device 3 generates a parallax image based on two captured images, and the fusion target generation unit 8 converts the depth distance to the radar target measured by the radar device 4 into a parallax in the parallax image, and compares the parallax of the camera target with the converted parallax value of the radar target to determine whether the camera target and the radar target are the same target. This makes it possible to stably determine whether the radar targets 12a, 14a and the camera targets 12b, 14b are the same target, regardless of the depth distance of the camera targets 12b, 14b.

[0058] Furthermore, the fusion target generation unit 8 in this embodiment determines that the camera target and the radar target are the same target when the difference between the camera target position and the radar target projection position is equal to or less than a predetermined first threshold and the difference between the parallax of the camera target and the parallax conversion value of the radar target is equal to or less than a predetermined second threshold, and determines that the camera target and the radar target are not the same target when the difference between the camera target position and the radar target projection position exceeds the first threshold or the difference between the parallax of the camera target and the parallax conversion value of the radar target exceeds the second threshold. This makes it possible to determine whether the radar targets 12a, 14a and the camera targets 12b, 14b are the same target based on the positions and parallaxes of the radar targets 12a, 14a and the camera targets 12b, 14b on the images.

[0059] A second embodiment of the present invention will be described, focusing on the differences from the first embodiment. Figure 7 is a functional block diagram of a driving assistance device 2 in the second embodiment. Similar to the first embodiment (see Figure 2), the driving assistance device 2 in this embodiment includes a stereo camera 3, a millimeter-wave radar 4, a radar target acquisition unit 5, a fusion target generation unit 8, a preceding vehicle detection unit 9, and an adaptive cruise control (ACC) unit 10. In this embodiment, the output of the radar target acquisition unit 5 and the parallax image and reference image from the stereo camera 3 are directly output to the fusion target generation unit 8, which generates camera target information.

[0060] The stereo camera 3 outputs a reference image (e.g., a right image) and a parallax image after correction processing such as affine transformation to the fusion target generation unit 8. The fusion target generation unit 8 projects the radar target information received from the radar target acquisition unit 5 onto the parallax image or the reference image from the stereo camera 3, or onto an image obtained by performing image processing such as edge processing on the reference image, and performs target detection in an image coordinate system.

[0061] 8 is a flowchart showing the processing of the fusion target generation unit 8 in the second embodiment. The fusion target generation unit 8 reads the reference image and the parallax image from the stereo camera 3 (steps S301 and S302), and reads radar target information from the radar target acquisition unit 5 (step S303). As the radar target information, the ID, the distance in the depth direction, the distance in the horizontal direction, and the speed are read in order of the reflection intensity.

[0062] Following step S303, the depth distance of the radar target is converted into a parallax using equation 3 (step S304), and the radar target position on the parallax image is calculated using equations 1 and 2 (step S305). At this time, vertical height information of the radar target is required, but since the target in the ACC application is limited to a vehicle, an estimated vehicle height value (for example, 1 m) may be used, or an estimated value may be obtained from the parallax image or the reference image.

[0063] Following step S305, the parallax conversion value of the radar target calculated in step S304 and the radar target position on the parallax image calculated in step S305 are used to plot the parallax conversion value at the radar target position on the parallax image (step S306). An example of a parallax image in which the parallax conversion values ​​of the radar targets are plotted is shown in the left diagram of Fig. 9. In the left diagram of Fig. 9, the parallax conversion values ​​of a near radar target 18a and a distant radar target 18b are plotted. By projecting the radar target on the parallax image 17 in this manner, if the parallax conversion value of the radar target and the parallax at the surrounding pixel positions are close, the size of the radar target can be estimated from the parallax at the surrounding pixel positions (the radar target information can be strengthened), thereby improving the reliability of the radar target information.

[0064] Here, an example of a method for estimating the size of a radar target from the parallax at pixel positions around the radar target by the fusion target generation unit 8 will be described. On the right side of the left diagram in Fig. 9 (disparity image 17 on which the parallax converted values ​​of a near radar target 18a and a distant radar target 18b are plotted), a graph 20a showing the relationship between the parallax and the vertical pixel position on a vertical cross section 19a passing through the plot position of the radar target 18a on the disparity image 17, and a graph 20b showing the relationship between the parallax and the vertical pixel position on a vertical cross section 19b passing through the plot position of the distant radar target 18b on the disparity image 17, are shown.

[0065] In the graphs 20a and 20b, the parallax of the road surface appears as a straight line 23 with a constant slope. Furthermore, when a camera target is detected, the parallax of the camera target appears as vertical lines 21a and 21b on the graphs 20a and 20b. The lengths of the vertical lines 21a and 21b, expressed in pixel units, and their respective parallaxes are converted from the target height and depth distance in meters. Therefore, target height information can be obtained from these and the camera parameters. While the vertical cross section of the parallax image has been described in this example, the horizontal cross section can also be used to estimate the horizontal size of the target. Furthermore, because the radar target is projected onto the parallax image 17, the errors 22a and 22b between the parallax of the camera target (vertical lines 21a and 21b) and the radar target's parallax conversion value can be evaluated with high accuracy. Generally, radar distance measurement errors in meters are small, so when this is converted to parallax, the parallax error is smaller than that of a stereo camera, etc. Therefore, by monitoring the difference between the converted parallax value of the radar target and the parallax of the surrounding pixel positions on the parallax image, the depth distance calculated from the camera target information can be corrected.

[0066] Returning to FIG. 8, following step S306, image processing required for detecting three-dimensional objects, such as edge processing and segmentation, is performed on the reference image (step S307), and the points on the image obtained in step S305 are plotted on the image after image processing (step S308).

[0067] Following step S308, a three-dimensional object detection process is carried out using the image that reflects the radar target information obtained in steps S306 and S308 (step S309).

[0068] Following step S309, target detection processing is performed (step S310), and the flow ends. Information on the detected target is sent to the preceding vehicle detection unit 9.

[0069] (Summary) The fusion target generating unit 8 in this embodiment enhances the radar target information fused into the fusion target information based on the parallax of the pixel positions surrounding the radar target projection position in the parallax image.

[0070] According to the present embodiment configured as described above, it is possible to improve the reliability of the radar target information fused into the fusion target information.

[0071] Furthermore, the fusion target generation unit 8 in this embodiment corrects the depth distance of the camera target calculated from the parallax of the camera target included in the camera target information fused with the fusion target information, based on the parallax of the pixel positions surrounding the radar target projection position in the parallax image. This makes it possible to improve the accuracy of the depth distance of the camera target calculated from the parallax of the camera target included in the camera target information fused with the fusion target information.

[0072] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are presented to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0073] 1...vehicle, 2...driving assistance device, 3...stereo camera (camera device), 4...millimeter-wave radar (radar device), 5...radar target acquisition unit, 6...camera target acquisition unit, 7...lane information acquisition unit, 8...fusion target generation unit, 9...preceding vehicle detection unit, 10...ACC unit, 11...controller unit, 12...preceding vehicle, 12a...radar target, 12b...camera target, 13l...left boundary line, 13r...right boundary line, 14...preceding vehicle, 14a...radar target, 14b...camera target, 15l...left boundary line, 15r...right boundary line, 17...parallax image, 18a, 18b...radar target, 19a, 19b...vertical cross section, 20a, 20b...graph, 21a, 21b...vertical line, 22a, 22b...error, 23...straight line, 100...target recognition device.

Claims

1. A target recognition device comprising: a camera device; a camera target acquisition unit that acquires, as camera target information, information including a camera target position, which is the position on an image of a camera target that is an object included in an image captured by the camera device; a radar device; a radar target acquisition unit that acquires, as radar target information, information about a radar target that is an object detected by the radar device; and a fusion target generation unit that fuses the camera target information and the radar target information to generate fusion target information, wherein the fusion target generation unit projects the radar target information onto the image to determine a radar target projection position, which is the position of the radar target on the image, and compares the camera target position and the radar target projection position to determine whether the camera target and the radar target are the same target.

2. A target recognition device as described in claim 1, characterized in that the fusion target generation unit determines the horizontal position of the radar target on the image based on the horizontal position of the radar target measured by the radar device, and determines the vertical position of the radar target on the image based on the depth distance to the radar target and an estimated value of the height of the radar target from the ground point measured by the radar device.

3. A target recognition device as described in claim 1, characterized in that: the camera device generates a disparity image based on two captured images; and the fusion target generation unit converts the depth distance to the radar target measured by the radar device into a disparity in the disparity image, and determines whether the camera target and the radar target are the same target by comparing the disparity of the camera target with the converted disparity value of the radar target.

4. In the target recognition device described in claim 3, the fusion target generation unit determines that the camera target and the radar target are the same target when the difference between the camera target position and the radar target projection position is equal to or less than a predetermined first threshold and the difference between the parallax of the camera target and the parallax conversion value of the radar target is equal to or less than a predetermined second threshold, and determines that the camera target and the radar target are not the same target when the difference between the camera target position and the radar target projection position exceeds the first threshold or when the difference between the parallax of the camera target and the parallax conversion value of the radar target exceeds the second threshold.

5. A target recognition device as described in claim 3, characterized in that the fusion target generation unit enhances the radar target information fused with the fusion target information based on the disparity of surrounding pixel positions of the radar target projection position in the disparity image.

6. A target recognition device as described in claim 3, characterized in that the fusion target generation unit corrects the depth distance of the camera target calculated from the camera target information fused with the fusion target information based on the parallax of surrounding pixel positions of the radar target projection position in the parallax image.

Images